RSS for offers?

Daniel · October 2021

Is it possible to get an RSS feed for the offers on this site? I tried https://talk.lowendspirit.com/categories/les-exclusive-offers/feed.rss but it redirects to the login page when my RSS reader tries to load it

I don't like checking lots of different sites, so I try to centralise everything via RSS.

deank · October 2021

???
People still use RSS?

Daniel · October 2021

@deank said: People still use RSS?

Yes!

Twitter and Facebook are good for aggregating content, but too noisy. On FB I mostly care about posts from friends and groups rather from pages (and that's mostly what I interact with, so I barely see any page posts), and Twitter is just a disorganized mess of a feed, plus not every company is on social media (and certainly not all the hosting deals on here and "the other forum" are posted to Twitter). With RSS I can see a feed of everything, plus I can just view individual categories (eg. "Programming", "Hosting Deals", etc) if I only want to view posts in that category.

I'm using FreshRSS and its web UI on my PC, along with the Readrops app on Android.

I'm also using "Kill the Newsletter" to convert email newsletters to RSS feeds as I've already got enough junk in my inbox... I don't want newsletters in there too!

deank · October 2021

Twitter is cespool of humanity, so I don't use it.
Facebook..., the last time I visited that site was more than 10 years ago.

Newsletters..., I sternly block'em all. Don't want them.

Decicus · October 2021

LES exclusive offers aren't available to the public, which I guess is the reason you're being redirected to login.

I'm not sure there's much you can do to get around that, besides including cookies when sending a request, but I don't know if FreshRSS supports that.

willie · October 2021

It would great if the site had an API.

Daniel · October 2021

@Decicus said:
LES exclusive offers aren't available to the public, which I guess is the reason you're being redirected to login.

I'm not sure there's much you can do to get around that, besides including cookies when sending a request, but I don't know if FreshRSS supports that.

FreshRSS can't do it today, but I can add it if needed. The only thing is that the cookies may periodically rotate for security reasons (fairly common these days).

@willie said:
It would great if the site had an API.

Isn't that basically what RSS for, at least for a read-only API? If it allowed username and password via HTTP Basic auth then that'd be useful.

Neoon · October 2021

not RSS but JSON:
https://github.com/Ne00n/Offers/blob/main/data/talk.lowendspirit-urls-offers.json

Daniel · October 2021

@Neoon said:
not RSS but JSON:
https://github.com/Ne00n/Offers/blob/main/data/talk.lowendspirit-urls-offers.json

Combining all offers into a single file would make for a very useful RSS feed. Even if it was just in one JSON file I could just use Huginn to convert it to an RSS feed (I'm already using Huginn to scrape offers from a few hosts that aren't listed on Serverhunter)

jaden · October 2021

+1 to RSS all the things. I too use FreshRSS and try to centralize as much as I can, but I've not found a good solution for RSS feeds that require a login other than basic auth, which works perfectly in FreshRSS.

Fritz · October 2021

@Daniel said:

@Neoon said:
not RSS but JSON:
https://github.com/Ne00n/Offers/blob/main/data/talk.lowendspirit-urls-offers.json

Combining all offers into a single file would make for a very useful RSS feed. Even if it was just in one JSON file I could just use Huginn to convert it to an RSS feed (I'm already using Huginn to scrape offers from a few hosts that aren't listed on Serverhunter)

Huggin seems the right one for me.

Now I need to find a proper guide to install it.

Daniel · October 2021

An approach I could take here is to write a script that:

Does a GET to /entry/signin to get the CSRF token to log in (although I don't see one there... I know Vanilla is kinda basic when it comes to security, but please tell me it at least has a CSRF token somewhere).
Does a POST to /entry/signin to log in, saving the cookie jar (eg cURL with --cookie-jar).
Does a GET to the RSS feed using the same cookie jar.

Or do the same using Huginn, which should be doable.

I think Vanilla also has an API of some sort, but knowing Vanilla it's probably basic and individual users can't use it without getting an API key from an admin (also why bother use an API if RSS is a lot simpler?)

@Fritz said: Huggin seems the right one for me.
Now I need to find a proper guide to install it.

I'm just using it via Docker(not even Docker-Compose yet) on one of my spare VPSes at the moment:

docker run -d -it -p 10.123.1.70:3000:3000 -v /opt/huginn/data:/var/lib/mysql -e SMTP_DOMAIN=notify.d.sb -e SMTP_USER_NAME=xxxxxx -e SMTP_SERVER=xxxxxxx -e SMTP_PASSWORD=xxxxxxx -e [email protected] huginn/huginn

I'm using it to create an RSS feed just for offers from particular users (like Hosthatch) on "the other forum". I'm using an RSS agent to read their RSS feed once per hour, a trigger agent to filter it to just items where the author = hosthatch, and a data output agent to output it to a private RSS feed for FreshRSS to consume plus a Pushover agent to push it to my phone.

Using it to scrape Wishosting's daily deals too, via a website agent that runs once per day:

{
  "expected_update_period_in_days": "3650",
  "url": "https://www.wishosting.com/order/main/packages/VPS/?group_id=7",
  "type": "html",
  "mode": "on_change",
  "extract": {
    "name": {
      "css": ".package-name",
      "value": "normalize-space(.)"
    },
    "price": {
      "css": ".price-box",
      "value": "normalize-space(.)"
    },
    "details": {
      "css": ".package",
      "value": "./node()"
    }
  },
  "template": {
    "url": "{{ url | to_uri: _response_.url }}",
    "price": "{{price | replace:'Starting at ',''}}",
    "subject_prefix": "Wishosting SALE: "
  }
}

mikho · October 2021

@willie said:
It would great if the site had an API.

It does check https://success.vanillaforums.com/kb/articles/40-api-v2-overview

Daniel · October 2021

@mikho said:

@willie said:
It would great if the site had an API.

It does check https://success.vanillaforums.com/kb/articles/40-api-v2-overview

How does one get an access token though? Plus I don't really want to spend time using the API to build an RSS feed if it already has an RSS feed built in 😛

mikho · October 2021

@Daniel said:

@mikho said:

@willie said:
It would great if the site had an API.

It does check https://success.vanillaforums.com/kb/articles/40-api-v2-overview

How does one get an access token though? Plus I don't really want to spend time using the API to build an RSS feed if it already has an RSS feed built in 😛

Never did look into the API, only know its there.

Regarding RSS, some RSS-clients can handle feeds behind a login.
I used to use Tiny Tiny RSS, you could enter user/pass with that one.

willie · October 2021

I guess RSS is like a read-only API and since there are readers for it, it is worthwhile. I wouldn't think of it as something obsolete. It went into relative disuse because evil sites like Facebook shut it down to stop people from bypassing ads and other manipulation. But it is still a thing.

A more general API that allowed posting etc. would be nice for people who want to bypass the web UI altogether. This could be limited to regular members who request access, and restricted to some posting velocity, to prevent spam.

Daniel · October 2021

@willie said: A more general API that allowed posting etc. would be nice for people who want to bypass the web UI altogether.

Yeah... The ideal design is for the website itself to be built on top of the same API, which automatically means that all features on the site are also accessible through the API. Some of these older forum systems have coupled the business logic too tightly to the web UI, and the API is just an afterthought that lacks all the same feature.

mikho · October 2021

@willie said:
A more general API that allowed posting etc. would be nice for people who want to bypass the web UI altogether. This could be limited to regular members who request access, and restricted to some posting velocity, to prevent spam.

@Daniel said:
Yeah... The ideal design is for the website itself to be built on top of the same API, which automatically means that all features on the site are also accessible through the API. Some of these older forum systems have coupled the business logic too tightly to the web UI, and the API is just an afterthought that lacks all the same feature.

Vanilla API allows posting comments and creating new topics/discussions/articles.

So if you want to build something extraordinary, feel free to do so.

Check the link I posted earlier and read up on the possibilities.

corbpie · October 2021

It's not enabled here though

https://success.vanillaforums.com/kb/articles/41-authentication-with-personal-access-tokens

mikho · October 2021

@corbpie said:
It's not enabled here though

https://success.vanillaforums.com/kb/articles/41-authentication-with-personal-access-tokens

Security setting needs to be changed to allow creating user tokens.

Is there a genuine interest of using the API?

Neoon · October 2021

You don't need an API to auth, login works fine with a POST request.
But with API keys would be better for sure.

how · October 2021

@Neoon said:
not RSS but JSON:
https://github.com/Ne00n/Offers/blob/main/data/talk.lowendspirit-urls-offers.json

It's too useful! thank you!

RSS for offers?

Comments