For example, https://www.ozbargain.com.au/deals?page=50 returns
404 Not Found
Page that you are trying to request cannot be found.
All pages from 50 onwards are not found.
Cannot View Old Deals past Page 50
alvian on 06/07/2018 - 01:19
Last edited 06/07/2018 - 08:05
Last edited 06/07/2018 - 08:05
Comments
We have to deliberately set a hard limit on the maximum number of old pages, because in the past we have bots iterating through thousands of pages of past deals. What do you think is a sensible limit there?
I don't have anything in mind scotty. It was the first time I have seen such a limit imposed on a public forum. I needed to go past page 50 today to hunt down the reason a particular deal was unpublished on a date that I know (so I cannot search for it), found this limit and thought it was a bug.
In light of your response perhaps return a more informative message than a 404 error code? The limit you've set is not an error but an artificial restriction. Clicking on an URL to an unpublished deal, for example, does not return a 404 but a content removed message.
How about a code 418? :)
Or imposed a request rate limiter for pages beyond 50 to get rid of bots but allow human?
I remember having your issue a while ago but I thought it no longer applied with OzBargain's new infinite scrolling feature? Of course it's not very convenient if you're looking back a few months/years, but an iterative process would be labour intensive regardless.
For the record you can easily find the deals leading up to an exact epoch timestamp using the OzBargain API, but I don't know if scotty wants the URL publicised (since I imagine this compromises the former limitation).
Does the API return unpublished deals and the reasons? I can use search to find normal and expired deals but not unpublished ones.
@peterpeterpumpkin: That good to know. The only API mentioned in the help pages is for submitting deals: https://www.ozbargain.com.au/wiki/help:api
@alvian: Sorry for the confusion. It's not in the "published" API. Apparently there's no plans for a universal API.
Can you set it so that it has a robot identification challenge to see pages 51-100 and then another challenge for 101-150?
Pages 51++ are just eneloop deals :D
You reminded me: When infinite scroll fails, the button to load more deals says "Load More Result…", which isn't pretty English.
Anyone else notice OzBargain has been down a lot lately (i.e. twice in a fortnight)?
We have been under quite a few DDoS attack recently.
- 16 June 22:58
- 18 June 21:35
- 1 July 20:30
- 3 July 19:04
Last around an hour each time.
thats not a bug, thats a feature