Clicks starting to take three to ten seconds
Seems like people are continuing posting new deals with ease, though!
General internet and navigating to direct deal store websites is fine.
Unable to see other people mentioning the site issues in forum topics
Actually it's the 5000+ logged in users that are heavy hitters on our database. Logging out if you are just browsing — you'll probably get a better experience on OzBargain today.
Surprising. Thanks for the insight.
Thanks scotty
The OzB tracking database!
Just tested a 20-second load on main page from a separate device. 👻
I am amazed at the users able to reply to new comments within a couple minutes
Just got a seven-second page load on deals with one-second homepage load 🏇
scotty, we need more power.
But Captain, she kenno take it!
Scotty doesn't know
Or incognito if you don't want to log out.
Nice insight. Just for my own curiosity as to the gravity of the situation today, what would you say is the average range of the number of logged in users on a normal day? I am just curious because I am never noticed any lag on OzB until today.
Usually around ~2,500 — so it's about double the usual logged in sessions right now. However the number of requests from logged in users have increased way more than 2 fold. People stayed on the site longer, viewing/reloading more pages, etc.
This could be a useful message to have as a sitewide banner
Logging out if you are just browsing — you'll probably get a better experience on OzBargain today.
Why is that?
Is it just the extra elements that are loaded for logged in users?
The entire page can be cached for guest sessions.
Probably 20k limit including logged users.
Still way better than centrelink website.
Noticed it, understandable given people will be keen for a bargain today
Hoped a CDN would at least cache the main displays.
I am developing an aversion to pressing direct reply or publish buttons now.
Ironically not seeing many stand out deals on the actual Black Friday day 😅
Time to fall back to generic price trackers
Dealbot is winning
Agree - the last few days have been great - nothing today!
That makes sense, thanks
That makes sense, thanks Cheers.
so laggy
scotty needs to spend some more coin
That goes against the OzBargain core principles
People keep refreshing because they are waiting for the actual bargains ;)
Free CDN would work super effective for that! :(
CDN can only cache web page / assets that can be cached. Each OzBargain page view from a logged in user can generate up to hundred queries customised for that specific user and can't really be cached on the CDN level.
Sorry but your argument does not work.
And we do already utilise a CDN.
@UsernameChecksIn: Take OzBargain home page for example,
including main pages, posts could be statically served
The list of deals are already cached. However the current status (whether a post is "new"), the number of new comments that you haven't read, whether you have voted on the deals, etc — those aren't cached for all the users and need to be individually queried for logged in sessions.
widget parts such as ozbdeal competitions forum
The sidebar listings are all cached. But again the status (whether a post is new or not) is queried once per block of deals / forum posts.
additionally for semi-frequently updated parts including voting and comments
Users would be very annoying by that during the majority not-so-busy period of the year (why my vote isn't showing up?!) and we aren't doing a separate code path for that 1% time of the year.
Other aggregate workarounds
Our caching strategy is already on the aggressive side which has caused issues in the past. Obviously these are all tech problems that can find solutions. Say, if you are a product manager looking at OzBargain's functionality and working out an estimate on the effort + cost to solve the scalability issue specifically on Black Friday — how many people do you need and how much would that cost?
That might be the reason why we are still having issues every year.
@scotty: Thanks Scotty
Appreciating the insights into how our much-loved OzBargain site works
Activity spikes generating commissions incommensurable to the design :(
@nuker: LOL.
There used to be a saying, there are only two hard things in Computer Science… For someone spending "10 years of doing this shit" you should know what I am talking about. At this stage I am sick of people saying "just use AWS" with minimum understanding of scalability. Or "just use CDN" or "just throw in some memcached" without understanding that the hardest problem with caching lies with granularity and invalidation.
Source: have worked on database/data storage scalability since '98.
downvoted.. did not really explain why the non-logged in cacheable pages were still lagging behind as well!
fan-ism!
why the non-logged in cacheable pages were still lagging behind as well
It's because the non-logged in session is still a session on the application servers, not on the CDN. CDN basically does not know whether a session ID belongs to a logged in user or a guest user (which also has session ID). Thus serving a page request for guest user still requires a round trip between the front end reverse proxy (CDN, in this case Cloudflare) and the backend application server(s).
Now, executing a page review for a guest user can be much faster in the backend in the case when the whole page HTML is cached in memcache. However,
Dammit, Fisher!
if it was gaming, I'd be whining - but this lag is saving me money…………
20% off (speed)
.
Took like 20s to upvote this gold.
ozbargain is being ozbargain-ed
When did Black Friday start being such a big deal in Oz? Serious question here- I have been away for a long time and going from the posts here, I thought it was last week as I don't track American holidays.
Its been gaining more momentum every single year. Its almost in line with the increasing popularity of Halloween.
The Black Friday deals have probably been getting worse though. You'll never get the best price for something on Black Friday.
I used to have easy access to US Black Friday deals, and after a while just started ignoring the whole thing because the deals were either for junk, or were discounts of jacked up pricing. Sounds like it might be similar here.
OzBargain website lagging during Black Friday?
Yes
Ah! Reminds me to post a poll next time!
Yes.
No?
Bikies!
https://files.delvu.com/images/competitions/ozbgangsign/terr…
*^also seeing a couple more of these images hosted via scotty's Delvu media domain rather than files.ozbargain
It's doing a lot better than the Harvey Norman website
Estimated wait time: 00:15:00
Your position in line: 35372
A virtual queue for the visitors might be an easier solution than scale our database at the moment… OzBargain Premium members might also get a fast pass :)
gulps
Can't wait for the 2024 Black Friday deals on OzB Premium subs
I’d pay a few quid for Ozbargain premium and to support the site!
@scotty - we need shiny graphs to show us the data :D
In Grafana we trust.
Here's the loadavg of the database server over the last 24 hours (at the time of posting this comment).
Love the stats! Thanks captain ;-)
I find these reasonable responses
Would be keen to examine clickthrough stats v Amazon though!
Wait, are you able to embed images only because you are admin?
scotty do have all the powers :)
You need OzBargain Premium
Took me 4 minutes to get to this page.
Checked forum to see if it's just me being throttled or everyone is having issues
Upvoting too many entries!
I did not see any duplicate posts this morning so made one with happy community response 🥳
Slow AF, looks like they have discounted on Speeds as well.
But all good
Ozbargain is getting ozbargained!
i must admit i am partially to blame. Logged in on an Officeworks display PC to check for mobile phone bargains while I was in store.
Ozbargception
Stop posting and commenting and voting! Ah damn now I'm part of the problem!
Yes.
Voted on your comment to bring awareness to this issue… damn oops!
i think the whole internet is taking a beating today….streaming is really bad too
We ozbargained ourself.
OK, I'll stop commenting for a bit…
So delayed that jv only caught the fiftieth comment!
It's still lightning fast if you have a platinum membership.
OzBargain Premium members might also get a fast pass
👀
Certainly not true. As Admin pointed it out above, guests have faster browsing experience than normal registered users. A user in the penalty box may be even better.
Alright. Yes, OzBargain is slow on almost every Black Friday weekend, and thanks, but no thanks to those who are trying to provide "technical solutions" because most are either already inplace, or irrelevant. The bottleneck has always been the database which at the moment is doing up to 15k queries per second. It's running on a dedicated hardware (8c/16t Xeon E-2288G) and the bottleneck is CPU-bound rather than memory or IO. It's already the fastest the hosting provider can offer so we can't really upgrade vertically. We also run dedicated instances of memcached for caching at various level of granularity, and redis for write-heavy operations.
The only feasible solution would be scale horizontally by replicating the database and the read-only sessions can be spreaded over to other replicated DB instances. We are already doing replication as live DB backups, and already have code in place for picking a database connection from a pool at the beginning of the request. However with master/slave replication arrangement, we still have issues in our code (i.e. can't do any state-changing statements on read-only connections). I'll put that in high priority to be rectified next year.
Do note that I'm probably the only system engineer here (moocher spent half of her time doing mostly feature development) + there are a lot of other work to be done in running OzBargain. Don't compare us with well-funded startups or Amazon that have an entire team doing operations.
Download more ram
picks the provider with hundreds of cores and terabytes of RAM 💸💸💸
Don't worry about those script kiddies. Some of us here appreciate your hard work and know to expect these kind of issues during sales.
Appreciate all your hard work! Ozbargain runs better than some well funded startups too 😉
Holy crap. An entire database server run of an 8c Xeon it's hard to believe that in this day and age. Proper Xeon servers have dozens of cores.
OzB Blockchain.
Maybe not worth it for the 2-3 times a year CPU use spikes on the database box, but the Xeon E-2288G is an old CPU. Came out in 2019. There's heaps of hosts that can give you dedicated servers with a shitload more cores if you want an upgrade.
Vultr has AMD EPYC boxes in MEL/SYD with 24c/48t for US$725/m
DigitalOcean has 16/32/48 vCPU boxes starting at US$437 in SYD
Even DigitalPacific have a wide range of EPYC boxes starting at 16c for A$1099/m
Well done @scotty and team to keep OzBargain up & running throughout. Without an architecture diagram of OzBargain, & understanding of where the problems/bottle necks are, a lot of solutions mentioned here are simply hypothetical and on paper although hopefully can give you some directions. Being "on-prem"/"with hosting provider" does not help either with its limited scaling flexibility, elasticity.
The only feasible solution would be scale horizontally
It's interesting that this topic is discussed like it is impossible to change one line of application code, when it's said there up to 100 customised queries that are run for a specific user. All discussion is on hardware or caching optimising..
but no thanks to those who are trying to provide "technical solutions"
Haha.
I feel like 99% of ozbargain usefulness relates to being able to load the home page during Black Friday or in general. These performance problems inhibit that.
As you don't want technical solutions, I will present you a non-technical solution. Put up a banner at the top of the page advising that load times are slow and that users can log out or open ozbargain in Incognito for a faster experience during that weekend.
This will reduce load on your infrastructure [as more people use Incognito] and actually provide a solution for people who are unable to load/access the website reliably. Right now, only people who dig deep in to the forums would find this post and know that logging out would improve their performance.
FWIW I accessed ozbargain on my phone and computer that day at different times and I thought I was having internet problems. A banner would have at least let me know that ozbargain had a known issue.
The public presumably read-only CDNed page per response was yet still taking tens of seconds to load.
Misconfig?
Community documentation. Stagnant team growth by design for multimillion revenue?
Cool diversity of feedback if only a single engineer - random inputs are even more valuable to keep probing for incremental improvements.
At least there is less blanket censorship like other times
Though that could still be coming
Just like every year, and the question gets asked every year.
Sounds like there could at least be a proactive banner similar to other themed events
Multi-year saga for eliciting improvements!
You could also just be patient for 5 seconds a couple of times a year?
Thanks for the tip to logout to get better performance.
That just means fewer votes on deals.
Just open an incognito window.
The public logged out pages were still lagging
Encountered sproadic normal load times on occasion
Significant issues with this site today
its been completely unusable today. Never had this issue before.
It's fine at nights,
ie. between midnight and 7am…muhaha.
14500 guests so doesn’t surprise me.