Been twiddling my thumbs for the last 2hrs at work, so i thought I'd do something useful.
Let's hear your stories about how you have been affected by this.
https://www.abc.net.au/news/2024-07-19/technology-shutdown-a…
Major IT Outage Affecting Banks, Media Outlets in Australia and Globally
gomertose on 19/07/2024 - 16:30
Last edited 19/07/2024 - 16:31
Last edited 19/07/2024 - 16:31
Related Stores
Comments
@try2bhelpful: I thought you would say that, because you are part of the weak cohort that relies on government handouts to survive. And that's what the government likes too, because people that rely on the government to survive like you, are going to do what they are told without questioning anything.
I noticed your editing of your previuosly passive aggressive post by the way.
@Mad Max: The point is I did decide to be more diplomatic; which apparently you aren’t. Given how quickly I modified it you must be hovering on the comments section.
Honestly mate you are digging yourself further down a rabbit hole with your comments. You go straight to aggressive. I don’t rely on Government handouts because I’ve built my own wealth base through my years of work. I will, probably, never be eligible for a pension. However, just keep posting what you “assume” to be true. Google that saying as well.
If you know anything about my posts you would know I’m always questioning how society operates, including the Government. Some people don’t like that either.
Government makes all decisions and people just follow blindly. No thinking, no checking, no scrutinising, no questioning.
This is the issue, there's too much groupthink and echo chambers these days. People aren't willing to question the validity of anything that doesn't fall on their side of the political spectrum and people are more polarised than before. For example if you call out the mass migration we have since Labor took over (one migrant every 42 seconds actually) you're automatically labelled a racist, even though immigrants actually come from countries across the globe and a lot of migrants actually come from the UK and NZ.
Forget the fact that the number of Aussies living in tents the past few years has grown, forget the fact most of the jobs created the past year were non-market jobs (i.e. gov jobs like the NDIS rort), forget the fact that rental vacancies are at an all time low and that vacancies are driven not only by supply (which the government and MSM constantly peddles to gaslight the public) but also by demand and that demand is coming from population growth which is mainly driven by net overseas migration and not natural increase… No, forget all that, if you call out immigration you are automatically reduced to one word — racist. It doesn't matter that people's overall social mobility has literally eroded because they can't live somewhere because it's simply become unaffordable to do so, no, that doesn't matter at all, what matters is that…. yOuR a RaCiSt!1!!11
It's sheer idiocy, complete brainwashing by the MSM. As the years have gone by the government has started to care less and less about this country and more and more about themselves, their property portfolios and their rich mates. Anyone who pays attention should be able to see that. FHB grants? Yeah no that's not to actually help you buy a house it's to keep the property ponzi going. Immigration? Yeah no that's not to make the country better, it's to flood the employment market with labour thereby increasing supply and suppressing wage growth for their big business mates. Buying $370 bn worth of nuclear subs? Obviously because China wants to invade us and it's the only way to protect our country, it's definitely not to feed more money to the US military industrial complex.
And then people will say "Well Australia is still a great country" and sure, we have been. But great countries only get great through hard work and care and if you drop the ball things will fall apart completely and that is where the real issue lies. We can't just rest on our laurels and act like we are great, it requires continuous effort.
@Ghost47: Confirmation bias. People (including intellectuals) only seek out opinions that agree with their own opinion.
To get a job in academia or the public service or the media you need to a member of the "Statelib Gang".
@Typical16-bitEnjoyer: It's not change that people are scared of, it's change that's ill-thought-out or done poorly (and likely to benefit the few and make life worse for the many) that people are scared of.
We've seen what happens when things fall over, there seems to be one payment outage a year for each year the past few years. If we want to go cashless then that is completely unacceptable. If you didn't have any cash in your wallet yesterday you would have had to go searching for a place where a terminal was working or go without.
I'm sorry but everything about modern life requires computer systems to function as intended to some degree.
And fossil fuels… which are finite.
That’s a falsehood to say we either use tech for everything or go back to the farm and that’s the only choice.
There’s nothing wrong with some safeguards and redundancy planning. There’s literally too many single points of failure shown time and time again when things go wrong from failed optus connection to this or some other single line of code waiting to fail again.
Totally agree, Vic had that storm earlier this year that took out a transmission tower for electricity and the widespread effect of that was eye opening. Virtually the same massive failure for EFTPOS. Cashless society and digital balances is a very risky utopian dream.
That’s a falsehood to say we either use tech for everything or go back to the farm and that’s the only choice.
It's also a falsehood when people accuse others who don't want to move to a completely cashless society as also wanting to rely on old technology or wanting to ride horses instead of driving cars. Some people don't want to go cashless simply because the risk is too great and the redundancies aren't robust enough.
There’s nothing wrong with some safeguards and redundancy planning.
I don't see the government doing this properly. Not only are politicians mostly career public servants, they also have the interest of their donors at heart as a priority. It would be naive to believe they actually care about the general public. Have we started moving electrical transmission wires underground for example in case we get a storm like what THICKnSLOW mentioned?
I don't see the government doing this properly. Not only are politicians mostly career public servants, they also have the interest of their donors at heart as a priority. It would be naive to believe they actually care about the general public. Have we started moving electrical transmission wires underground for example in case we get a storm like what THICKnSLOW mentioned?
The reasons why redundancy planning doesn’t exist is because no one wants to talk about it no one wants to pay for it no one cares until something happens and then something happens for a short time and then the next issue takes over the public discourse and it’s forgotten about.
I don’t envy a politicians job.
Let's not forget the single input stream from one sensor that Boeing deemed good enough for their MCAS system.
Then the cost blowouts, delays, software bugs and ultimate stranding of their "Starliner".Everythings fine here, situation normal!
Unfortunately too many sheep here in Oz, hardly anyone carries cash because it's more convenient to carry a card…….. Until yesterday.
At least no one here was on a Plane. Doors weren't able to open due to the outage 🤣🤣.
They should have thrown some cash at the doors.
So you parrot comments about sheep?
Yep 🦜, get another shoot of your booster 🐑 🤣
@Bretttick: Why aren’t I surprised, in the least, with your comments? The ship logic has sailed right past you and you have become becalmed in the websites of paranoia.
I was in the drive-thru at KFC at what seemed to be moments before the issue happened. By that point I had already specified I was picking up my order (which was only for the $2 mousse) in drive-thru and had the order number ready to quote. By the time I got to the window they said the system crashed, I didn't think much of it at the time so went and waited for about 15 minutes. Came back and was told by the manager the outage was a bit more serious. I told them I had already put the order through on the app and had the order number on my phone and asked if I could just get the mousse because surely it would've gone through already if I got the order number confirmation but was told to come back later.
So went and did some shopping, came back after 1.5 hours and the drive-thru line was closed off, went in store and saw they were doing everything manually and people were paying by cash. Waited in line told the staff about the issue again, a different manager came out and I told him the same story "I was in drive-thru, clicked to pick up what seemed to be moments before the crash, my order number is ####, can I just take the mousse? Won't the order show up on your end because I already specified I'd be picking it up in drive-thru?" Manager said yes it should show up on our end when the system comes back up, the staff wrote down my order number and I got the mousse. Talk about a pain in the arse to get the $2 mousse. It better be a damn good mousse.
In the past 2-3 years we seem to be having a mass outage on average once a year or something. This one seems really serious because it's affected services globally. I thought our systems were supposed to be robust 🤷♂️ I was listening to the radio and the host said that the issue seems to have occurred because of a software update which I found funny because it seems to me now that every time something like this happens the go-to excuse is that it was a "software update issue". Absolutely zero transparency whatsoever to gaslight the public, just like when Optus was hacked and the CEO said "It's a sophisticated attack, the hacker is changing their IP constantly and they're located overseas" as if we live in a Hollywood movie because the reality was that the data was accessed via an unprotected API.
Whatever happened today happened because someone(s) didn't do their job correctly. Maybe it was a once-off massive screw up or maybe it was a series of minor mistakes along the way that have led to a complete catastrophe. Maybe whoever was working wasn't trained up properly or couldn't be bothered doing their job correctly or maybe someone who had the knowledge of how to perform the job properly was laid off and took that tacit knowledge with them and didn't document it (just like what's happening at Boeing).
Verdict on mousse?
Haven't had it yet but I'll report back once I have.
Well done on writing that monster post instead of eating said mousse.
If you’re going to kfc for a mousse, wouldn’t you eat it asap? I mean if you were going to wait so long, wouldn’t it be more economical to get it from a supermarket?
Anyway, hope it is good and glad you go the mousse@john_conner: Haha thanks.
If you’re going to kfc for a mousse, wouldn’t you eat it asap?
Nah my intention was to have it after a meal.
I mean if you were going to wait so long, wouldn’t it be more economical to get it from a supermarket?
Yeah probably, but I had the day off so wasn't in a rush. On top of that I wasn't going to let KFC get away with taking my money and not getting my mousse lol.
Anyway, hope it is good and glad you go the mousse
Thanks mate, turned out to be just ok lol.
That's a well earned mouse
It was ok but not as good as Red Rooster's IMO (just cos RR has caramel sauce which really takes it up a notch). Not worth the struggle lol.
I find KFC moose alright for $2 but wouldn't pay much more for it. I find mixing it up before helps a ton with consistency. - I am associated with kfc
@Quarter Pounder: Yeah that's a fair call, anything more than $2 seems a bit steep. You should try the Red Rooster mousse, that is some good stuff.
@Ghost47: I’ve tried it in the past, cant lie pretty good. Red roosters chicken to me is better then kfc
Crowdstrike shares down 13.5% in pre-market lol.
I don't see them surviving the lawsuits.
That's fair. This seems like a huge stuff up. On top of your comment above I've read that to reboot systems people need to take a USB to them manually? This is huge.
If it is stuck in an infinite BSOD loop that doesn't even boot into networking, then yes some kind of physical intervention for every single device in that state is needed. It is huge. I'm already reading some companies have 300,000 endpoints, which will include PC's, terminals, hand held devices, etc.
It is possible not all devices are stuck in this loop, but my entire company's employee laptops certainly are.
@lawyerz: That's what was happening at the KFC I visited. The screens were showing a continuous boot cycle into a BSOD lol, at the login screen there was just a spinning circle then it rebooted. Holy shit this is insane. All those KFCs that need to be visited… and that's just KFC.
Someone on reddit said they work at an ED and it's chaos and others have said their hospital is down and surgeries have been cancelled. Wow.
@lawyerz: every register and almost every PC in the (hospitality) venue went into BSOD boot loop.
last night we resorted to cash sales and writing every sale down to be input later.
all meal orders were hand written to give to the chefs.
ATM went offline for about 4 hours.
Pokies didnt crash themselves but had to close for regulatory reasons for an hour or so. because of lack of back of house systems.we fell back to a Chromebook for back of house stuff to get open this morning.
i had half the registers open by 10am open
but i finally got the last register up and running by 5pm Saturday.some didn't use bit locker, and those that do required a different process but skipped using the bitlocked C: drive and seemed to boot to a network drive X:
apparently, their EULA covers limited liability. so it's going to be a painful battle to win. And defended by CS for sure.
Ever lived in a third world country? All they have is cash. Speaking from first had experience.
More than 5 hours at Cairns airport (including one hour spent on the plane sitting on the tarmac) waiting to fly back home to Melbourne then told flight has been cancelled. Staff suggested finding accomodation in Cairns until they can figure something out!!!!! Even though Melb is cold I’d rather be home!
Sorry to hear, I know your pain, I had a delayed flight on Monday, missed all the last trains in Melbourne
Update: Jetstar never communicated with us as they said they would in their pre-recorded phone message and on their website. After nearly two hours waiting on their online chat they found us flights, earliest Tues 23!!!! Not even direct but at least we have a returning home date!
I called Jetstar last time they had long delays and the call center took ages to tell me they couldn’t help. Didn’t even sound sincere. I use Jetstar twice a year in peak times for over 15 years visiting relatives up north.
Now, I noticed Virgin were only down for an hour for this CrowdStrike screwup so I am going to pay more and preference them in the future.also their boarding gates are closer, quieter and generally nicer down that end of the terminal in Melbourne.
Trump's revenge for all the jokes about wishing he died.
You mean the potus
Absolute rubbish company - who will deploy at the same time something untested on millions of computers.
After 5 pm today, I wanted to send money to my Wise account but Upbank app notified about Osko/PayID payments are unavailable and also traditional bank transfers (via BSB/Account number) are unavailable. So, I cannot use Upbank to send money. Then I tried to send money through St. George bank app via BSB/account number and it went through, but the money didn't arrive until few hours later … At least the money has arrived. I read the news about this after I send the money …
Many times I informed in Ozb about keeping some cash just in case you cannot use your card or network problem (visa/MasterCard/bank problem), but some or many people here don't want/like to keep some cash …
And how are you going to use cash when you can't even scan items through the checkout?
Did you read the comments above me? Mapax said "Just got home from coles, It was cash only"
Ghost47: "The KFC I went to was definitely not ok. I think it took them nearly two hours to figure out "Let's just start accepting cash orders".
If you think cash is like insurance, you don't need it until you really need it … Have some cash as a backup at least …
For all Cash deniers, here are proofs that everyone need to have Cash at least as a backup (news in Australia) :
https://www.9news.com.au/national/eftpos-outage-woolies-cust…
https://www.abc.net.au/news/2023-06-26/commbank-commonwealth…
https://www.9news.com.au/national/mcdonalds-reports-nationwi…
https://7news.com.au/news/nab-customers-hit-by-widespread-ou…
https://www.abc.net.au/news/2024-03-04/australians-lose-one-…
https://www.abc.net.au/news/2023-12-30/will-consumers-have-t…
https://7news.com.au/sunrise/sunrise-star-monique-wright-exp…
I hardly use cash anymore but I still think it's silly to move completely cashless. Even one outage a year is too many in a cashless society, and I'm pretty sure we've had outages at least once the past two or three years where people have been unable to pay for things.
Outages is only a matter of time, not IF. Imagine natural disaster like flooding, no electricity, undersea internet cables broken that affect global Internet outage, etc. Microsoft may not have good quality control/assurance for Windows Update. How many times already Windows Update problems, lol. I read a lot of news …
Yeah that's a fair point. I was actually wondering if this outage was due to a data centre going up in flames or getting flooded somewhere.
@Ghost47: Wonder why maybe no lawsuits for bad Windows Update? If only happened once, people still think okay … But how many times already bad Windows Update? Really bad effects for businesses and also affect consumers too.
It's all very good to have an option to pay with cash, but how much cash do you actually have? (Rhetorical, don't want hear about the stash 😉)
When all the digital goes to hell, you're not getting anything out of an ATM (especially if you're not one of the first to get to it)
I have hundreds $ cash at least, not bad to survive for few weeks :P
That's why certain people have GOLD for example, because it's easier to cash out. Even people can sell their jewelry, watch, diamond, etc. in Pawn shop, jewelry shop, etc.
@neoleo: Now's the perfect time for anyone to invest in bullion. Silver is a great time to invest in, platinum is also very cheap atm
What's the point of cash when the point of sale systems are down. Unless they're small stores, woolies and Coles aren't going to take out a calculator and start giving you hand written receipts.
Not against cash, and not pro cashless. Just saying if the checkout machine itself is down, and it controls inventory management as well, cash won't help
All for small stores! Stop thinking the big boys are the only stores to go to.
Did you read the comments above me? Mapax said "Just got home from coles, It was cash only"
Ghost47: "The KFC I went to was definitely not ok. I think it took them nearly two hours to figure out "Let's just start accepting cash orders".
At least if this happens to me in Coles/Wollies etc., I can take photos of the goods and prices, show them to the Cashier and pay cash according to the prices in the photos or they can recheck the prices in the shelf again if they are not sure or forget the prices ;-)
IGA and milk bars are always happy on the odd occasion I pay cash
Unless they're small stores, woolies and Coles aren't going to take out a calculator and start giving you hand written receipts.
That's funny because that's exactly what was happening at the KFC I was at last night. One girl was taking orders on the blank side of the receipt paper and another girl was standing next to her using a calculator to add it up, turning around to the menu boards (which luckily were still functioning) to check prices. They didn't give out a receipt but they were still taking orders.
If stores actually care about serving customers (and also not losing money) they'll do exactly that IMO. It'll be a pain in the arse to reconcile at COB and they'll have to keep thorough written records to put the orders through when the system does come back up but at least the day won't be written off because of an outage. If this outage is prolonged stores will have no choice but to do exactly that IMO.
And given nearly all majors will be paying wages regardless and customer counts will be down majorly, why not just get staff to do some manual work around, good for customer who still wants goods, gets some sales, better than nothing.
@cloudy: Yeah exactly, the important thing is that the exchange takes place. Who cares if they have to do it manually, both parties will be happy in the end.
Cash should not be a backup going into the future - having proper redundancy systems and consequences for breaches/outages is.
We didn't keep the horse and cart around because cars could break down 😂
Certain places still have horse and cart for tourism ;-)
How can we use digital money or card/online payment when electricity is down, natural disaster or whatever? Very strong solar storm that affect electronics. Cash is still king :P
when did woolworths update their windows 95 checkouts? lol
I can't believe self-checkout kiosks are running Windows. TIL
No joke, what a nightmare this situation is.The downscaling and consolidation of companies has its repercussions. It is now shown to be true, having all your eggs in one basket is not a good idea.
I got to finish work an hour early, was awesome. Let's do this again next Friday.
I was working from home. I picked up my kid early, took him to the doctor and pharmacy, came home. The whole incident had unfolded while I was out, leaving me as one of the only people with a functioning PC. No early mark!
All the work will be done by the only man standing! Not so lucky. LOL. Just kidding.
I liked the overseas report quoting Australia as early sufferers with a story from some Ozbargainer at the airport saying now that flights were going again he was kinda sorry about missing out on a few more beers at the airport
'A man must believe in something - I believe I'll have another beer'
How do you know they were an OzBargainer? Did the they say something?
Where was affected? I finished work at 15:30, did my shopping, picked up miccy d's on the way home, went to the pharmacy and picked up 15 Grams of Cannabis, continued to shop online and on amazon then played vidya til 3am and all I saw was people complain, no websites I accessed were down, no payment facilities were offline, everything in my life was normal. People on the internet posted and complained a LOT though.
Was it just a (profanity) driver that a lot of people had installed causing BSOD's?Crowdstrike is primarily used by enterprises so it was on their computers (windows endpoint devices) NOT by choice. Or not even aware. So when it goes wrong the reaction is more dramatic. If don't really have a fleet of computers at work, you are not affected, if your IT department had chosen to use any of other endpoint security vendors, you are not affected.
There is some inconvenience from retail and services being down but I took it as early mark and start the weekend early. LOL.
Alot of people jumping on bandwagon and bash everything and everyone that had to do with a simple error (and I appreciate caused widespread impact).
Weird. I primarily use Palo Alto for Firewall, Sentinel for SIEM, Global Protect for VPN, Microsoft Defender for endpoint security and had no such issue.
Not weird at all. Lucky you.
The BSOD is caused by a Crowdstrike driver that is loaded at startup. If your endpoints are not protected with cs they won't have the file or the need to load it. (It will load MS defender in your case or similar realtime agent drivers for those other products).
Probably any company that had automatic updates 'on' or something like that. I only just found out about it when I was checking what KFC's deals were for today and was notified there was an IT outage.
Yeah, it's a balancing act with auto update tho.
Latest and greatest, or tested and compatible.
This wasn’t a patch management or testing failure on the part of those affected. Every customer had this pushed regardless of their update channel configs. I’m sure more will come to light today, but it seems CS has ignored update configs and pushed this to everyone.
@djsweet: Yeah, something went wrong in the build, test and deployment pipeline for sure. You usually wouldn't release a sys driver to 'everyone'.
We used to release it as new installer. After a week of no issue then you push it out to existing customers in batches of OS major builds. But that thoroughness also made it slow and expensive…hence why Symantec is no longer a player i guess.
The KFC I went to had a Macca's next door and the drive-thru line seemed to be moving at a steady pace, I think Maccas's might've been unaffected. There were a lot of visitors when I was there.
Knocked off work early, then was happily gaming on my Linux box. Livin' the dream.
So no one at cyberstrike tested their updates?
Who is cyber strike?
And yes they do, they would probably release daily if not more frequently. Something went wrong, someone clicked the wrong button at CrowdStrike.
We all need to have more appreciation of the work that goes un-noticed by the guys in the basements.
From which country this basement is based at?
Don't see how that matters.
There are stories coming out now that crowdstrike devs can push directly to production and it has caused problems in the past.
I was talking to someone who manages package updates at a large organisation yesterday. They didn't use the default crowdstrike configuration for falcon on updates, they rolled their own to test and install as packages because it was too risky otherwise. He's having an awesome weekend right now, because they had zero issues (beyond vendors who use falcon going down).
Push to prod and on a (profanity) Friday hahahaah.
@Darude Sandstorm: What’s really incredible is it was apparently compiled on Jul 9th, and had just been sitting around untested, waiting to blow everything up. And it’s just a data file with a bunch of nulls. There’s a bigger problem that the software was already so broken that one bad data file will crash millions of computers.
If something of this scope is based on something they do daily then they really need to review their processes because that would be fracking scary. There is no way the system should allow changes that can have this sort of impact without a proper implementation, and testing, process in place. Better living in a basement than in a state of denial. Frankly if the IT people are doing their work properly it remains unnoticed. When I worked in the industry our mantra was “don’t appear on the front page of the newspapers”.
Chemist warehouse too
Got an email that Selfwealth was down too
CMC went off air.
Imagine how many heads are rolling off the team reaponsible for authorising and uploading that lovely piece of work update
The peeps whose heads should roll are already looking for a scapegoat I predict.
at Woolworths Sydney Town Hall yesterday I saw a long queue for the downstairs self-checkouts so got on the escalator to the upstairs self-checkouts, only to see a stressed staff female telling people to go to the manned checkout - huh ?
I looked across and saw the BSOD on the self-checkouts - the girl was saying 'only cash' - but as I looked one screen came up normal so I waved us in and went and it worked fine - as we walked out another screen came up good so I waved them to that as well
later reading suggested that the first notice of this crash was probably on US servers around 3pm yesterday Sydney time. In which case we saw it about an hour after the start of the SNAFU.
as for heads rolling - the head of Crowdstrike has already apologised and that share price has tanked - as for paying for the damages - uh, guess a few $B in cancelled flights alone - I doubt he'll be opening his wallet for that ;-)
Calling somebody "a staff female" sounds very odd. Just "a female staff member" or "a staff member" works better.
@Mad Max: Google Soma.
Frankly I’m sick of the “cave man” movement. Our ancestors had an average life expectancy of 35 years. They had little in the way of medical knowledge, although trepanning seemed to be rather popular. The main preoccupations were food and shelter. Me, I prefer an intelligent person to a “strong” one. Someone who can push the boundaries.
I don’t think people do want it “easy” nowadays. I think most people take pride in what they do. They want to do their jobs well, they want to help others and they are curious about what the world offers. They aren’t interested in a meagre existence eked out of the immediate landscape. It does seem a tad limiting.
There are different definitions around what is strong and what is weak. It is best not to use them in simplistic forms.