Been twiddling my thumbs for the last 2hrs at work, so i thought I'd do something useful.
Let's hear your stories about how you have been affected by this.
https://www.abc.net.au/news/2024-07-19/technology-shutdown-a…
Major IT Outage Affecting Banks, Media Outlets in Australia and Globally
gomertose on 19/07/2024 - 16:30
Last edited 19/07/2024 - 16:31
Last edited 19/07/2024 - 16:31
Comments
Had zero idea.
The title of that other thread doesn’t sound related to this issue despite being the same issue.
I might open up a third thread then with an obscure title…
That would be bold of you…
Yep that other thread did not come up as suggested duplicates when I did the post
We'll forgive you this time.
Do at your own peril.. but if your dead in the water, you don't have much to lose.
"apparently" a work around is boot in safe mode
Goto
windows\system32\drivers\CrowdStrike and look for file matching c-00000291*.sys and rename itthen reboot normally… I guess if it fails, reverse what you changed and wait.
I havent tested or tried it as we dont use CrowdKiller CrowdCrash… but plenty of chat on skype with others i know
That is totally bonkers. What a terrible work around.
What's terrible about it?
yes this workaround is confirmed but……. if your PC is encrypted by bitlocker, then what?
Just turn it off and back on
Turn off now and back on again tomorrow.
But is KFC okay? I'm hungry
It has caused slightly dry chicken
Oh wow, still yum
According to reddit, no :(
"Someone didn't say KFC"
:(
App says "did someone say global IT outage?"
Edit: my local is closed
The KFC I went to was definitely not ok. I think it took them nearly two hours to figure out "Let's just start accepting cash orders".
Is Skynet Microsoft?
yeah, grrrr! my card kept getting declined at woolworths this afternoon. they wouldn't accept my hand-written IOU, so i left empty handed
some money transfers gone AWOL
we're screwed if we go cashless
if the instore scanners are affected cash won't save you.
Just got home from coles, it was cash only.
Thanks for update.
My weird neg stalker is at it again.
Fixed
Thanks.
Must have moved on from me. I don't feel as validated without that weirdo little neg vote.
It is just weird. It isn’t like they are negging something that is controversial.
@try2bhelpful: Yeah I picked up on it because it was the absolute blandest comments. Guess I got to be someones arch enemy for a while, funny as.
I’ve been a PM, and technical implementer, for a couple of major organisations and our mantra for changes was don’t appear on the front page of the newspapers. These guys, certainly, failed the test.
When were you prime minister?
You stole my lame joke…
The country would be much better if I was Prime Minister :)
Microshaft winblows gonna get a hurt real bad.
This is catastrophic, not only does the infinite BSOD doesn't even boot the PCs up so a widespread fix cant be automatically deployed, if the affected machines also use Bitlocker; and if those bitlocker encryption keys are also on a server which itself is itself is in infinite BSOD..
Also, any fix would be a device-by-device manual fix? That sounds catastrophic to me for companies with hundreds of thousands of endpoints that might be scattered around the country or world. This sounds absolutely catastrophic. least the memes are funny
Yup
If it’s bsod and you can’t boot safe mode you’re screwed
Also if it’s happening on servers around the world most of them need a physical reboot rather than a code patch
Current fix is apparently a Safe Mode boot and deleting the offending CrowdStrike .sys file, so it seems safe mode booting is still 'safe'
Servers should have appropriate headless management systems in place that should still function… unless the management servers are inaccessible!
yes but… if encrypted by bitlocker which itself is a fairly standard practice for corporate PCs - you can't access the C: without encryption keys.. even with those, it's still an incredibly manual fix for each individual endpoint?!
If a company stores the bitlocker key of a server holding bitlocker keys in itself… they're total idiots. Having it unrecoverable would be just a matter of time
Hasn't affected me whatsoever yet but damn it seems like a massive headache.
Kinda shit it happened at end of work week in my office job. Would have been nice if it was a Monday and we all got told to just go home/chill out for a few days.
I got an email from our IT department explaining that we weren't affected by it.
We didn't use Crowdstrike until we were bought out… luckily our VMs are still safe from the buyout but the employees MDM laptops weren't.
Also, they should've followed rule #1 for change implementation - "Read Only Friday".. although that wouldn't have stopped the issue, it would've stopped it on a Friday afternoon for AUS
Hopefully this will put another nail in the coffin of all the digital id and cashless society plans bs.
Negs go your hardest 🤣I'm so sick of the government going on about that. We have a really terrible track record of data breaches in our publicly listed companies, what makes the government think they can do a better job especially with all the bureaucracy and red tape inherent in government?
If they're going to go with a digital ID and cashless system they should make it an opt-in system and I'll happily opt out.
Gov designed websites/apps/forms/UX and poor security should be enough to halt any digital ID. Does anyone trust them at all with IT?
I definitely opted out of the my health record fiasco for this reason a number of years ago.Does myGov allow authenticator apps yet or are they still only offering SMS for 2FA?
I personally am very skeptical of them being able to achieve anything remotely difficult at all.
myGov allows three different 2FAs; a secret question (which I think might be feature carried over from before the platform was updated), SMS and a proprietary Code Generator app (on iOS at least). I haven't tried the app but you have to be very careful with it because it's been designed so that if it's accidentally deleted (and if you account for the lowest common denominator there is a 100% chance someone out there will eventually do this) or if you switch devices you can lock yourself out of your myGov account so you have to make sure you de-authorise it properly. I'm not sure how difficult it is to recover your account if you do either of those things, let's hope it's easy.
they should make it an opt-in system and I'll happily opt out.
Sure. They will. But if you don't "opt in" you will not be able to have a bank account, work, drive a car, go on holidays overseas, go to a doctor, have a job, ……
It will be your choice to make……
They have already rehearsed that type of free choice before.
I'm sorry but everything about modern life requires computer systems to function as intended to some degree.
The only way to avoid that is to go live in the medieval ages - live on a farm, where you farm your own vegetables and livestock (tractors have software too, supermarkets has software too), hold gold coins or batter with your neighbours (currency/bank uses software too), send your ozbargain comments via carrier pigeons (mail uses software). Maybe do your taxes with abacus, ride horse driven carriage too?
Where do you draw the line?
Things have been working ok so far without digital id and with cash alongside electronic transactions.
If it aint broke don't fix it.I mean I'm sure that's what they said about using gold coins and travelling uses horses too…
I'm sorry but everything about modern life requires computer systems to function as intended to some degree.
And fossil fuels… which are finite.
I was in the drive-thru at KFC at what seemed to be moments before the issue happened. By that point I had already specified I was picking up my order (which was only for the $2 mousse) in drive-thru and had the order number ready to quote. By the time I got to the window they said the system crashed, I didn't think much of it at the time so went and waited for about 15 minutes. Came back and was told by the manager the outage was a bit more serious. I told them I had already put the order through on the app and had the order number on my phone and asked if I could just get the mousse because surely it would've gone through already if I got the order number confirmation but was told to come back later.
So went and did some shopping, came back after 1.5 hours and the drive-thru line was closed off, went in store and saw they were doing everything manually and people were paying by cash. Waited in line told the staff about the issue again, a different manager came out and I told him the same story "I was in drive-thru, clicked to pick up what seemed to be moments before the crash, my order number is ####, can I just take the mousse? Won't the order show up on your end because I already specified I'd be picking it up in drive-thru?" Manager said yes it should show up on our end when the system comes back up, the staff wrote down my order number and I got the mousse. Talk about a pain in the arse to get the $2 mousse. It better be a damn good mousse.
In the past 2-3 years we seem to be having a mass outage on average once a year or something. This one seems really serious because it's affected services globally. I thought our systems were supposed to be robust 🤷♂️ I was listening to the radio and the host said that the issue seems to have occurred because of a software update which I found funny because it seems to me now that every time something like this happens the go-to excuse is that it was a "software update issue". Absolutely zero transparency whatsoever to gaslight the public, just like when Optus was hacked and the CEO said "It's a sophisticated attack, the hacker is changing their IP constantly and they're located overseas" as if we live in a Hollywood movie because the reality was that the data was accessed via an unprotected API.
Whatever happened today happened because someone(s) didn't do their job correctly. Maybe it was a once-off massive screw up or maybe it was a series of minor mistakes along the way that have led to a complete catastrophe. Maybe whoever was working wasn't trained up properly or couldn't be bothered doing their job correctly or maybe someone who had the knowledge of how to perform the job properly was laid off and took that tacit knowledge with them and didn't document it (just like what's happening at Boeing).
Verdict on mousse?
Haven't had it yet but I'll report back once I have.
That's a well earned mouse
Crowdstrike shares down 13.5% in pre-market lol.
I don't see them surviving the lawsuits.
That's fair. This seems like a huge stuff up. On top of your comment above I've read that to reboot systems people need to take a USB to them manually? This is huge.
If it is stuck in an infinite BSOD loop that doesn't even boot into networking, then yes some kind of physical intervention for every single device in that state is needed. It is huge. I'm already reading some companies have 300,000 endpoints, which will include PC's, terminals, hand held devices, etc.
It is possible not all devices are stuck in this loop, but my entire company's employee laptops certainly are.
@lawyerz: That's what was happening at the KFC I visited. The screens were showing a continuous boot cycle into a BSOD lol, at the login screen there was just a spinning circle then it rebooted. Holy shit this is insane. All those KFCs that need to be visited… and that's just KFC.
Someone on reddit said they work at an ED and it's chaos and others have said their hospital is down and surgeries have been cancelled. Wow.
Feels like we live in a third world country.
Priority should be to restore critical infrastructure such as banks and payment systems first.
Sometimes it makes me wonder where contractor's priorities are and whether they are just taking on easier jobs because they don't want to be responsible. If everything crowdstrike is telling the world is correct, then it is an easy fix if your systems have a good restoration protocol, but I have this feeling crowdstrike are lying to us somehow because of how long the response time is.
How do we know the crowdstrike malware has not altered any files in the meantime? Integrity of data is an issue.
Ever lived in a third world country? All they have is cash. Speaking from first had experience.
More than 5 hours at Cairns airport (including one hour spent on the plane sitting on the tarmac) waiting to fly back home to Melbourne then told flight has been cancelled. Staff suggested finding accomodation in Cairns until they can figure something out!!!!! Even though Melb is cold I’d rather be home!
Trump's revenge for all the jokes about wishing he died.
Absolute rubbish company - who will deploy at the same time something untested on millions of computers.
After 5 pm today, I wanted to send money to my Wise account but Upbank app notified about Osko/PayID payments are unavailable and also traditional bank transfers (via BSB/Account number) are unavailable. So, I cannot use Upbank to send money. Then I tried to send money through St. George bank app via BSB/account number and it went through, but the money didn't arrive until few hours later … At least the money has arrived. I read the news about this after I send the money …
Many times I informed in Ozb about keeping some cash just in case you cannot use your card or network problem (visa/MasterCard/bank problem), but some or many people here don't want/like to keep some cash …
And how are you going to use cash when you can't even scan items through the checkout?
Did you read the comments above me? Mapax said "Just got home from coles, It was cash only"
Ghost47: "The KFC I went to was definitely not ok. I think it took them nearly two hours to figure out "Let's just start accepting cash orders".
If you think cash is like insurance, you don't need it until you really need it … Have some cash as a backup at least …
For all Cash deniers, here are proofs that everyone need to have Cash at least as a backup (news in Australia) :
https://www.9news.com.au/national/eftpos-outage-woolies-cust…
https://www.abc.net.au/news/2023-06-26/commbank-commonwealth…
https://www.9news.com.au/national/mcdonalds-reports-nationwi…
https://7news.com.au/news/nab-customers-hit-by-widespread-ou…
https://www.abc.net.au/news/2024-03-04/australians-lose-one-…
https://www.abc.net.au/news/2023-12-30/will-consumers-have-t…
https://7news.com.au/sunrise/sunrise-star-monique-wright-exp…
I hardly use cash anymore but I still think it's silly to move completely cashless. Even one outage a year is too many in a cashless society, and I'm pretty sure we've had outages at least once the past two or three years where people have been unable to pay for things.
Outages is only a matter of time, not IF. Imagine natural disaster like flooding, no electricity, undersea internet cables broken that affect global Internet outage, etc. Microsoft may not have good quality control/assurance for Windows Update. How many times already Windows Update problems, lol. I read a lot of news …
Yeah that's a fair point. I was actually wondering if this outage was due to a data centre going up in flames or getting flooded somewhere.
@Ghost47: Wonder why maybe no lawsuits for bad Windows Update? If only happened once, people still think okay … But how many times already bad Windows Update? Really bad effects for businesses and also affect consumers too.
It's all very good to have an option to pay with cash, but how much cash do you actually have? (Rhetorical, don't want hear about the stash 😉)
When all the digital goes to hell, you're not getting anything out of an ATM (especially if you're not one of the first to get to it)
What's the point of cash when the point of sale systems are down. Unless they're small stores, woolies and Coles aren't going to take out a calculator and start giving you hand written receipts.
Not against cash, and not pro cashless. Just saying if the checkout machine itself is down, and it controls inventory management as well, cash won't help
when did woolworths update their windows 95 checkouts? lol
No joke, what a nightmare this situation is.The downscaling and consolidation of companies has its repercussions. It is now shown to be true, having all your eggs in one basket is not a good idea.
Was at Kmart picking up photos, right as their system went down. Luckily pre-paid online and was able to collect and walk out.
Went to Woolworths after, all their self-checkout screen was showing Recovery Mode/BSoD.