Been twiddling my thumbs for the last 2hrs at work, so i thought I'd do something useful.
Let's hear your stories about how you have been affected by this.
https://www.abc.net.au/news/2024-07-19/technology-shutdown-a…
Major IT Outage Affecting Banks, Media Outlets in Australia and Globally
gomertose on 19/07/2024 - 16:30
Last edited 19/07/2024 - 16:31
Last edited 19/07/2024 - 16:31
Related Stores
Comments
How come OzB still up and running when it happened? maybe on Linux?
Almost all webservers are on Linux
LEMP (Linux, Nginx, MySQL, PHP) is by far the most popular web server stack.
Had zero idea.
I was home all day cleaning and sitting on arse. I was oblivious.
No it wasn’t a zero day attack, it was dodgy upgrade
I wonder how many people's data went missing during the event?
The title of that other thread doesn’t sound related to this issue despite being the same issue.
I might open up a third thread then with an obscure title…
That would be bold of you…
Yep that other thread did not come up as suggested duplicates when I did the post
Do at your own peril.. but if your dead in the water, you don't have much to lose.
"apparently" a work around is boot in safe mode
Goto
windows\system32\drivers\CrowdStrike and look for file matching c-00000291*.sys and rename itthen reboot normally… I guess if it fails, reverse what you changed and wait.
I havent tested or tried it as we dont use CrowdKiller CrowdCrash… but plenty of chat on skype with others i know
That is totally bonkers. What a terrible work around.
What's terrible about it?
Deleting system files in dangerous for eh inexperienced + you're leaving your device open to security vulnerabilities - better to wait for your organisation to patch your device.
@S Baldrick: ….there is no way to get a patch. You never get into windows long enough for a CS patch (instructions to remove file) to come down.
CS agent loads very early in the startup process, which is why you see windows load for a few seconds, on the login screen THEN it bsod seconds after.
You will need to delete that file in order to get windows to boot.
FYI you will be in safe mode (no network connection) so unlikely any new malware will come down while in safe mode. You will only need to be in safe mode for around 1 minute. Then reboot back into normal windows.
But if you are not sure (or unable to BC of bitlocker) get your IT guys to do it.
DO NOT try the options in recovery screen or 'reset windows'. You may lose all data.
@hippo2s: Maybe I got lucky… workstations at my workplace only Bluescreened once, then we were unable to connect to our internal network. Tech literacy isn't the greatest at my place, IT would be pulling their hair out of they heard people trying to delete system files manually.
@S Baldrick: our organisation's patch was for front of house staff / managers to boot cash registers and PCs into safe mode and rename the \Crowdstrike directory \CrowdstrikeOld.
@Antikythera: err… this will get you back into windows. however when you do - machine will be without any endpoint protection. As windows will not load any crowdstrike drivers/your rename hid it essentially. So it will need someone to actually remediate after, so you are not completely out of the woods / CrowdStrike host will probably push down a new install of the agent.
What's bonkers is people still using Skype.
yes this workaround is confirmed but……. if your PC is encrypted by bitlocker, then what?
a world of pain…..
Second temporary boot (windows) drive. Most enterprises issues laptops so installing a second drive may also not be likely. Maybe Windows on a USB stick?
Get into original windows drive and delete file. Shut down. Remove temporary boot, reboot with original drive.
Edit: actually this might not be possible as well. Temporary windows may prompt you for BL key in order to make changes on the original drive.
I believe you would need the recovery key?
Doesn't even have to be bitlocker, we lock plenty of end users out of bios because some are idiots are brick things that we are 2 weeks postage away from.
If it's locked out or inaccessible, rip to those IT peeps recovering that shit.
Delete that file, no need to rename it.
I heard it was boot into safe mode and delete
C:\windows\System32\ folder in its entirety.Another was delete
C:\Windows\Worked for me :D
Just turn it off and back on
Turn off now and back on again tomorrow.
But is KFC okay? I'm hungry
It has caused slightly dry chicken
Oh wow, still yum
According to reddit, no :(
"Someone didn't say KFC"
:(
Kernel
File
Corrupted
App says "did someone say global IT outage?"
Edit: my local is closed
The KFC I went to was definitely not ok. I think it took them nearly two hours to figure out "Let's just start accepting cash orders".
Is Skynet Microsoft?
yeah, grrrr! my card kept getting declined at woolworths this afternoon. they wouldn't accept my hand-written IOU, so i left empty handed
some money transfers gone AWOL
we're screwed if we go cashless
if the instore scanners are affected cash won't save you.
Just got home from coles, it was cash only.
Thanks for update.
@try2bhelpful: Just went to 7-eleven for a coffee. Cash only and they were writing all purchases into a book with a pen, with a pen!
Got my wife a burrito from gyg, PayPal and app worked as per normal.
@mapax: That is a tad bonkers isn’t it. We need to go back to the card imprints for backup. We went to one place where the imprint machine wasn’t working so the guy used his stapler to rub the imprint with the carbon. Now that is a backup system.
My Coles couldn't take cash, only physical cards from banks that hadn't gone down.
My weird neg stalker is at it again.
Fixed
Thanks.
Must have moved on from me. I don't feel as validated without that weirdo little neg vote.
It is just weird. It isn’t like they are negging something that is controversial.
@try2bhelpful: Yeah I picked up on it because it was the absolute blandest comments. Guess I got to be someones arch enemy for a while, funny as.
@try2bhelpful: I have noticed a few strange negs around. I wonder if there are people who scroll around accidentally negging or maybe they just want to watch the world burn so they just randomly neg comments. There are definitely some people who are a bit too invested in some discussions though and will go around trying to aggressively shut down anyone who doesn't agree with them and I do feel like they're the type who would be happy to neg stalk people for at least a few months after if that person happened to still voice their opinion.
@Miss B: The people living under bridges. We do seem to have a few on this site. Too cowardly to make a comment because their names would be exposed. I don’t mind a neg when I make a controversial comment, because I don’t expect everyone to agree with me, but the random ones are just odd.
I’ve been a PM, and technical implementer, for a couple of major organisations and our mantra for changes was don’t appear on the front page of the newspapers. These guys, certainly, failed the test.
When were you prime minister?
You stole my lame joke…
Mine too
The country would be much better if I was Prime Minister :)
Go for it!
I'm old enough to remember a PM who actually got elected with a promise that "I'll put sport back on the front page of the newspaper" (ie no more political scandals or controversies).
Ours was not to appear on ACA as the lead article!
Any project teams I've worked on the last few years, I am certain at least 50% of participants are blatantly lying about task completion, especially OS sourced IT. Recently most stuff doesn't even have testing as part of the plan, when I ask about it I get treated weird , probably because they don't plan on even attempting the main objectives when they can just lie and say it's been done and bury under email trail.
100% they are full of shit.
We have a small segment of our IT outsourced because the patent company is international.
If you escalate certain things to these guys they'll ask you to recreate the case to reset all the KPIs lol.
Like no man, if we spent hours researching and testing we aren't resetting shit.
Which is exactly why I kept KPIs, way too many ways to manipulate things to look good.
This is catastrophic, not only does the infinite BSOD doesn't even boot the PCs up so a widespread fix cant be automatically deployed, if the affected machines also use Bitlocker; and if those bitlocker encryption keys are also on a server which itself is itself is in infinite BSOD..
Also, any fix would be a device-by-device manual fix? That sounds catastrophic to me for companies with hundreds of thousands of endpoints that might be scattered around the country or world. This sounds absolutely catastrophic. least the memes are funny
Yup
If it’s bsod and you can’t boot safe mode you’re screwed
Also if it’s happening on servers around the world most of them need a physical reboot rather than a code patch
Current fix is apparently a Safe Mode boot and deleting the offending CrowdStrike .sys file, so it seems safe mode booting is still 'safe'
Servers should have appropriate headless management systems in place that should still function… unless the management servers are inaccessible!
yes but… if encrypted by bitlocker which itself is a fairly standard practice for corporate PCs - you can't access the C: without encryption keys.. even with those, it's still an incredibly manual fix for each individual endpoint?!
a lot of servers are now VMs on the cloud, and they don't offer a KVM-like system.
so workaround is to detach the disk and attach to a working VM and remove the file and attach back and start.
10-15min job but if you have 1000s of servers, yikes.@greyeye: At least a cloud VM means you only have to use one keyboard to fix
I feel sorry for the plebs having to put hand on every single client that needs fixing like the Woolies self serve machines, or the airport display screens mounted ten feet in the air that should not need to be physically touched to fix 'normally'.
…and then imagine the problems that inevitably come with years of dust and grime, or loose cables after the patch is applied causing the display to blank or ethernet to drop out, yikes!
@greyeye: If you have 1000 servers on cloud VMs, you write a small script, right?
The problem all the physical PCs running POS etc, and nobody onsite knows what Bitlocker is, let alone has the key.
If a company stores the bitlocker key of a server holding bitlocker keys in itself… they're total idiots. Having it unrecoverable would be just a matter of time
I'll bet we are about to discover how many total idiots of companies there are though …
Hasn't affected me whatsoever yet but damn it seems like a massive headache.
Kinda shit it happened at end of work week in my office job. Would have been nice if it was a Monday and we all got told to just go home/chill out for a few days.
I got an email from our IT department explaining that we weren't affected by it.
We didn't use Crowdstrike until we were bought out… luckily our VMs are still safe from the buyout but the employees MDM laptops weren't.
Also, they should've followed rule #1 for change implementation - "Read Only Friday".. although that wouldn't have stopped the issue, it would've stopped it on a Friday afternoon for AUS
Hopefully this will put another nail in the coffin of all the digital id and cashless society plans bs.
Negs go your hardest 🤣I'm so sick of the government going on about that. We have a really terrible track record of data breaches in our publicly listed companies, what makes the government think they can do a better job especially with all the bureaucracy and red tape inherent in government?
If they're going to go with a digital ID and cashless system they should make it an opt-in system and I'll happily opt out.
Gov designed websites/apps/forms/UX and poor security should be enough to halt any digital ID. Does anyone trust them at all with IT?
I definitely opted out of the my health record fiasco for this reason a number of years ago.Does myGov allow authenticator apps yet or are they still only offering SMS for 2FA?
I personally am very skeptical of them being able to achieve anything remotely difficult at all.
myGov allows three different 2FAs; a secret question (which I think might be feature carried over from before the platform was updated), SMS and a proprietary Code Generator app (on iOS at least). I haven't tried the app but you have to be very careful with it because it's been designed so that if it's accidentally deleted (and if you account for the lowest common denominator there is a 100% chance someone out there will eventually do this) or if you switch devices you can lock yourself out of your myGov account so you have to make sure you de-authorise it properly. I'm not sure how difficult it is to recover your account if you do either of those things, let's hope it's easy.
they should make it an opt-in system and I'll happily opt out.
Sure. They will. But if you don't "opt in" you will not be able to have a bank account, work, drive a car, go on holidays overseas, go to a doctor, have a job, ……
It will be your choice to make……
They have already rehearsed that type of free choice before.
I'm sorry but everything about modern life requires computer systems to function as intended to some degree.
The only way to avoid that is to go live in the medieval ages - live on a farm, where you farm your own vegetables and livestock (tractors have software too, supermarkets has software too), hold gold coins or batter with your neighbours (currency/bank uses software too), send your ozbargain comments via carrier pigeons (mail uses software). Maybe do your taxes with abacus, ride horse driven carriage too?
Where do you draw the line?
Things have been working ok so far without digital id and with cash alongside electronic transactions.
If it aint broke don't fix it.I mean I'm sure that's what they said about using gold coins and travelling uses horses too…
Things were working fine before air travel
Things were working fine before sea travel
Things were working fine sitting around camp fires and living in caves….
Just admit you're Abe Simpson scared of change.@Typical16-bitEnjoyer: I'd rather be living in caves and sit around camp fires than be wired in, in The Matrix.
@Mad Max: In High School I studied both 1984 and Brave New World. I decided I would prefer Brave New World. Give me comfort and SOMA.
@try2bhelpful: Everyone wants everything easy without effort these days. Government makes all decisions and people just follow blindly. No thinking, no checking, no scrutinising, no questioning. Easy life. Just do as you are told and you'll be right.
Hard times create strong men. Strong men create good times. Good times create weak men. And, weak men create hard times
I believe we are between the last two stages.
No idea what SOMA means…
Was at Kmart picking up photos, right as their system went down. Luckily pre-paid online and was able to collect and walk out.
Went to Woolworths after, all their self-checkout screen was showing Recovery Mode/BSoD.