My laptop run into recovery mode approx 30 minutes ago. My friends reported the same. My colleagues in Singapore, China, Hong Kong also reported the same issue.Anyone know what happened? Crowdstrike was hacked?
Computer Run into Recovery Mode
gstfree on 19/07/2024 - 16:10
Related Stores
Comments
- 1
- 2
Got complacent with their testing and release.
Probably had an vital fix they needed to push out therefore didn't do enough UAT (something this bad should have been caught in UAT before it gets to release) and pushed out the release update without segmentation.
Dropped the ball for this release. Nothing nefarious as suggested IMO. They can't afford to be nefarious, they have too many gov and critical infrastructure enterprises as clients, it's be committing business suicide.
Speaking from experience of having worked for Symantec for 10 years in the support and process improvement area.
Speaking from experience of having worked for Symantec for 10 years in the support and process improvement area.
You know how it is then.
I worked on process improvement and you'd be surprised how many people don't spot the extra steps they are doing in the process for no benefit and the amount of people who just do their job and fly under the covers of policy that is obviously deficient.
Edit: they will probably pin it one some poor employee until upper management fronts up and have to quit
@netjock: Yes I know what you mean, especially with the push to be lean, automate. Removing 'extra steps' for sake of 'efficiency'. It does cause problem when the planets align ;)
Customer and people working on release become complacent as the process worked every day the last X years. Not the first time, not the last time this will happen.
It does point to a need to use multiple vendors for critical systems instead of, again…for sake of efficiency, buy more from the same one vendor to save costs and maintenance upkeep costs.
Exactly. Someone thought, “This is too important and should be low-risk, so let’s skip UAT/CAB and just push it out. What could go wrong?” Even the smallest change can kill your system, hence why it’s important to treat every single one with the same rigour before releasing.
@Chazzozz: I am sure they would do UAT for major versions of windows prior to release. We were doing that a decade ago!
The only reason it would have gotten to release is that it was skipped, missed or ignored (on purpose or not).
'She'll be right….'
@hippo2s: Been there, done that, carry the scars of lost weekends and all-nighters…
I've learnt the hard way that shortcuts only end up in making more work for yourself. I think CrowdStrike have learned that lesson, too.
Maybe I should drop my ex-Manager a line for his viewpoint on this. /s
One of my duties was Windows patch deployment and I'd dropped the delay from release to Prod from about a month to a week.
First, test on suite of laptops and VMs representing typical business unit builds, checking for boot loops, noting error messages and such.
Second, pilot to about 0.5% of fleet who were recruited for their willingness to report issues.
Third, pilot to about 1% of fleet across wider business roles.
Each phase at least 24hrs apart with reboots.
Prod deployment around one week from release day.Manager spouted "infosec best practice" and pushed for pilot to "all of IT" globally and if no issues by end of day, deploy to entire fleet.
All so he could boast to the C-suite how he'd revolutionised our security standards.
When he left, that was exactly how he was described. No mention of drone who pushed back and got spiteful performance reviews as a result.Many in patch management know of several occasions when Microsoft released a bad patch that boot-looped and required 2 further updates to resolve.
Behind every worker who'll be blamed for this is a manager shooting for glory.
LOL yeah it is great idea until things go wrong. All that time saved goes down the drain.
Yeah I knew of a manager that was like that. At some point you run out of luck.
You have no idea what you are talking about. Again you use some words that is not the typical definition. This was not a malware attack, Crowd strike is not a malware.
It's just a Crowdstrike agent file update that caused a BSOD in Windows boot up process. Deleting the file means that file is not loaded by windows, and because it is not loaded it does not BSOD. Nothing to do with a malware. The same could happen with ANY windows driver file by every company that makes the parts of your software and hardware.
Eg. Logitech could also release a bad driver for your mouse that cause BSOD or NVIDIA release a bad driver for their video card and cause the same.
so.. for people with work laptop (locked hard by IT of course cant access bios, lots settings are under admin users only) and currently having blue screen, restarting 10x doesnt fix it. what will be the most likely scenario that will happen? how someone can delete that agent file if i cant even go into windows? will this something can be done remotely? or need to be plugged in to company network by cable? or ???
If you can get into Safe Mode to get Windows loaded, you can delete the files just with a command prompt. However many corporate PCs have that locked as well, and a technician with physical access (and bitlocker key) would be required to resuscitate the computer.
Which is why it's going to take a long time for companies with large Windows fleet.
Your best option is to contact your I.T. people and see what plan they have in place. Most companies are likely doing one of two things:
- Asking staff to bring their computer to the I.T. people so it can be fixed in person.
- Contacting remote workers by phone and walking them through the manual fix, which will involves reading out the Bitlocker key to them. (This will also require generating a new BL key afterwards to maintain security, but I.T. can do that part.)
It's a frustrating situation that's made worse by the fact the majority of the affected computers cannot be managed remotely because they are unable to get into a running state where networking makes them visible. It's also placing some of the burden on possibly non-technical people to actually perform the fix, but that's a reality we cannot escape.
Please do not go into the BIOS. Nothing to do with that.
You'd have to hack bitlocker (of which it's intended purpose is to prevent accessing the files in the first place) so the best solution is to get IT guys to unlocker Bitlocker and safe mode, delete driver file, no data loss this way.
The other choice: new hard drive (yes you can't even pull the HDD and use it on a different machine), new install of windows on same HDD but this result in complete data loss. (Actually you might not even be able to do this).
The other other alternative choice. Would be to install a separate hard drive to boot up non affected Windows, then access the original drive, delete driver file, then switch boot back to original drive and remove the temporary boot drive. This may not even work, from memory the temporary windows will prompt you for BL key in order to make changes on the original drive.
@hippo2s
Hey yo. How can I help? As Scotty have said. Most enterprise installs they also turn on BitLocker, which you need to get into safe mode. This is the reason pretty much all machines need hands on by IT guy to fix.
I would be willing to bet quite a number of IT guys are having to learn about BitLocker keys and how to get them!
Latest advice is to keep letting it BSOD, restart, BSOD, select to restart in recovery, and do this loop until it finally stays on login screen (no more bsod).
If you have a ethernet port/USB dongle it should be faster than on wifi.
Heaps of reports this has worked. One colleague calculated it rebooted sixty something times till it stopped BSODing.
Good luck all.
- 1
- 2
Odd is they didn't release it by batch (say 10k units) to see if anything goes wrong. Or they could have released it internally to their own systems first to see.
It is a big fail is nobody picked up tens of thousands of falcon sensors are going offline through crashes because being cloud security those PCs would be reporting back their behavior.
It isn't a cyber attack but has similar impacts on daily life. That is also why even with iOS / Android updates I look at roll out dates and give it a few days before updating.