5
Our SOC intern mistook a test push for a real breach, alert fatigue is real
Last Tuesday during our scheduled patching window, our new intern saw a flood of alerts from our test environment and triggered a full incident response. By the time I got there from the gym, he had already called our lead analyst at 2am saying the sky was falling. We spent 45 minutes tracing logs before someone realized the firewall rules were set to "log all" on a dev VLAN. Has anyone else had to rebuild their alert thresholds after a false alarm like this?
2 comments
Log in to join the discussion
Log In2 Comments
lily5114d ago
Our 3am call last month because someone fat-fingered a "deny all" rule in prod instead of a log filter was a similar masterpiece. That intern is now a cautionary tale we tell new hires, complete with a photo of him mid-panic. At least your guy had the excuse of being new, ours was just awake and clicking without coffee.
6
juliam404d ago
Wow, a cautionary tale with a photo too? That's brutal but honestly kind of hilarious, @lily511. I can see it now, that picture probably gets more use than any runbook they've written. You have to wonder how many people have made that same click without coffee mistake.
9