Almost lost a client because of a false positive alert at 2am

I was sitting in my home office in Austin last Tuesday when our SIEM flagged a login from Russia on a server that's been decommissioned for 6 months. Spent an hour trying to trace it before realizing the alert rule never got updated after the migration. Anyone else dealing with alert fatigue from rules that nobody cleans up?

2 comments

2 Comments

barnes.jamie2d ago

About a year ago I had a false positive from a stale rule at 3am for a server that was literally unplugged in a closet. It took me two hours to drive to the office and check it myself before I realized the rule was pointing at an IP that hadn't been used in months. The fix that actually worked for us was setting up a quarterly rule review session with the team where we go through every alert and kill the ones tied to old infrastructure. We put it on the calendar as a recurring thing and it cut our false positives by like 60 percent in the first round. It is a boring task but way better than getting woken up for a ghost server.

susan3502d ago

Eh, 60 percent is a bit generous, I'd say closer to half from our own quarterly cleanups.