I finally stopped trusting our SIEM alerts after the 3AM false alarm
We spent two weeks chasing a critical alert that turned out to be a dev box sending test packets, and the on-call guy almost drove to the office in his pajamas. I learned to cross-check every alert against the actual source IP before waking anyone up, and now I want to know how many of you have tuned your SIEM to the point where you miss real threats instead?
Man, come on. One false alarm from a dev box is not some big lesson about the whole system, that is just normal noise and you fix the rule for that one source. People acting like missing a real threat is way worse than a 3AM wake up call, and if you tune things too loose you will find out the hard way. Your source IP check is fine, just do not fool yourself into thinking you cracked the code.