Vent: Our SIEM went silent for 6 hours before I caught it

I was sitting in my home office last Thursday around 2 AM doing a late shift review when I noticed the alert count on our dashboard had dropped to ZERO. That's never a good sign, especially for a mid-size company that gets like 200 alerts a day. Turns out the log collector service had crashed silently after a Windows update rebooted the server. I had to manually restart the service and then backfill 6 hours of logs from our backup syslog server. Has anyone else had their monitoring tools go dark like that without any warning?

1 comments

1 Comment

scott.grace1d ago

turned out the log collector service had crashed silently after a Windows update rebooted the server" - I gotta push back a bit here. That Windows update didn't just reboot your server, it rebooted it and you had no alerts? Thats on your monitoring setup, not the update. If your SIEM going dark for 6 hours is something you catch by accident, you've got a gap in how you watch the watchers. Set up a heartbeat check on the SIEM itself or a simple cron job that emails you if the alert count drops below 5 for 10 minutes. Stuff like this is a lesson for the team, not a surprise.