Patch Tuesday at 3am in Austin wiped out our SIEM rules

Last month I pushed a Windows update batch during a change window and it silently broke all our Sysmon event IDs. Our SIEM went blind for about 11 hours before anyone noticed, and by then we had 6 compromised endpoints in Dallas that no one flagged. I spent the next 2 days rebuilding detection rules from backups and testing against our sandbox. Has anyone else had a vendor patch nuke their custom parsing logic like this?

1 comments

1 Comment

matthew_baker13d ago

Betting you don't have a config baseline for the Windows event log side either. We got burned the same way once and now we snapshot the Sysmon config and the SIEM parsing regex before every patch wave. Takes 10 minutes to check the diff but saves a whole weekend of chaos. The other thing people sleep on is testing the patch against a single staging host first and checking the actual event IDs still flow through, not just that the service starts. Vendor updates love to change field names or dump extra junk into event data without saying anything in the release notes.