14
The data center meltdown at Gatewood that flipped my backup routine
Last June I was onsite at Gatewood, the old server room off Route 9. A coolant line burst at 2 AM, took out three racks before anyone noticed. Since then I test my backup restores every 90 days, not just the images. Has anyone else had a disaster that forced a new habit?
1 comments
Log in to join the discussion
Log In1 Comment
hayden_rodriguez4d ago
Wait, does anyone else here just mentally block out the month they spent rebuilding a half-corrupted file tree from tapes? Because that was me back in 2017, not a coolant leak but a contractor who "accidentally" ran a disk-wiping script on our live backup NAS. I remember standing in the closet-sized server room at 6 AM, holding a coffee that was mostly sweat, and realizing the last good full backup was from eleven days prior. That's when I started doing what you do, but with a twist: I actually restore to a throwaway VM and open random files to make sure they're not just empty shells. It takes twice as long, but it caught a silent corruption bug that would've bricked our entire archive last year. So yeah, your 90-day rule is smart, but if you wanna feel real pain, try recovering 40k photos from a tape drive that smells like burnt toast and hope the catalog file isn't lying to you.
3