They were just in the same disks as the data. And because everything is abstracted two times into virtual disks on virtual machines, and containers and volumes, the people responsible for the backups didn't even know it.

[–] Winter_Oven@piefed.social 16 points 2 days ago* (last edited 2 days ago) (1 children)

But won't you like...check? That the backups are own their own drive? The whole 3-2-1 rule kinda make you want to check this, no?

Or was it like they knew where the drives of the backups were, but they didn't know those drives were being virtualized away and were in like production use?

[–] marcos@lemmy.world 14 points 2 days ago* (last edited 2 days ago)

I dunno what possibilities they actually had. But knowing the place, I can fully believe both that they weren't allowed to check and that they never bothered.

The most likely scenario in my head was that they sent a request to the provisioning team asking for the volume to be in a different disk, and that detail never made into the technician actually doing the work (that sits on the next chair, but the requests have to come from the system).

(And the long term backups were fine. We lost 3 days of data.)

[–] death_to_carrots@feddit.org 8 points 2 days ago

If everything is serverless, hardware can't fail!

[–] falseWhite@lemmy.world 6 points 2 days ago

Maybe they stored backups on-site