Misc How do you store your game backups and protect against data rot?

Nynrah · Sep 24, 2022

Over the years I've built up quite the game library for my many consoles and made sure to back everything I got up. Cartridges, discs, digital games, you name it. By now I've reached a point where my entire collection is a good 3 TB large. So far I've just stored my backups on an external HDD. However, after one of my HDDs died on me (fortunately not one with backups on it!), I began thinking about how I can ensure my backups will be protected and it's like going down the rabbit hole.

I first thought I just had to worry about disk failure and that redundancy would fix the issue, but apparantly data rot is a thing as well. The idea that a backup can silently become corrupted without being aware of it worries me. Apparantly hashes and error correcting codes would be a good measure on top of redundancy to ensure the integrity of the backups. And that doesn't even cover protecting the metadata of the file tree and files themselves or the extra storage space needed for the error correction codes.

For those of you who also own large game collections, how do you guarantee the safety and integrity of your backups? What's your process or protocol (so to speak)? What mediums and tools do you use?

I'd definately like to hear some thoughts on this as I'm trying to figure out what I should do. Right now my thoughts are along the lines of storing two copies of each backup on separate HDDs, generate a hash for each file so I can periodically check if my backups are not corrupt and to generate error correction codes that can help repair my files if neccesary.

Sypherone · Sep 24, 2022

Use a journaling filesystem like ntfs or ext4 ( uses checksums) wich can correct automaticalky errors like filesystem and sector errors. They have extra features to keep the files and filesystem in correct state. For exampke Btrfs has some more festures to prevent file errors.

To use error correction code, archives like zip,7zip support them automatically, thats why this archives can be repaired. There exists also QuickPar to generate Par2 parity files for your data.

Then i have one normal drive for regular use and one Backup drive that i dont really touch, only for backup reasons. The secured would be a RAID 5 wich gives redundancy and create correction code of the hole drive. But at least you would need 3 drives only for backup.

Maybe Veam, i was going to check out (there is also a free version), can also provide some extra features.

Hayato213 · Sep 24, 2022

In term of files backup, you can do one or more HDD for redundancy, in addition you could use a NAS with like raid 5/raid 6.

mrdude · Sep 24, 2022

If your hdd died, is it the actual disc having bad sectors on it, or is it the hdd motherboard? If it's the motherboard you can most likely swap it out with another of the same type of drive if you can't find out what the bad component is. Usually I have a few drives of the same make for this kind of thing happening.

tech3475 · Sep 24, 2022

Currently I have software do monthly hash checks on my unraid server (which offers parity protection, which is also checked monthly) and have backups made using Macrium reflect to a different system.

RAHelllord · Sep 24, 2022

Your backup plan should always include redundancies, the only proper way to protect data with near certainty is to have another copy of it somewhere else.

Even if you have 5 full copies of the data at home, your home burning down while you're out for groceries would mean potentially losing all 5 at once. As such, off-shore backups would be the way to go, or literally just making a copy and leaving it with someone else if you don't want to pay another company to keep it safe for you.

But if you want to be realistic, have at least one additional backup of everything somewhere safe, preferably a place that's dry, dark, and at least somewhat climate controlled, and maybe if you have a friend you trust give them another copy to keep safe for you. Check all backups regularly, like once per year at least, and try to restore a random selection of files to verify they still work properly.

AncientBoi · Sep 24, 2022

I send mine to the clouds.

tech3475 · Sep 24, 2022

RAHelllord said:
Your backup plan should always include redundancies, the only proper way to protect data with near certainty is to have another copy of it somewhere else.

Even if you have 5 full copies of the data at home, your home burning down while you're out for groceries would mean potentially losing all 5 at once. As such, off-shore backups would be the way to go, or literally just making a copy and leaving it with someone else if you don't want to pay another company to keep it safe for you.

But if you want to be realistic, have at least one additional backup of everything somewhere safe, preferably a place that's dry, dark, and at least somewhat climate controlled, and maybe if you have a friend you trust give them another copy to keep safe for you. Check all backups regularly, like once per year at least, and try to restore a random selection of files to verify they still work properly.

If you go offsite, I would suggest encryption with a custom key securely stored elsewhere.

Personally, I have things like family photos backed up online (encrypted and stored on Office 365/One Drive using Arq) and most other files backed up to a server at my parents house.

Previously used Crashplan but that was unreliable and slow, other services may be better but didn’t support unraid and/or the costs added up per TB.

Nynrah · Sep 24, 2022

Probably good to add, but for my own situation I'm looking at the best way to implement these measures using two external HDDs that I access from a Windows computer. That means no NAS or RAID setups - just cold storage. Part of the reason being that it's not storage I need to access often, so it didn't make sense for me to have something that's running 24/7. Granted, there's also the financial aspect.

Sypherone said:
Use a journaling filesystem like ntfs or ext4 ( uses checksums) wich can correct automaticalky errors like filesystem and sector errors. They have extra features to keep the files and filesystem in correct state. For exampke Btrfs has some more festures to prevent file errors.

To use error correction code, archives like zip,7zip support them automatically, thats why this archives can be repaired. There exists also QuickPar to generate Par2 parity files for your data.

Then i have one normal drive for regular use and one Backup drive that i dont really touch, only for backup reasons. The secured would be a RAID 5 wich gives redundancy and create correction code of the hole drive. But at least you would need 3 drives only for backup.

Maybe Veam, i was going to check out (there is also a free version), can also provide some extra features.

I heard that DAR and zip deflate are good archive formats to use compared to solid archives. DAR sounded quite enticing with native PAR2 error correction, but I've used 7zip before, so that feels more familiar. So let's say I use 7zip with the deflate algorythm, how do I apply the error correction to it?

Also, I'm considering whether to pack each game backup into its own zip archive or pack the entire disc into one big zip archive. The reason for that is because if things crash and metadata like the file tree is damaged, I'd be in for a merry time scraping a ton of files. If there's only one big file, file scraping wouldn't be so bothersome as that single file basically contains the whole file system on the HDD. What I wonder in this case, though, is whether the error correction functionality is incremental if adding new files to the archive and how generating and verifying hashes would work in this case.

Sypherone · Sep 24, 2022

The CRC correction is in 7zip and zip already nativ in the archiv header implemented and calculated when the archiv is created. By changing the archiv content it will be automatically new calculated.

I would consider to pack every game in one single archiv. E.G. by a lot of files wich are going in to the same folder, i would write a short bash/ Powershell commandline script.
If you create one big archiv and it gets really bad damaged, all is gone. Done know if backing up the header of the archiv is possible.

SylverReZ · Dec 3, 2022

I love backing up my physical media and that this has been my passion for so many years thanks to the Redump project. It's always important to not only have all of your games backed up but also other important stuff you may have.

Nynrah · Dec 4, 2022

M4x1mumReZ said:
I love backing up my physical media and that this has been my passion for so many years thanks to the Redump project. It's always important to not only have all of your games backed up but also other important stuff you may have.

How does Redump factor into this?

I settled for using multiple HDDs with each game file being hashed so I can periodically verify file integrity. I didn't add error correction codes because I just couldn't get a good idea of how to go about it. Things like how much ECC data to generate per block of file data. Given the size of my library, the ECCs would inevitably cost lots of space as well. Since I couldn't determine what's optimal both in practice and financially, I settled for triple redundancy on separate HDDs with hashes.

SylverReZ · Dec 4, 2022

Nynrah said:
How does Redump factor into this?

Helped me made accurate disc backups using their guides. I'm not referring to HDDs but rather physical media such as CDs, DVDs, you name it.

Nynrah · Dec 4, 2022

M4x1mumReZ said:
Helped me made accurate disc backups using their guides. I'm not referring to HDDs but rather physical media such as CDs, DVDs, you name it.

Ah, I get what you mean now. I did consider using Bluray discs, but that didn't seem like such a hot idea when picturing how many discs I'd need for 3 TB worth of games.

Marc_LFD · Dec 5, 2022

I have game console backups stored in TB HDDs, but yeah, they could eventually fail. And honestly, I'm okay with that as I'm not that much into gaming too much.

I even RAR'd PS2 games which saved an incredible amount of space although GCN/Wii/PS3/PS4 it really didn't do much of a difference. :-/

Misc How do you store your game backups and protect against data rot?

Well-Known Member

Gaming Ninja

Newcomer

Developer

Well-Known Member

Literally the wurst.

THEE GAY WARLORD™ - Defender of GAY Rights

Well-Known Member

Well-Known Member

Gaming Ninja

Dat one with the Rez

Well-Known Member

Dat one with the Rez

Well-Known Member

Well-Known Member

Similar threads

Popular threads in this forum