Misc How do you store your game backups and protect against data rot?

Nynrah

Well-Known Member
OP
Member
Joined
Jun 27, 2020
Messages
207
Trophies
0
Age
32
XP
1,038
Country
Netherlands
Over the years I've built up quite the game library for my many consoles and made sure to back everything I got up. Cartridges, discs, digital games, you name it. By now I've reached a point where my entire collection is a good 3 TB large. So far I've just stored my backups on an external HDD. However, after one of my HDDs died on me (fortunately not one with backups on it!), I began thinking about how I can ensure my backups will be protected and it's like going down the rabbit hole.

I first thought I just had to worry about disk failure and that redundancy would fix the issue, but apparantly data rot is a thing as well. The idea that a backup can silently become corrupted without being aware of it worries me. Apparantly hashes and error correcting codes would be a good measure on top of redundancy to ensure the integrity of the backups. And that doesn't even cover protecting the metadata of the file tree and files themselves or the extra storage space needed for the error correction codes.

For those of you who also own large game collections, how do you guarantee the safety and integrity of your backups? What's your process or protocol (so to speak)? What mediums and tools do you use?

I'd definately like to hear some thoughts on this as I'm trying to figure out what I should do. Right now my thoughts are along the lines of storing two copies of each backup on separate HDDs, generate a hash for each file so I can periodically check if my backups are not corrupt and to generate error correction codes that can help repair my files if neccesary.
 

Sypherone

Gaming Ninja
Member
GBAtemp Patron
Joined
Apr 28, 2019
Messages
1,655
Trophies
2
Age
44
XP
2,813
Country
Germany
Use a journaling filesystem like ntfs or ext4 ( uses checksums) wich can correct automaticalky errors like filesystem and sector errors. They have extra features to keep the files and filesystem in correct state. For exampke Btrfs has some more festures to prevent file errors.

To use error correction code, archives like zip,7zip support them automatically, thats why this archives can be repaired. There exists also QuickPar to generate Par2 parity files for your data.

Then i have one normal drive for regular use and one Backup drive that i dont really touch, only for backup reasons. The secured would be a RAID 5 wich gives redundancy and create correction code of the hole drive. But at least you would need 3 drives only for backup.

Maybe Veam, i was going to check out (there is also a free version), can also provide some extra features.
 
Last edited by Sypherone,

mrdude

Developer
Developer
Joined
Dec 11, 2015
Messages
3,071
Trophies
1
Age
56
XP
8,227
If your hdd died, is it the actual disc having bad sectors on it, or is it the hdd motherboard? If it's the motherboard you can most likely swap it out with another of the same type of drive if you can't find out what the bad component is. Usually I have a few drives of the same make for this kind of thing happening.
 

tech3475

Well-Known Member
Member
Joined
Jun 12, 2009
Messages
3,653
Trophies
2
XP
6,041
Country
Currently I have software do monthly hash checks on my unraid server (which offers parity protection, which is also checked monthly) and have backups made using Macrium reflect to a different system.
 

RAHelllord

Literally the wurst.
Member
Joined
Jul 1, 2018
Messages
707
Trophies
1
XP
2,732
Country
Germany
Your backup plan should always include redundancies, the only proper way to protect data with near certainty is to have another copy of it somewhere else.

Even if you have 5 full copies of the data at home, your home burning down while you're out for groceries would mean potentially losing all 5 at once. As such, off-shore backups would be the way to go, or literally just making a copy and leaving it with someone else if you don't want to pay another company to keep it safe for you.

But if you want to be realistic, have at least one additional backup of everything somewhere safe, preferably a place that's dry, dark, and at least somewhat climate controlled, and maybe if you have a friend you trust give them another copy to keep safe for you. Check all backups regularly, like once per year at least, and try to restore a random selection of files to verify they still work properly.
 

tech3475

Well-Known Member
Member
Joined
Jun 12, 2009
Messages
3,653
Trophies
2
XP
6,041
Country
Your backup plan should always include redundancies, the only proper way to protect data with near certainty is to have another copy of it somewhere else.

Even if you have 5 full copies of the data at home, your home burning down while you're out for groceries would mean potentially losing all 5 at once. As such, off-shore backups would be the way to go, or literally just making a copy and leaving it with someone else if you don't want to pay another company to keep it safe for you.

But if you want to be realistic, have at least one additional backup of everything somewhere safe, preferably a place that's dry, dark, and at least somewhat climate controlled, and maybe if you have a friend you trust give them another copy to keep safe for you. Check all backups regularly, like once per year at least, and try to restore a random selection of files to verify they still work properly.

If you go offsite, I would suggest encryption with a custom key securely stored elsewhere.

Personally, I have things like family photos backed up online (encrypted and stored on Office 365/One Drive using Arq) and most other files backed up to a server at my parents house.

Previously used Crashplan but that was unreliable and slow, other services may be better but didn’t support unraid and/or the costs added up per TB.
 

Nynrah

Well-Known Member
OP
Member
Joined
Jun 27, 2020
Messages
207
Trophies
0
Age
32
XP
1,038
Country
Netherlands
Probably good to add, but for my own situation I'm looking at the best way to implement these measures using two external HDDs that I access from a Windows computer. That means no NAS or RAID setups - just cold storage. Part of the reason being that it's not storage I need to access often, so it didn't make sense for me to have something that's running 24/7. Granted, there's also the financial aspect.

Use a journaling filesystem like ntfs or ext4 ( uses checksums) wich can correct automaticalky errors like filesystem and sector errors. They have extra features to keep the files and filesystem in correct state. For exampke Btrfs has some more festures to prevent file errors.

To use error correction code, archives like zip,7zip support them automatically, thats why this archives can be repaired. There exists also QuickPar to generate Par2 parity files for your data.

Then i have one normal drive for regular use and one Backup drive that i dont really touch, only for backup reasons. The secured would be a RAID 5 wich gives redundancy and create correction code of the hole drive. But at least you would need 3 drives only for backup.

Maybe Veam, i was going to check out (there is also a free version), can also provide some extra features.
I heard that DAR and zip deflate are good archive formats to use compared to solid archives. DAR sounded quite enticing with native PAR2 error correction, but I've used 7zip before, so that feels more familiar. So let's say I use 7zip with the deflate algorythm, how do I apply the error correction to it?

Also, I'm considering whether to pack each game backup into its own zip archive or pack the entire disc into one big zip archive. The reason for that is because if things crash and metadata like the file tree is damaged, I'd be in for a merry time scraping a ton of files. If there's only one big file, file scraping wouldn't be so bothersome as that single file basically contains the whole file system on the HDD. What I wonder in this case, though, is whether the error correction functionality is incremental if adding new files to the archive and how generating and verifying hashes would work in this case.
 

Sypherone

Gaming Ninja
Member
GBAtemp Patron
Joined
Apr 28, 2019
Messages
1,655
Trophies
2
Age
44
XP
2,813
Country
Germany
The CRC correction is in 7zip and zip already nativ in the archiv header implemented and calculated when the archiv is created. By changing the archiv content it will be automatically new calculated.

I would consider to pack every game in one single archiv. E.G. by a lot of files wich are going in to the same folder, i would write a short bash/ Powershell commandline script.
If you create one big archiv and it gets really bad damaged, all is gone. Done know if backing up the header of the archiv is possible.
 

SylverReZ

Dat one with the Rez
Member
GBAtemp Patron
Joined
Sep 13, 2022
Messages
7,108
Trophies
3
Location
The Wired
Website
m4x1mumrez87.neocities.org
XP
21,797
Country
United Kingdom
I love backing up my physical media and that this has been my passion for so many years thanks to the Redump project. It's always important to not only have all of your games backed up but also other important stuff you may have.
 
Last edited by SylverReZ,

Nynrah

Well-Known Member
OP
Member
Joined
Jun 27, 2020
Messages
207
Trophies
0
Age
32
XP
1,038
Country
Netherlands
I love backing up my physical media and that this has been my passion for so many years thanks to the Redump project. It's always important to not only have all of your games backed up but also other important stuff you may have.
How does Redump factor into this?

I settled for using multiple HDDs with each game file being hashed so I can periodically verify file integrity. I didn't add error correction codes because I just couldn't get a good idea of how to go about it. Things like how much ECC data to generate per block of file data. Given the size of my library, the ECCs would inevitably cost lots of space as well. Since I couldn't determine what's optimal both in practice and financially, I settled for triple redundancy on separate HDDs with hashes.
 
  • Like
Reactions: SylverReZ

Nynrah

Well-Known Member
OP
Member
Joined
Jun 27, 2020
Messages
207
Trophies
0
Age
32
XP
1,038
Country
Netherlands
Helped me made accurate disc backups using their guides. I'm not referring to HDDs but rather physical media such as CDs, DVDs, you name it.
Ah, I get what you mean now. I did consider using Bluray discs, but that didn't seem like such a hot idea when picturing how many discs I'd need for 3 TB worth of games.
 
  • Like
Reactions: SylverReZ

Marc_LFD

Well-Known Member
Member
Joined
Nov 3, 2021
Messages
5,487
Trophies
1
Age
34
XP
8,871
Country
United States
I have game console backups stored in TB HDDs, but yeah, they could eventually fail. And honestly, I'm okay with that as I'm not that much into gaming too much.

I even RAR'd PS2 games which saved an incredible amount of space although GCN/Wii/PS3/PS4 it really didn't do much of a difference. :-/
 

Site & Scene News

Popular threads in this forum

General chit-chat
Help Users
  • No one is chatting at the moment.
    NinStar @ NinStar: where are my cute girls