I am not so worried about ROM sets. If something gets nuked and the nuke sticks it is usually for good reason. Most of the more notable modified things should also be around if you are going for scene history as well. There is next to no chance the 16 bit + N64 and older sets are going to vanish any time soon (the entire thing* is in the range of a cheap USB drive these days, add in GBA and DS and it is still a fairly cheap drive). The multi gig stuff seen in optical media is a different matter, indeed I would not be surprised to find some of the rarer xbox stuff is already next to impossible to find on the internet -- taking all bets on chances of people being able to find 2005-09-16 NHL.06.SWEDiSH.PAL.XBOX-SVAMP and 2005-09-15 Madagascar.PAL.ITALIAN.DVD.XBOX-ReGiT. People have different opinions on "every major game release" (my time doing various things from https://gbatemp.net/threads/links-to-various-gbatemp-features-over-the-years.352851/ had me accused of going for obscure stuff more often than I imagined it would) so OK.
I have met a few in recent times that are experiencing their first ROM site smackdown session. Ask yourself though why those that have been doing this sort of thing for years are not bothered -- it is because it has happened before and will happen again, indeed this last batch was a long time coming it seems.
If the undubs are for the DS I would not worry so much as 99% of them are trivial to recreate, especially if you have a full set (unpack ROMs, copy and rename audio to relevant ROM, rebuild ROM, play game as normal is the usual procedure, no hacking skills involved). Indeed most undubs for things with a file system are trivial to recreate -- some of the PS1 and PS2 stuff with hidden audio/audio outside the conventional iso file system can be trickier but in turn they are the things likely to actually stick. Hacks do tend to be archived in multiple places too (I assume you are familiar with http://www.romhacking.net/ ).
You did not mention the wii but while I am at it. I would suggest you scrub your games (DVD chip safe scrub, no rebuild methods) for that as it makes things so much smaller. I can't rule out Nintendo releasing a wii mini in the future and wanting all the quite literal random junk in there but I am not seeing the scope for something like what happened to the original xbox where a lot of the ripped games were useless on the DVD modded 360s.
As for creating matching sets of archives and parity data I would probably look at command line options for both of them and create a batch file to handle it. If they are all in a given folder tree it should be simple enough; "dir /b >> a.txt" in the command line does a nice list of names into a file called a.txt, from there you can just make something with a spreadsheet program. I do such things all the time.
That said this has the makings of an awful lot of data and this is where things start getting tricky.
You may have heard people talk of mean time between/before failures (mtbf/mtf), the however many thousands of hours they usually quote is somewhat misleading as a) it only applies for the first few years of life and b) does not mean they leave drives running and then count how many thousand hours they have run at time of failure. What it actually means is how many hours of runtime you would get if you had a whole batch of them.
For instance if you had 2000 drives rated at 2000 hours then in one hour you might expect one to pop. Similar things apply to write failures and the like (see also fault tolerant drive arrays).
If it was all doable on just an 8TB setup or a couple of them then that would be one thing, however that is barely going to make a dent on sets of multi gig isos (just for the wii then http://www.abgx.net/wii_releases_date.txt reckons 4800 or so including nukes and ignoring dual layer, also ignoring wiiware and vc releases, multiply those by 4.35 and you are north of 20 TB. Also if you are not familiar with http://www.abgx.net/index.php then I suggest being so). Decide to collect some TV shows and films of reasonable resolution and compression standards and that is right out.
At the same time hard drives are large enough here that "all your eggs in one basket" becomes a thing.
*
Goodsets/goodtools also include hacks, dupes, nukes and more.
What does this massive collection clock in at? Something truly obscene that only a company would dare consider storing?
Answer
33.07 GiB (35511297544 Bytes)
That was circa 2012 but most of those were long dead by then. Also looking at undumped for some of those on http://forums.no-intro.org/viewforum.php?f=8 there is not a lot still outstanding.
I am going to have to think about this one.
I don't like arrays for this as you are going to have to keep them synchronised and you might not be able to get some of the real benefits of such things like drop in replacements (if I have 4 identical drives I can hopefully substitute one in if one goes and basically lose nothing, harder if one also carries your saves you are using at the time and thus different data). Tapes are an option to mix into this, refurb old server gear being fairly affordable and being up around the 50TB mark at times.
Definitely RAID, possibly even nested. Multiple copies of that around the place (maybe even 4 of them), a plan to check them from time to time, a plan to change out things in 5-7 years. A few tapes also distributed.
I have met a few in recent times that are experiencing their first ROM site smackdown session. Ask yourself though why those that have been doing this sort of thing for years are not bothered -- it is because it has happened before and will happen again, indeed this last batch was a long time coming it seems.
If the undubs are for the DS I would not worry so much as 99% of them are trivial to recreate, especially if you have a full set (unpack ROMs, copy and rename audio to relevant ROM, rebuild ROM, play game as normal is the usual procedure, no hacking skills involved). Indeed most undubs for things with a file system are trivial to recreate -- some of the PS1 and PS2 stuff with hidden audio/audio outside the conventional iso file system can be trickier but in turn they are the things likely to actually stick. Hacks do tend to be archived in multiple places too (I assume you are familiar with http://www.romhacking.net/ ).
You did not mention the wii but while I am at it. I would suggest you scrub your games (DVD chip safe scrub, no rebuild methods) for that as it makes things so much smaller. I can't rule out Nintendo releasing a wii mini in the future and wanting all the quite literal random junk in there but I am not seeing the scope for something like what happened to the original xbox where a lot of the ripped games were useless on the DVD modded 360s.
As for creating matching sets of archives and parity data I would probably look at command line options for both of them and create a batch file to handle it. If they are all in a given folder tree it should be simple enough; "dir /b >> a.txt" in the command line does a nice list of names into a file called a.txt, from there you can just make something with a spreadsheet program. I do such things all the time.
That said this has the makings of an awful lot of data and this is where things start getting tricky.
You may have heard people talk of mean time between/before failures (mtbf/mtf), the however many thousands of hours they usually quote is somewhat misleading as a) it only applies for the first few years of life and b) does not mean they leave drives running and then count how many thousand hours they have run at time of failure. What it actually means is how many hours of runtime you would get if you had a whole batch of them.
For instance if you had 2000 drives rated at 2000 hours then in one hour you might expect one to pop. Similar things apply to write failures and the like (see also fault tolerant drive arrays).
If it was all doable on just an 8TB setup or a couple of them then that would be one thing, however that is barely going to make a dent on sets of multi gig isos (just for the wii then http://www.abgx.net/wii_releases_date.txt reckons 4800 or so including nukes and ignoring dual layer, also ignoring wiiware and vc releases, multiply those by 4.35 and you are north of 20 TB. Also if you are not familiar with http://www.abgx.net/index.php then I suggest being so). Decide to collect some TV shows and films of reasonable resolution and compression standards and that is right out.
At the same time hard drives are large enough here that "all your eggs in one basket" becomes a thing.
*
some nfo/release notes from a scan of a pirate site for a ROM collection said:
Goodsets/goodtools also include hacks, dupes, nukes and more.
What does this massive collection clock in at? Something truly obscene that only a company would dare consider storing?
Answer
33.07 GiB (35511297544 Bytes)
That was circa 2012 but most of those were long dead by then. Also looking at undumped for some of those on http://forums.no-intro.org/viewforum.php?f=8 there is not a lot still outstanding.
I am going to have to think about this one.
I don't like arrays for this as you are going to have to keep them synchronised and you might not be able to get some of the real benefits of such things like drop in replacements (if I have 4 identical drives I can hopefully substitute one in if one goes and basically lose nothing, harder if one also carries your saves you are using at the time and thus different data). Tapes are an option to mix into this, refurb old server gear being fairly affordable and being up around the 50TB mark at times.
Definitely RAID, possibly even nested. Multiple copies of that around the place (maybe even 4 of them), a plan to check them from time to time, a plan to change out things in 5-7 years. A few tapes also distributed.
