A standard container format for game dumps - possible?

[Absestos]

I have been dumping a lot of my collection recently and even just keeping up with the assortment of tools/formats for each system is challenging - which made me start wondering if it would be possible to make an equivalent of the MKV container format for games: a single, standard container format able to store game data efficiently and not bound to a single toolset/emulator.
Right now the closest thing we have to this is CHD, but that only fits 5th gen consoles' optical media formats.

What would the design goals for such a format be, to begin with? I can think of the following:

• Archive based: the format would store game data as files in a hierarchy, without storing the original filesystem itself. Like what zip archives do
• Encryption: game files would be stored unencrypted, for compressibility and easier access to game assets (less overhead for emulators too). Original media and encryption metadata would not take up much extra space to store to allow for rebuilding the original image (or something close to it that would still work on devices)
• Storage efficiency: the format would only store actual data, padding to accomplish a given media layout would be expressed as metadata
• Random access to game assets: like CHD, compressed data would include metadata to enable random access. This would decrease compression efficiency but enable emulators to run the format directly, without needing to install/decompress
• Metadata: ability to embed a game system identifier, game product ID and some other metadata (things like cover image, game title/icon) to help emulators/launchers create a game catalog without manual intervention or internet access. What id3 tags do for MP3s
• Multidisc: ability to bundle multiple discs in a single file for multidisc games

How would such a format map to the way games are actually stored, though, since it would be a major factor in feasibility? My understanding (for home/handheld game systems) is as follows, in broad strokes:

• Up to 4th gen, some later carts (NES/SNES/N64/Genesis/GameGear/GameBoy/GBA/PCEngine/NeoGeo Pocket/WonderSwan): games are a single ROM, memory mapped to device's addressing space - there is no concept of files and assets are accessed by memory location. No encryption
• 5th gen and optical 4th gen (PSX/Saturn/DreamCast/PCEngineCD/PCFX): game data on one data track (usually ISO9660 file system), audio often stored as separate audio CD tracks. No encryption.
• 6th gen (PS2/GC/OG Xbox?): Game data stored in a single data track on optical media (ISO9660/UDF file system). Audio stored as any other game file. Padding/dummy files used to optimize disk layout for faster access on physical systems and balloon sizes
• 6th gen handheld (DS): Game data stored as files in a proprietary file system. No encryption
• 7th gen+, optical (PS3/PSP/PS4/Wii/Wii U/Xbox 360): Game data stored as files in a standard file system (usually UDF) in a single track. Usually encrypted with a per disc key, either at file level or block level. PSP has no encryption
• 7th gen+ carts (3DS/Switch/PSVita): Game data stored as files in a proprietary file system, encryption at block level.
• 7th gen+ digital: Game data stored in compressed, then encrypted archive files.
• All PC games

What do you think?

 

[godreborn]

I think that would cause a lot of confusion. systems (especially newer ones) have their own file system and encryption, so I think it's good as is. you don't want to mix up as a ps4 title with a switch one if it's the same game. we know pkg = ps4; nsp/xci = switch.

 

[Absestos]

They do indeed, which is why I think the idea has merit: when emulating encryption is just overhead and having a standard archive with all assets unencrypted would make game modding a lot easier.

Also, physical media ends up being ISO files for a lot of systems (PS2/PS3/GC/Wii/WiiU). Having metadata about what system a game is meant for in the file would make it possible for tools to make sure they are operating on things they are meant to.

 

[FAST6191]

No love for the Amiga? It had floppy discs, though custom ones (see ADF).

Some of those occasionally do have custom data on disc read via raw reads outside the file system.

Various things there did have encryption as well (the DS technically has some, and while the PSP keys are all now known it had some too -- see if you can find a guide about ripping the eboot, decrypting on a PSP and injecting it back in before a new custom firmware was made to handle the version the PSP game expected), sometimes more as time went on, though some things will be dumped without it.

I am not sure what major problem this solves that is not otherwise handled fairly well, or will likely continue to plague us for decades to come (the PS1 arrived at a time with 900 competing ISO formats and unofficially they all competed on who could rip PS1 games the best, thus we have games using a variety of formats, and the original xbox stuff with ripped games vs what happened when people wanted to play it on the 360 with just a DVD hacked drive).

ROMs and ISOs have 3 main uses

1) Simple play in emulators. Fair enough and can get more complicated, though if the archive format gets too complicated then embedded emulators start to have hard time and it does also mean new emulator authors get to fiddle.

2) Burning to disc/using in flash carts.

3) ROM hacks. Most of these are distributed as patches and patches tend to want to be based on either a Scene release or the "as ripped" condition. You could dodge this a bit but you do have legacy stuff to consider.

I don't dislike the idea of having a more limited archive that maybe has some icon/boxart, manuals, updates and other info bundled in to make things nice.