Hacking WIA = Wii ISO Archive

[wishmasterf]

Ok so do you think it is ready to start backup my wbfs-drive with WIA?

I think "yes" but I say "no" until I have finished my (long time) tests.

OK. I understand. I will use it and we will see ;-)

 

[ssjkakaroto]

@Wiimm: I tried using wit (r1645) to convert Metroid Other M (dual-layer) ISO to WIA and back to ISO but the MD5 of the resulting ISO was different from the original. Is this expected?
The commands I used were:

wit COPY --raw --progress --wia --no-compress file.iso file.wia
wit COPY --raw --progress --iso --no-compress file.wia file2.iso

 

[Wiimm]

First: You can shorten the commands:

wit COPY --raw --progress --no-compress file.iso file.wia
wit COPY --raw --progress file.wia file2.iso

a) If the file extension is well known the output format is clear. The option like --wia overrides the output format.
b) --no-compress is only relevant for converting *to wia*.

Your question:
Have both ISO the same length?
Try "wit diff --raw file.iso file2.iso --long --long" or "wit diff --raw file.iso file2.iso --long --long". It compares the content and prints the block relevant blocks.

I will do the same test.

 

[ssjkakaroto]

Both ISO's were the same size, only the WIA file was smaller.
Here's the output: http://paste2.org/p/980071

 

[Wiimm]

I have made the same test with similar result. The ISOs differ in the space behind the last partition (=waste). Here is a disc memory layout of "Other M":

Spoiler

# wit DUMP mom.wdf --show usage,dmap --raw
ÂÂISO Usage Map:

ÂÂÂÂÂÂÂÂÂÂÂÂ0: **********AAAAaa aaaaaaaaaaaaaaaa aaaaaaaaaaaaaaaa aaaaaaaaaaaaaaaa
ÂÂÂÂÂÂ 200000: aaaaaaaaaaaaaaaa aaaaaaaaaaaaaaaa aaaaaaaaaaaaaaaa aaaaaaaaaaaaaaaa *ÂÂ100
ÂÂÂÂÂÂca00000: aaaaaaaaaaaaaaaa aaaaaaaaaaaaaaaa aa************** ****************
ÂÂÂÂÂÂcc00000: **************** **************** **************** **************** *ÂÂ 22
ÂÂÂÂÂÂf800000: BBBBbbbbbbbbbbbb bbbbbbbbbbbbbbbb bbbbbbbbbbbbbbbb bbbbbbbbbbbbbbbb
ÂÂÂÂÂÂfa00000: bbbbbbbbbbbbbbbb bbbbbbbbbbbbbbbb bbbbbbbbbbbbbbbb bbbbbbbbbbbbbbbb * 3925
ÂÂÂÂ1fa400000: bbbbbbbbbbbbbbbb bbbbbbbbbbbbbbbb bbbbbbbbbb****** ****************
ÂÂÂÂ1fa600000: **************** **************** **************** **************** *ÂÂÂÂ9


ÂÂISO Memory Map:

ÂÂÂÂÂÂÂÂÂÂunused :ÂÂ off(beg) ..ÂÂ off(end) :ÂÂÂÂÂÂsize : info
ÂÂÂÂ------------------------------------------------------------------------------------------
ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ :ÂÂÂÂÂÂÂÂÂÂ0 ..ÂÂÂÂÂÂÂÂ100 :ÂÂÂÂÂÂ 100 : Header, magic=5d-1c-9e-a3, id=R3OP01
ÂÂÂÂÂÂÂÂÂÂ 3ff00 :ÂÂÂÂÂÂ40000 ..ÂÂÂÂÂÂ40020 :ÂÂÂÂÂÂÂÂ20 : Partition address table, N=2+0+0+0
ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ :ÂÂÂÂÂÂ40020 ..ÂÂÂÂÂÂ40030 :ÂÂÂÂÂÂÂÂ10 : Partition table #0 with 2 partitions
ÂÂÂÂÂÂÂÂÂÂÂÂdfd0 :ÂÂÂÂÂÂ4e000 ..ÂÂÂÂÂÂ4e020 :ÂÂÂÂÂÂÂÂ20 : Region settings, region=2
ÂÂÂÂÂÂÂÂÂÂÂÂ1fdc :ÂÂÂÂÂÂ4fffc ..ÂÂÂÂÂÂ50000 :ÂÂÂÂÂÂÂÂ 4 : Magic2: c3-f8-1a-8e
ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ :ÂÂÂÂÂÂ50000 ..ÂÂÂÂÂÂ502c0 :ÂÂÂÂÂÂ 2c0 : P.0.0 UPDATE: ticket, id=.UPP
ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ :ÂÂÂÂÂÂ502c0 ..ÂÂÂÂÂÂ504c8 :ÂÂÂÂÂÂ 208 : P.0.0 UPDATE: tmd, ios=4, id=.UPP
ÂÂÂÂÂÂÂÂÂÂÂÂÂÂ18 :ÂÂÂÂÂÂ504e0 ..ÂÂÂÂÂÂ50ee0 :ÂÂÂÂÂÂ a00 : P.0.0 UPDATE: cert
ÂÂÂÂÂÂÂÂÂÂÂÂ7120 :ÂÂÂÂÂÂ58000 ..ÂÂÂÂÂÂ70000 :ÂÂÂÂ 18000 : P.0.0 UPDATE: h3
ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ :ÂÂÂÂÂÂ70000 ..ÂÂÂÂcb08000 :ÂÂ ca98000 : P.0.0 UPDATE: data+fst, N(fst)=66
ÂÂÂÂÂÂÂÂ 2cf8000 :ÂÂÂÂf800000 ..ÂÂÂÂf8002c0 :ÂÂÂÂÂÂ 2c0 : P.0.1 DATA: ticket, id=R3OP
ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ :ÂÂÂÂf8002c0 ..ÂÂÂÂf8004c8 :ÂÂÂÂÂÂ 208 : P.0.1 DATA: tmd, ios=56, id=R3OP
ÂÂÂÂÂÂÂÂÂÂÂÂÂÂ18 :ÂÂÂÂf8004e0 ..ÂÂÂÂf800ee0 :ÂÂÂÂÂÂ a00 : P.0.1 DATA: cert
ÂÂÂÂÂÂÂÂÂÂÂÂ7120 :ÂÂÂÂf808000 ..ÂÂÂÂf820000 :ÂÂÂÂ 18000 : P.0.1 DATA: h3
ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ :ÂÂÂÂf820000 ..ÂÂÂÂfad8000 :ÂÂÂÂ2b8000 : P.0.1 DATA: data+fst, N(fst)=5070
ÂÂÂÂÂÂÂÂ177e8000 :ÂÂ 272c0000 ..ÂÂ1f7500000 : 1d0240000 : P.0.1 DATA: data+fst
ÂÂÂÂÂÂÂÂ 2fc8000 :ÂÂ1fa4c8000 ..ÂÂ1fa550000 :ÂÂÂÂ 88000 : P.0.1 DATA: data+fst
ÂÂÂÂÂÂÂÂ 12b0000 :ÂÂ1fb800000 ..ÂÂ1fb800000 :ÂÂÂÂÂÂÂÂ 0 : -- End of file/disc --

You see: The useful data ends at offset 0x1fa550000 and your diffs (and my too) are behind that point. If you use "wit diff" without --raw only needed/used sectors are compared.

That's bad for raw mode and I have improve this. It's possible that this is only a problem for double layer.


EDIT:
I store all data from 0x0..0x4ffff and all partition data (in scrubbed or raw mode). Data between partitions and behind the last partition is not copied. This can be changed without changing the WIA format, because I have already spend a raw data area in the data structure.

Another file format independent question is: Is it really needed to store such data?

 

[Remfin]

The archivers/collectors are going to want ALL the data, even the garbage--they want stuff bit-perfect no matter what. So a proper archive format should probably keep it.

If it helps, it's suspected it's simply an artifact of the encryption process...try running a decryption with the various title keys on the disc (and maybe with different alignments?) and see if one of them gets you zeroes.

 

[wishmasterf]

Both ISO's were the same size, only the WIA file was smaller.
Here's the output: http://paste2.org/p/980071

Is the ISO still useable? I mean could you play the game after converting to WIA and back?

 

[Wiimm]

Above I told:
a) Only filling data behind the last partition is missed. The game is still playable and scrubbing will ignore this data area and more.
b) I'm working to improve the raw mode to store *all* data. But this takes time because I'm very busy in this week.


@all:
Don't use WIA permanent because I'm on the way to change the format. After the raw data discussion I see a way to simplify the WIA format. This makes it easier to implement it into other tools. And this test phase is a good time for changing.

 

[W hat]

I hope an option will exist for heavy compression - FreeARC or 7zip (LZMA). I know you've already thought about this, and that all WIA development is just for fun for you...

B - Does this mean you will try to go from Superdump->ISO->WIA->ISO without losing any garbage data?

 

[Wiimm]

The WIA structure supports 4294967296 possible compression methods. At the moment 0=uncompressed and 1=bzip2 are implemented. My road map:
1.) finish WIA implementation with 2 methods and a better raw support (good for gamecube discs too)
2.) finish gamecube support (incl. mix command and wia support)
3.) implement trimming in 3 levels (disc, partition, file system)
...
n) view the 7z lib source and ....

B - Does this mean you will try to go from Superdump->ISO->WIA->ISO without losing any garbage data?

Yes if using --raw (because scrubbing is the default)
This is already true for all other conversions with "wit copy --raw ..."

 

[W hat]

That's a bold move; I'd expect nothing less from Wiimm.

I can't wait to see all of it, even if what I'm looking forward to is shoved aside right now for GameCube support and trimming. Trimming means using 00 to scrub, not FF, so you'll get an ISO that is 2.35 GiB in modern filesystems, but has a real size of 4.3GiB?

 

[ssjkakaroto]

...
Another file format independent question is: Is it really needed to store such data?
My idea was to use the WIA image to create a conversion patch between the JAP and US versions of the game. I thought the patches would be smaller than using the original ISOs because the data would be decrypted, but first I wanted to make sure if I could go back and forth between the formats without any loss. So yes, if one has the intention of eventually returning to the original ISO, that data is definitely needed.

QUOTE(wishmasterf @ Sep 8 2010, 12:28 PM) Is the ISO still useable? I mean could you play the game after converting to WIA and back?

I don't know. I don't use USB loaders, so I didn't try burning the disc just to test.


Thanks for looking into it Wiimm

 

[W hat]

...
Another file format independent question is: Is it really needed to store such data?

My idea was to use the WIA image to create a conversion patch between the JAP and US versions of the game. I thought the patches would be smaller than using the original ISOs because the data would be decrypted, but first I wanted to make sure if I could go back and forth between the formats without any loss. So yes, if one has the intention of eventually returning to the original ISO, that data is definitely needed.

1. Garbage data is not needed ever - it may as well be storing Wikipedia archives, or other more useful data. Different people have different needs, of course, but I think the garbage data may be specifically designed to be incompressible.

2. My hopes for Wii storage formats are as follows:

a. Store patches for games efficiently (For example, Disney Princess and the Frog regular edition -> deluxe edition, Wii Sports v1.00 -> v1.01) and automate this patching process
b. Store Wii games that I never play as efficiently as I can, with a maximum compression time of two hours and maximum decompression time of one hour.
c. Superdumps of the same disc from multiple sources should still be identical after being archived, provided "nodates" and other modes are used to ensure uniform compression.

 

[wishmasterf]

a. Store patches for games efficiently (For example, Disney Princess and the Frog regular edition -> deluxe edition, Wii Sports v1.00 -> v1.01) and automate this patching process

Do we get patches for games or only if i download the whole ISO on warez sites?

 

[W hat]

I'm not going to lie - I have pirated games before. That does not mean I'm filthy scum - I've bought Super Mario Galaxy 2, New Super Mario Bros. Wii, Endless Ocean, Endless Ocean 2, and many more. (I've even bought games I didn't like because I don't believe in the "pirate before you buy" policy.)

Did I buy "The Princess and the Frog"? No. Did I play it? No. Did I use it as an example, and would I be interested to see a patch from the normal to deluxe edition? From a theoretical standpoint, yes. This is not because I wish to steal the deluxe features of the Deluxe edition. Same thing with Wii Sports 1.00 to 1.01. I'm sure if Nintendo had the means to distribute patches, a Wii Sports update would be free. Even so, I have not downloaded Wii Sports v1.01, merely used it as an example.

Wiimm seems to be very honest - I don't believe he ever downloads games, even if downloading a JPN or NTSC-U game would help his testing.

 

[Wiimm]

I can't wait to see all of it, even if what I'm looking forward to is shoved aside right now for GameCube support and trimming. Trimming means using 00 to scrub, not FF, so you'll get an ISO that is 2.35 GiB in modern filesystems, but has a real size of 4.3GiB?

For my tools scrubbing means to leave out unused areas while writing. This is similar like writing zeros, but for plain iso file the OS creates sparse files for the unused areas. WBFS and CISO files are also candidates for sparse files because of the large block size (>=2MB). WDF and WIA manage this holes. If reading the holes seems to be zeroed.

With trimming I mean moving data. For details read this and this. Wii Scrubber moves also data.

 

[wishmasterf]

@W hat

I started a List of "new edition games". Maybe you know more of this games and add them.

new edition games

 

[wishmasterf]

@all:
Don't use WIA permanent because I'm on the way to change the format. After the raw data discussion I see a way to simplify the WIA format. This makes it easier to implement it into other tools. And this test phase is a good time for changing.

When will the next release available?

 

[Wiimm]

@all:
Don't use WIA permanent because I'm on the way to change the format. After the raw data discussion I see a way to simplify the WIA format. This makes it easier to implement it into other tools. And this test phase is a good time for changing.

When will the next release available?

I will start the re-implementation this week end.

 

[Wiimm]

I have re-implemented WIA. It support now three compression methods:

• --compression NONE
  Data of used (=non scrubbed sectors) are stored 1:1 with 0x8000 bytes (32KiB) or 0x7c00 bytes for decrypted data partitions (hash removed). This format is very good for an external packer. No hash values for the chunks available.
  .
• --compression PURGE
  Data is purged like WDF: Areas with zeros are detected and skipped. The image is smaller than 'NONE' but contains more management info. There is a SHA1 hash for each data chunk so that data errors can be noticed.
  .
• --compression BZIP2
  Each chunk is compressed with bzip2 level 9. bzip2 has its own verification code.
  .

Here are some first compression ratings. As expected WIA/NONE+7Z is the smallest. I'm wondering that WIA/BZIP2 is smaller than WIA/NONE+BZIP2

Summary of RZDP01, The Legend of Zelda: Twilight Princess:

ÂÂ 808195658ÂÂ70.21%ÂÂ 10:34.731 m:sÂÂWIA/NONE + 7Z
ÂÂ 808527639ÂÂ70.24%ÂÂ 11:03.371 m:sÂÂWIA/PURGE + 7Z
ÂÂ 834192659ÂÂ72.46%ÂÂ 10:35.820 m:sÂÂ WDF/DECRYPT + 7Z
ÂÂ1010092171ÂÂ87.75%ÂÂÂÂ9:43.185 m:sÂÂWIA/BZIP2
ÂÂ1010175516ÂÂ87.75%ÂÂÂÂ9:44.945 m:sÂÂWIA/NONE + BZIP2
ÂÂ1010696787ÂÂ87.80%ÂÂÂÂ9:43.515 m:sÂÂWIA/PURGE + BZIP2
ÂÂ1014462607ÂÂ88.13%ÂÂ 18:46.240 m:sÂÂWIA/BZIP2 + BZIP2
ÂÂ1022830826ÂÂ88.85%ÂÂ 17:08.357 m:sÂÂWIA/BZIP2 + 7Z
ÂÂ1038300726ÂÂ90.20%ÂÂÂÂ9:43.655 m:sÂÂ WDF/DECRYPT + BZIP2
ÂÂ1089171376ÂÂ94.62%ÂÂÂÂ1:40.799 m:sÂÂWIA/PURGE
ÂÂ1119341208ÂÂ97.24%ÂÂÂÂ1:38.125 m:sÂÂWIA/NONE
ÂÂ1124536808ÂÂ97.69%ÂÂÂÂ1:27.135 m:sÂÂWDF/DECRYPT
ÂÂ1151087416 100.00%ÂÂÂÂÂÂ25.264 secÂÂ WDF
ÂÂ1156120637 100.43%ÂÂ 10:46.613 m:sÂÂ WDF + BZIP2
ÂÂ1166488810 101.33%ÂÂÂÂ8:49.884 m:sÂÂ WDF + 7Z

At the moment I'm working at the reading part.