Hacking WIA = Wii ISO Archive

[moosehunter]

Do you think that this format would work well if it was implemented in Dolphin?

 

[Wiimm]

Here are some ratings to compare the old and new WIA implementation. The images compressed with the new algorithm is a little better:

Summary of RUUP01, Animal Crossing: Let's Go to the City:

ÂÂ 123325063ÂÂ34.96%ÂÂÂÂ3:38.056 m:sÂÂ+ WIA/NONE + BZIP2
ÂÂ 123687704ÂÂ35.06%ÂÂÂÂ3:16.845 m:sÂÂ+ WIA/BZIP2
ÂÂ 124191993ÂÂ35.21%ÂÂÂÂ4:26.899 m:sÂÂ+ WIA/BZIP2 + BZIP2
ÂÂ 127243676ÂÂ36.07%ÂÂÂÂ3:23.276 m:sÂÂ+ WIA/PURGE + BZIP2
ÂÂ 127436075ÂÂ36.13%ÂÂÂÂ2:36.284 m:sÂÂold WIA/NOCOMPRESS + BZIP2
ÂÂ 127696363ÂÂ36.20%ÂÂÂÂ2:31.820 m:sÂÂold WIA/BZIP2
ÂÂ 128007670ÂÂ36.29%ÂÂÂÂ3:40.140 m:sÂÂold WIA/BZIP2 + BZIP2
ÂÂ 312521336ÂÂ88.60%ÂÂÂÂÂÂ52.595 secÂÂ+ WIA/PURGE
ÂÂ 312976812ÂÂ88.73%ÂÂÂÂÂÂ27.094 secÂÂold WIA/NOCOMPRESS
ÂÂ 347369520ÂÂ98.48%ÂÂÂÂÂÂ53.021 secÂÂ+ WIA/NONE

Summary of RHAP01, Wii Play:

ÂÂÂÂ52899190ÂÂ52.03%ÂÂÂÂ1:16.932 m:sÂÂ+ WIA/NONE + 7Z
ÂÂÂÂ52934196ÂÂ52.07%ÂÂÂÂÂÂ56.550 secÂÂold WIA/NOCOMPRESS + 7Z
ÂÂÂÂ52935121ÂÂ52.07%ÂÂÂÂÂÂ59.450 secÂÂ+ WIA/PURGE + 7Z
ÂÂÂÂ80179986ÂÂ78.87%ÂÂÂÂÂÂ50.465 secÂÂ+ WIA/BZIP2
ÂÂÂÂ80299088ÂÂ78.98%ÂÂÂÂ1:31.244 m:sÂÂ+ WIA/NONE + BZIP2
ÂÂÂÂ80301483ÂÂ78.99%ÂÂÂÂÂÂ50.565 secÂÂold WIA
ÂÂÂÂ80354061ÂÂ79.04%ÂÂÂÂÂÂ51.282 secÂÂold WIA/NOCOMPRESS + BZIP2
ÂÂÂÂ80420690ÂÂ79.10%ÂÂÂÂÂÂ54.095 secÂÂ+ WIA/PURGE + BZIP2
ÂÂÂÂ80531997ÂÂ79.21%ÂÂÂÂ1:34.511 m:sÂÂ+ WIA/BZIP2 + BZIP2
ÂÂÂÂ80581068ÂÂ79.26%ÂÂÂÂ1:33.696 m:sÂÂold WIA + BZIP2
ÂÂÂÂ81167196ÂÂ79.84%ÂÂÂÂ1:28.863 m:sÂÂ+ WIA/BZIP2 + 7Z
ÂÂÂÂ81215012ÂÂ79.88%ÂÂÂÂ1:27.386 m:sÂÂold WIA + 7Z
ÂÂÂÂ98903128ÂÂ97.28%ÂÂÂÂÂÂ11.849 secÂÂ+ WIA/PURGE
ÂÂÂÂ99358124ÂÂ97.73%ÂÂÂÂÂÂ 8.297 secÂÂold WIA/NOCOMPRESS
ÂÂ 101458756ÂÂ99.80%ÂÂÂÂÂÂ29.764 secÂÂ+ WIA/NONE

* Remark: "+ 7Z" or "+ BZIP2" mans: compress the resulting file with the external tool.

Reading is ready too. At the moment a test script (takes many hours) is running.

Do you think that this format would work well if it was implemented in Dolphin?

It will work, but 'well'? Decompression takes time and will slow down the game whenever it loads data drom the dsic.

 

[wishmasterf]

The difference between BZIP2 and 7z seems to be not small. Do not need an external tool make the use of the implemented BZIP2 more handy.

To use 7z - Libary is not a choice?

 

[Wiimm]

The difference between BZIP2 and 7z seems to be not small. Do not need an external tool make the use of the implemented BZIP2 more handy.I use the bzip2 lib to create the wia files. The external bzip2 is ony used to compare the results.

To use 7z - Libary is not a choice?
My answer:
QUOTE(Wiimm @ Sep 8 2010, 10:41 PM)

My road map:
1.) finish WIA implementation with 2 methods and a better raw support (good for gamecube discs too)
2.) finish gamecube support (incl. mix command and wia support)
3.) implement trimming in 3 levels (disc, partition, file system)
...
n) view the 7z lib source and ....

 

[wishmasterf]

My road map:
n) view the 7z lib source and ....

Nice to know it is still on your roadmap. The Number "n" tells me it wont be in near future.

 

[Wiimm]

n and the ellipses before means: I don't know if some other more important things pop up.

 

[Wiimm]

A new WIT beta is out: http://gbatemp.net/index.php?act=findpost&pid=3110656
"Metroid Other" works fine, tested with --raw --compression purge

 

[ssjkakaroto]

Thanks Wiimm, it's working now

Edit: Bah, the xdelta patch between the US and JAP versions of the game (using WIA) was >6 GiB :/
At least it was smaller than the patch using the regular ISOs, which was >8 GiB.

 

[Wiimm]

Bug found:
Creation of a WIA file fails if option --psel is set and at least one partition is removed (scrubbed). The result is an invalid disc image.

• Don't use --psel when creating a WIA file.

The beta repository contains an update -- no time to produce binaries.

btw: lzma/lzma2 compressions seem easy ....

 

[hdhacker]

I guess the lesson here is if you're going to use compression, don't try to help it.

 

[Wiimm]

Here are some compression rates. All games are scrubbed and all non DATA partitions are removed. 100% is the WDF size of the game:

ÂÂ-------------------------------------------------------------------------------
ÂÂÂÂÂÂ bytesÂÂÂÂWDF%ÂÂÂÂcompr.timeÂÂÂÂfile typeÂÂid6ÂÂÂÂ game
ÂÂ-------------------------------------------------------------------------------
ÂÂÂÂ 7823978ÂÂ31.07%ÂÂÂÂÂÂ23.994 secÂÂWIA/BZIP2ÂÂRTYP01, Wii Chess
ÂÂ 123687748ÂÂ35.06%ÂÂÂÂ2:43.546 m:sÂÂWIA/BZIP2ÂÂRUUP01, Animal Crossing
ÂÂ6121472090ÂÂ78.57% 1:09:07.263 hmsÂÂWIA/BZIP2ÂÂR3OP01, Metroid: Other M
ÂÂ2615893038ÂÂ78.76%ÂÂ 30:27.045 m:sÂÂWIA/BZIP2ÂÂRLRP4F, Tomb Raider: Anniversary
ÂÂÂÂ80180000ÂÂ78.87%ÂÂÂÂÂÂ48.991 secÂÂWIA/BZIP2ÂÂRHAP01, Wii Play
ÂÂ 531405683ÂÂ80.21%ÂÂÂÂ5:28.472 m:sÂÂWIA/BZIP2ÂÂRSPP01, Wii Sports
ÂÂ5898426072ÂÂ81.50% 1:16:52.591 hmsÂÂWIA/BZIP2ÂÂRSBP01, Super Smash Bros. Brawl
ÂÂ 115261529ÂÂ82.06%ÂÂÂÂ1:15.351 m:sÂÂWIA/BZIP2ÂÂR64P01, Wii Music
ÂÂ1010092197ÂÂ87.75%ÂÂÂÂ9:37.114 m:sÂÂWIA/BZIP2ÂÂRZDP01, Twilight Princess
ÂÂ2566905355ÂÂ92.58%ÂÂ 26:04.568 m:sÂÂWIA/BZIP2ÂÂRMCP01, Mario Kart Wii
ÂÂ-------------------------------------------------------------------------------

 

[Wiimm]

Yesterday I have implemented LZMA support (was easier than thought) and over night I have run my test script. Here are the ratings of some games (my test suite):

Spoiler

Summary of RTYP01, Wii Chess:
ÂÂÂÂ 6524978ÂÂ25.91%ÂÂÂÂÂÂ15.172 secÂÂWIA/LZMA
ÂÂÂÂ 7823979ÂÂ31.07%ÂÂÂÂÂÂ24.085 secÂÂWIA/BZIP2
ÂÂÂÂ25178940 100.00%ÂÂÂÂÂÂ 0.707 secÂÂWDF

Summary of RHAP01, Wii Play:
ÂÂÂÂ72698350ÂÂ71.51%ÂÂÂÂ1:04.470 m:sÂÂWIA/LZMA
ÂÂÂÂ80179991ÂÂ78.87%ÂÂÂÂÂÂ48.773 secÂÂWIA/BZIP2
ÂÂ 101659348 100.00%ÂÂÂÂÂÂ 2.799 secÂÂWDF

Summary of RUUP01, Animal Crossing: Let's Go to the City:
ÂÂÂÂ96078956ÂÂ27.24%ÂÂÂÂ3:49.777 m:sÂÂWIA/LZMA
ÂÂ 123687750ÂÂ35.06%ÂÂÂÂ2:43.188 m:sÂÂWIA/BZIP2
ÂÂ 352695380 100.00%ÂÂÂÂÂÂ 9.153 secÂÂWDF

Summary of RSPP01, Wii Sports:
ÂÂ 531405724ÂÂ80.21%ÂÂÂÂ5:28.194 m:sÂÂWIA/BZIP2
ÂÂ 536655222ÂÂ81.01%ÂÂÂÂ6:37.131 m:sÂÂWIA/LZMA
ÂÂ 662451000 100.00%ÂÂÂÂÂÂ14.744 secÂÂWDF

Summary of RZDP01, The Legend of Zelda: Twilight Princess:
ÂÂ 934501837ÂÂ81.18%ÂÂ 10:46.498 m:sÂÂWIA/LZMA
ÂÂ1010092224ÂÂ87.75%ÂÂÂÂ9:35.750 m:sÂÂWIA/BZIP2
ÂÂ1151087416 100.00%ÂÂÂÂÂÂ30.494 secÂÂWDF

Summary of RMCP01, Mario Kart Wii:
ÂÂ2445917344ÂÂ88.21%ÂÂ 25:53.659 m:sÂÂWIA/LZMA
ÂÂ2566905494ÂÂ92.58%ÂÂ 26:01.525 m:sÂÂWIA/BZIP2
ÂÂ2772579124 100.00%ÂÂÂÂ1:57.141 m:sÂÂWDF

Summary of RLRP4F, Tomb Raider: Anniversary:
ÂÂ2366869769ÂÂ71.26%ÂÂ 33:21.783 m:sÂÂWIA/LZMA
ÂÂ2615893146ÂÂ78.76%ÂÂ 30:56.137 m:sÂÂWIA/BZIP2
ÂÂ3321312056 100.00%ÂÂÂÂ4:49.390 m:sÂÂWDF

Summary of RSBP01, Super Smash Bros. Brawl:
ÂÂ5463767379ÂÂ75.49% 1:13:42.016 hmsÂÂWIA/LZMA
ÂÂ5898426268ÂÂ81.50% 1:20:57.812 hmsÂÂWIA/BZIP2
ÂÂ7236890204 100.00%ÂÂÂÂ9:45.384 m:sÂÂWDF

Summary of R3OP01, Metroid: Other M:
ÂÂ5848173608ÂÂ75.06% 1:24:21.627 hmsÂÂWIA/LZMA
ÂÂ6121472308ÂÂ78.57% 1:09:21.359 hmsÂÂWIA/BZIP2
ÂÂ7790477636 100.00%ÂÂÂÂ5:24.926 m:sÂÂWDF

The test scripts includes decryption and comparing and all works fine. The next steps are LZMA2, fine tuning (LZMA has much compression parameters), --raw tests and code cleanup.

 

[Wiimm]

LZMA and LZMA2 (7zip main compressors) implemented!! See beta release.

It's now your turn to test all ...

 

[moosehunter]

Wii Play
ÂÂÂÂ53457608ÂÂ55.86%ÂÂÂÂÂÂWIA+LZMA2
ÂÂÂÂ53471437ÂÂ55.88%ÂÂÂÂÂÂWIA+LZMA
ÂÂÂÂ70627126ÂÂ73.80%ÂÂÂÂÂÂWIA/LZMA2
ÂÂÂÂ70637763ÂÂ73.81%ÂÂÂÂÂÂWIA/LZMA
ÂÂÂÂ95696044ÂÂ100.00%ÂÂÂÂ WIA/none

 

[Wiimm]

Quick infos:
- The endian tests on a big endian machine are positive.
- I have tested all 5 compression methods my test suite (10 scrubbed images) without failures.
- I think about changing the format again and adding a field 'chunk_size' to support not only 2MiB.

@moosehunter
What will you say to me?

 

[moosehunter]

@moosehunter
What will you say to me?

Hmm?

 

[Richardarkless]

I find this incredibly interesting and care about saving as much space as possible but is it possible to implement this into usb loaders and if so do you know any developers that are willing to implement it into their usb loaders once of course it is final and ready for public use

 

[moosehunter]

Probably not. Either it would create too much overhead to play smoothly, or there wouldn't be enough memory to decompress it.

And I am very impressed by this. I was able to compress a raw unscrubbed MPT to 66% of it's original size.

Would I be correct in assuming that compressing a raw game that takes up most of the disk would compress to a smaller using wia than one that only takes up only 100 mb or so? It only seems logical.

 

[Wiimm]

I have yet not done much experiments with raw copies. I never use them because I'm sure that well1 scrubbed images are the right way. So I don't see the advantage of raw dumps at this moment. The disadvantge is that non scrubbed games consumes allways much disc spase. Is there any known game that doesn't run in well1 scrubbed mode?

1Remark: I use the word "well" before "scrubbed" because I know, that some old tools works wrong when scrubbing. 1.) There was an error in the old wiidisc lib and 2.) some tools forget to rewrite the partition table.

--- snip ---

Here is an other statistic:

Summary of RLRP4F, Tomb Raider: Anniversary:

ÂÂ2131723546ÂÂ64.18%ÂÂ 18:34.390 m:sÂÂWIA/NONE + 7Z
ÂÂ2366324457ÂÂ71.24%ÂÂ 20:21.218 m:sÂÂWIA/LZMA.9
ÂÂ2366648574ÂÂ71.25%ÂÂ 20:20.319 m:sÂÂWIA/LZMA2.9
ÂÂ2366869775ÂÂ71.26%ÂÂ 18:26.365 m:sÂÂWIA/LZMA.5
ÂÂ2367264288ÂÂ71.27%ÂÂ 18:29.362 m:sÂÂWIA/LZMA2.5
ÂÂ2464080815ÂÂ74.18%ÂÂÂÂ9:50.797 m:sÂÂWIA/LZMA.1
ÂÂ2464467418ÂÂ74.20%ÂÂ 10:04.271 m:sÂÂWIA/LZMA2.1
ÂÂ2610128645ÂÂ78.58%ÂÂ 12:10.654 m:sÂÂWIA/BZIP2.1
ÂÂ2614198548ÂÂ78.70%ÂÂ 13:35.218 m:sÂÂWIA/BZIP2.5
ÂÂ2615893189ÂÂ78.76%ÂÂ 15:05.526 m:sÂÂWIA/BZIP2.9
ÂÂ2616222433ÂÂ78.77%ÂÂ 16:19.569 m:sÂÂWIA/NONE + BZIP2
ÂÂ3073221032ÂÂ92.53%ÂÂÂÂ2:21.765 m:sÂÂWIA/PURGE
ÂÂ3219977080ÂÂ96.94%ÂÂÂÂ2:19.033 m:sÂÂWIA/NONE
ÂÂ3244354612ÂÂ97.68%ÂÂÂÂ2:12.695 m:sÂÂWDF/DECRYPT
ÂÂ3321312056 100.00%ÂÂÂÂ1:33.369 m:sÂÂWDF
ÂÂ3333941547 100.38%ÂÂ 19:16.613 m:sÂÂWDF + BZIP2
ÂÂ3362583983 101.24%ÂÂ 18:59.207 m:sÂÂWDF + 7Z

I compared compression levels 1 + 5 +9 for BZIP2 and LZMA(2). I have analyzed more games in this way and selected Anniversary as a typical example. My conclusions:[*] In general LZMA is a little bit better than LZMA2 and clear better than BZIP2. Sometimes LZMA2 or BZIP2 beat LZMA, but only minimal. And this means: LZMA is the best (and the default) compression method for WIA.
.
[*] Somtetimes a smaller BZIP2 level is the better one. I don't know why?
.
[*] With 64-bit binaries on 64-bit linux BZIP2 seems to be faster than LZMA. With 32-bit binaries LZMA beats BZIP2.
.
[*] The general design of WIA is to enable random access to the image to support fast reading with the WIT tools. Because of this the data is splitted in chunks of 2 MiB and each chunk is compressed independent of others. In the last release I have added the data field 'chunk_size' into the WIA format. This make it easy to experiment with other chunk sizes.
.
[*] If you want the best compression at all than store your file in WIA/NONE and compress it with an external packer (like 7zip). The NONE format is designed for this use and minimize the management info. If you want WIA check sums than use WIA/PURGE instead. The compressed files are minimal larger. Before reading such image must be completely uncompressed.
.
[*] WIA is also good for raw copies because the decryption allow much better compression.

 

[wishmasterf]

LZMA and LZMA2 (7zip main compressors) implemented!! See beta release.
Whats the difference between LZMA2 an 7zip?

• The general design of WIA is to enable random access to the image to support fast reading with the WIT tools. Because of this the data is splitted in chunks of 2 MiB and each chunk is compressed independent of others. In the last release I have added the data field 'chunk_size' into the WIA format. This make it easy to experiment with other chunk sizes.
  .

Do this mean a implementation into USB-Loaders could be possible with useable speed? Maybe an extension to the wbfs-filesystem would be great to store them.

QUOTE(Wiimm @ Sep 18 2010, 03:35 PM)

• If you want the best compression at all than store your file in WIA/NONE and compress it with an external packer (like 7zip). The NONE format is designed for this use and minimize the management info. If you want WIA check sums than use WIA/PURGE instead. The compressed files are minimal larger. Before reading such image must be completely uncompressed.
  .

Is there any possibility to implement full 7zip support? If it is done by your tool it is more handy to use.