Hacking WIA = Wii ISO Archive

[Wiimm]

I have developed a new WIi ISO disc format:
WIA = Wii ISO Archive
This article describes the background and some aspects of the implementation.


Classic File formats

A PLAIN ISO file stores the content of the DVD byte by byte. It has a usual size of 4.7 GB. Most WBFS and ISO tools are able to detect unused areas (=holes) and fill them with a zeros. This method is called scrubbing. There are 4 classic formats:

• PLAIN ISO (1:1 copy)
• CISO (Compressed ISO, ignore unused blocks)
• WBFS (Multi ISO container, ignore unused blocks)
• WDF (Wii Disc File, mange holes directly)

A game like Animal crossing (my example for this article) has only about 350 MB real data.

-----------------------------------------------
size   percent   creation time  format
-----------------------------------------------
352695380 100.00%       8.939 sec  WDF
358645760 101.68%      14.946 sec  CISO
360448000 102.19%       8.842 sec  WBFS
4699979776   1332%       8.798 sec  PLAIN ISO
-----------------------------------------------

You see, that WDF, CISO and WBFS are similar small, because they do almost the same with different containers. WDF is the smallest format, but not playable by USB loaders. I set the WDF size to 100%.


Compressing with BZIP2 and RAR

ISO images can be compressed with with tools like bzip2 or rar. But this is very ineffective because most of the data is encrypted and encrypted data looks like random data and compressors can't compress random data. See this table:

------------------------------------------------------------
size   percent   creation time  format
------------------------------------------------------------
352695380 100.00%       8.939 sec  WDF
353345496 100.18%    6:24.597 m:s  WDF + RAR
354147564 100.41%    6:04.515 m:s  PLAIN ISO + BZIP2
354162098 100.41%    3:19.670 m:s  WDF + BZIP2
355472608 100.78%   12:52.996 m:s  PLAIN ISO + RAR
4699979776   1332%       8.798 sec  PLAIN ISO
------------------------------------------------------------

WDF is a little better as BIZP2 or RAR, but don't need as much time as the compressors. This is the main reason that I had developed WDF. WDF manage the holes and allow random access and is easy expandable.


Decrypted images

3-4 month ago I made test with decrypted images. If encrypting a decrypted images the result is the original ISO image. The advantage of decrypted images are that data don't looks like random data and that make decrypted images compressible:

------------------------------------------------------------
size   percent   creation time  format
------------------------------------------------------------
84717686  24.02%    2:40.267 m:s  WDF/DECRYPT + RAR
133009483  37.71%    2:40.254 m:s  WDF/DECRYPT + BZIP2
330201884  93.62%      25.208 sec  WDF/DECRYPT
352695380 100.00%       8.939 sec  WDF
353345496 100.18%    6:24.597 m:s  WDF + RAR
354162098 100.41%    3:19.670 m:s  WDF + BZIP2
------------------------------------------------------------

Since that time I know that decrypting and compressing is the best way to keep Wii discs small. Decrypted images contain all the hash data which takes 3% of the partition data. Hash data looks also like random data and is not compressible. But the hash values are reproducible from the war data. So the best is to throw it away.


WIA = Wii ISO Archive

With all this knowledge I have designed a new Wii disc image format: WIA = Wii ISO Archive. My main interest is to find out, what is possible. I don't know If I ever convert my few games into WIA because the handling is much slower than WDF and I have enough space on my drives. The WIT tools already support this new format, long term tests are running and a first beta will be released in the next days.

A WIA archive allows random access reading and is implemented in the file layer of the WIT tools. The WIA files can be read and written like the other supported formats (ISO,CISO,WDF,WBFS). Only the direct editing and patching is not possible. Converting a WIA into a PLAIN ISO restores all non scrubbed data including wrong hash values.

I support 2 compression formats: Uncompressed and compressed with bzip2 (default). My decision for bzip2 was easy: Easy to use lib for all systems and good compression ratio for small (2MB) chunks.

Here is the size statistic:

------------------------------------------------------------
size   percent   creation time  format
------------------------------------------------------------
79942145  22.66%    2:41.197 m:s  WIA/NOCOMPRESS + RAR
84717686  24.02%    2:40.267 m:s  WDF/DECRYPT + RAR
127436075  36.13%    2:36.284 m:s  WIA/NOCOMPRESS + BZIP2
127603521  36.17%    4:43.668 m:s  WIA + RAR
127696363  36.20%    2:31.820 m:s  WIA
128007670  36.29%    3:40.140 m:s  WIA + BZIP2
133009483  37.71%    2:40.254 m:s  WDF/DECRYPT + BZIP2
312976812  88.73%      27.094 sec  WIA/NOCOMPRESS
330201884  93.62%      25.208 sec  WDF/DECRYPT
352695380 100.00%       8.939 sec  WDF
353345496 100.18%    6:24.597 m:s  WDF + RAR
354162098 100.41%    3:19.670 m:s  WDF + BZIP2
------------------------------------------------------------

The WIA compression is really good and an additional compression with BZIP2 or RAR has no positive effect. The smallest images are made by creating uncompressed WIA images and then compressing it with an external tool. You see also that WIA/NOCOMPRESS is better than WDF/DECRYPT because of throwing away unneeded hash values.


Summary

Last not least all formats in one table:

Summary of RUUP01, Animal Crossing: Let's Go to the City:

------------------------------------------------------------
size   percent   creation time  format
------------------------------------------------------------
79942145  22.66%    2:41.197 m:s  WIA/NOCOMPRESS + RAR
84717686  24.02%    2:40.267 m:s  WDF/DECRYPT + RAR
127436075  36.13%    2:36.284 m:s  WIA/NOCOMPRESS + BZIP2
127603521  36.17%    4:43.668 m:s  WIA + RAR
127696363  36.20%    2:31.820 m:s  WIA
128007670  36.29%    3:40.140 m:s  WIA + BZIP2
133009483  37.71%    2:40.254 m:s  WDF/DECRYPT + BZIP2
312976812  88.73%      27.094 sec  WIA/NOCOMPRESS
330201884  93.62%      25.208 sec  WDF/DECRYPT
352695380 100.00%       8.939 sec  WDF
353345496 100.18%    6:24.597 m:s  WDF + RAR
354147564 100.41%    6:04.515 m:s  PLAIN ISO + BZIP2
354162098 100.41%    3:19.670 m:s  WDF + BZIP2
355472608 100.78%   12:52.996 m:s  PLAIN ISO + RAR
358645760 101.68%      14.946 sec  CISO
360448000 102.19%       8.842 sec  WBFS
4699979776   1332%       8.798 sec  PLAIN ISO
------------------------------------------------------------

Other images

Animal crossing is an extreme example. The image is reducible to 23%. Other games are not so good for compression. The next table shows the WIA rates of some other games. And remember: 100% is the WDF size of a scrubbed image.

8054276  31.98%      24.213 sec  WIA  RTYP01, Wii Chess
127696363  36.20%    3:00.483 m:s  WIA  RUUP01, Animal Crossing
531722793  80.26%    5:25.284 m:s  WIA  RSPP01, Wii Sports
153673062  86.74%    1:35.435 m:s  WIA  RHAP01, Wii Play
1010766389  87.80%    9:36.026 m:s  WIA  RZDP01, Twilight Princess
2567636439  92.60%   25:50.572 m:s  WIA  RMCP01, Mario Kart Wii

 

[MelodieOctavia]

That's nice, but....why?

 

[Wiimm]

That's nice, but....why?

It was a provocation of my subconscious for the thinking part of my brain and I like programming such things.
(And now I can sleep again

)

Why not?

 

[XFlak]

Why not indeed! I can't see this hurting anyone... although I doubt it will be used often (but I'm sure u knew that already).

Is there anything unique about this format? I read your description, so I'm not just blindly asking. Basically, I'm curious if there is a certain app or loader or something that this format could help lead to improvements and/or more possibilities. Or is this format just another way to store/compress Wii ISOs for now?

 

[W hat]

I guess I'm the only one who thought this was a good idea, if you remember my posts about compressing FST-dumps with 7z. In fact, besides patching, this is the best feature WIT will ever get.

I almost asked for built-in compressor support, but I didn't ask because I wasn't sure if you would want to. I also wasn't sure how long it would take to code.

Is 7-zip out of the question? 7zip (Ultra) is somewhat overkill, but maybe it would be worth testing with normal, fast, or fastest.

That data is 7z (ultra) and Winrar (best).

 

[Wiimm]

@XFlak

The main goal is to save disc space in an backup archive. Because of the compression reading data is slow. For me it's more an academic feature because I will never use it. The attraction for me was to implement it, because I'm thinking about this since 3-4 month, and to find out the compression ratio of different discs.

BTW: My beta repository contains the source code: lib-wia.h and llib-wia.c

 

[s3phir0th115]

This looks very logical actually. I'd fully support the format so long as it be converted to other formats and be useful as more than just better size, but it looks like you've taken care of that already.

 

[Rydian]

Just to clarify, there's not a way to get the wii to directly load these and it's just for archival purposes on a PC, correct?

 

[vexing]

nice work, but i agree with w hat, you should try 7zip's compression...
it'd probably end up half the current size
http://www.7-zip.org/sdk.html

also, if you need a new project, how about a wbfs to fat32 file system converter.

 

[dodol]

nice work, but i agree with w hat, you should try 7zip's compression...
it'd probably end up half the current size
http://www.7-zip.org/sdk.html

also, if you need a new project, how about a wbfs to fat32 file system converter.

I thought 7zip compression = good compression, slow extract/compress
is it change nowadays ?

 

[W hat]

nice work, but i agree with w hat, you should try 7zip's compression...
it'd probably end up half the current size
http://www.7-zip.org/sdk.html

also, if you need a new project, how about a wbfs to fat32 file system converter.

I thought 7zip compression = good compression, slow extract/compress
is it change nowadays ?

It's true, but "normal" or "fast" mode in 7zip might be okay.

 

[Wiimm]

I guess I'm the only one who thought this was a good idea, if you remember my posts about compressing FST-dumps with 7z. In fact, besides patching, this is the best feature WIT will ever get.

I almost asked for built-in compressor support, but I didn't ask because I wasn't sure if you would want to. I also wasn't sure how long it would take to code.

Is 7-zip out of the question? 7zip (Ultra) is somewhat overkill, but maybe it would be worth testing with normal, fast, or fastest.

That data is 7z (ultra) and Winrar (best).

Here are the results of a size/time test with a scrubbed (data partition only) Wii Play disc to compare it with W hat data:
CODESummary of RHAP01, Wii Play:

ÂÂÂÂ52934196ÂÂ52.07%ÂÂÂÂÂÂ56.550 secÂÂWIA/NOCOMPRESS + 7Z
ÂÂÂÂ55081533ÂÂ54.18%ÂÂÂÂÂÂ56.444 secÂÂWDF/DECRYPT + 7Z
ÂÂÂÂ62592874ÂÂ61.57%ÂÂÂÂ1:04.793 m:sÂÂWIA/NOCOMPRESS + RAR
ÂÂÂÂ65496994ÂÂ64.42%ÂÂÂÂ1:06.582 m:sÂÂWDF/DECRYPT + RAR
ÂÂÂÂ80301483ÂÂ78.99%ÂÂÂÂÂÂ50.565 secÂÂWIA
ÂÂÂÂ80354061ÂÂ79.04%ÂÂÂÂÂÂ51.282 secÂÂWIA/NOCOMPRESS + BZIP2
ÂÂÂÂ80426780ÂÂ79.11%ÂÂÂÂ2:12.930 m:sÂÂWIA + RAR
ÂÂÂÂ80581068ÂÂ79.26%ÂÂÂÂ1:33.696 m:sÂÂWIA + BZIP2
ÂÂÂÂ81215012ÂÂ79.88%ÂÂÂÂ1:27.386 m:sÂÂWIA + 7Z
ÂÂÂÂ82908135ÂÂ81.55%ÂÂÂÂÂÂ47.266 secÂÂWDF/DECRYPT + BZIP2
ÂÂÂÂ99358124ÂÂ97.73%ÂÂÂÂÂÂ 8.297 secÂÂWIA/NOCOMPRESS
ÂÂ 101431468ÂÂ99.77%ÂÂÂÂÂÂ 7.496 secÂÂWDF/DECRYPT
ÂÂ 101659348 100.00%ÂÂÂÂÂÂ 2.626 secÂÂWDF
ÂÂ 101922073 100.25%ÂÂÂÂ1:50.465 m:sÂÂWDF + RAR
ÂÂ 102111030 100.44%ÂÂÂÂÂÂ57.631 secÂÂWDF + BZIP2
ÂÂ 103038269 101.35%ÂÂÂÂÂÂ50.475 secÂÂWDF + 7Z

Interpret it by yourself

 

[WiiPower]

Wiimm do you know how the game loading works in CFG? I mean with the offset list and stuff.

I think you should find the smallest format that:
1. Contains all data the scrubbed image contains, and extracting and/or converting it results in a 100% match to the scrubbed image.
2. Can be read by usb loaders, or to be precise by the cIOS modules of them.
This could be used for storage and game loading. A solution with good compression for storage is nice, but if you need another format for gaming, you need your games 2x, and the advantage is lost.

Removing the hashes and using decrypted images should work for both. But what about an easy(implementable with VERY FEW code) compression that allows random access of the .isos? You were speaking of 2 MB chunks, is that the smallest that can be used? I think 2 MB is still too big to be usable from within the cIOS. Or wouldn't that mean if data was at offset 1.8MB of a 2MB chunk, that the cIOS needs to read the whole 2MB, then decompress it, and then it can return the 0.2 MB?

PS: Sorry for expecting so much from you, but you seem to be the only one working on anything in that direction
PPS: If you find such a format, i'm sure we can convince team CFG to implement it in CFG

 

[Wiimm]

@WiiPower

First, I know nothing about USB loading, put perhaps this will change in the future.

Chunk size: If I remember the details of bzip2 it works always with independent chunks between 100k and 900k (options -1 .. -9) and decompression is relative fast while compression is slow. My decision for 2MiB chunks (that bzip2 divides internal in 900KB chunks) is the natural structure of of sector groups (1 group = 64 sectors a 32 KiB), but it is not mandatory. By now the WIA will support any chunk size with only little modifications.

PS: Sorry for expecting so much from you, but you seem to be the only one working on anything in that direction

I like such provocations (good for brain storming) as long as I can decide what I do.

 

[Wiimm]

I have released a binary beta of WIT with a first WIA support:
http://gbatemp.net/index.php?act=findpost&pid=3077163

 

[WiiPower]

Do your tools allow to defrag the files? I know it's a complicated task, but it could improve the reading speeds on some setups.

Also i'm asking because i have a new .iso tool for GC games in mind, and that would require defrag code. Would you be interested in writing tools for GC games?

 

[Wiimm]

@WiiPower

On my personal road map there are some extensions. One of it is adding and replacing files (removing is already implemented). For this I need (partial) remapping of the files. And trimming is also a kind of remapping. And if I implement this it is usual for me to do this with many options. So far I have not yet thought about the details.

I have never seen a GC disc and I don't know anything about the data structure. I thing that GC disc support is not my goal. But it does not hurt if you can give me technical info about GC discs.

 

[Tagg7]

Does this mean that you are able to convert unscrubbed Wii ISOs (aka untouched) into WIA format and back again? This has always been a problem in the Wii scene (that all releases are 4.7GB because of the encrypted filler data).

 

[giantpune]

@ wimm
the structure for GC games is very similar to a wii partition. you use the same method to find the fst.bin, apploader, dol etc. the main difference in GC games is that there is only 1 partition, there is no ticket/tmd/cert, there is no data encryption, and offsets are exactly as you read them from the disc ( no need to do "

 

[Fat D]

Does this mean that you are able to convert unscrubbed Wii ISOs (aka untouched) into WIA format and back again? This has always been a problem in the Wii scene (that all releases are 4.7GB because of the encrypted filler data).

That would require the garbage data to make any sense when decrypted - but it does not, that is why it is called garbage. Unless some game comes out that uses some of the garbage data for copy-protection, keeping the data, by the way, would not make any technical sense at all and is just some elitist scene rule like using certain distribution methods, selecting the appropriate part size, compression and so on.