WIA = Wii ISO Archive

Discussion in 'Wii - Hacking' started by Wiimm, Aug 27, 2010.

Aug 27, 2010

WIA = Wii ISO Archive by Wiimm at 8:03 PM (99,429 Views / 1 Likes) 169 Comments

    • Member

    Wiimm Developer

    Member Since:
    Aug 11, 2009
    Message Count:
    1,809
    Location:
    Germany
    Country:
    Germany
    I have developed a new WIi ISO disc format:
    WIA = Wii ISO Archive
    This article describes the background and some aspects of the implementation.


    Classic File formats

    A PLAIN ISO file stores the content of the DVD byte by byte. It has a usual size of 4.7 GB. Most WBFS and ISO tools are able to detect unused areas (=holes) and fill them with a zeros. This method is called scrubbing. There are 4 classic formats:
    • PLAIN ISO (1:1 copy)
    • CISO (Compressed ISO, ignore unused blocks)
    • WBFS (Multi ISO container, ignore unused blocks)
    • WDF (Wii Disc File, mange holes directly)
    A game like Animal crossing (my example for this article) has only about 350 MB real data.
    Code:
    -----------------------------------------------
    size   percent   creation time  format
    -----------------------------------------------
    352695380 100.00%       8.939 sec  WDF
    358645760 101.68%      14.946 sec  CISO
    360448000 102.19%       8.842 sec  WBFS
    4699979776   1332%       8.798 sec  PLAIN ISO
    -----------------------------------------------
    
    You see, that WDF, CISO and WBFS are similar small, because they do almost the same with different containers. WDF is the smallest format, but not playable by USB loaders. I set the WDF size to 100%.


    Compressing with BZIP2 and RAR

    ISO images can be compressed with with tools like bzip2 or rar. But this is very ineffective because most of the data is encrypted and encrypted data looks like random data and compressors can't compress random data. See this table:
    Code:
    ------------------------------------------------------------
    size   percent   creation time  format
    ------------------------------------------------------------
    352695380 100.00%       8.939 sec  WDF
    353345496 100.18%    6:24.597 m:s  WDF + RAR
    354147564 100.41%    6:04.515 m:s  PLAIN ISO + BZIP2
    354162098 100.41%    3:19.670 m:s  WDF + BZIP2
    355472608 100.78%   12:52.996 m:s  PLAIN ISO + RAR
    4699979776   1332%       8.798 sec  PLAIN ISO
    ------------------------------------------------------------
    
    WDF is a little better as BIZP2 or RAR, but don't need as much time as the compressors. This is the main reason that I had developed WDF. WDF manage the holes and allow random access and is easy expandable.


    Decrypted images

    3-4 month ago I made test with decrypted images. If encrypting a decrypted images the result is the original ISO image. The advantage of decrypted images are that data don't looks like random data and that make decrypted images compressible:
    Code:
    ------------------------------------------------------------
    size   percent   creation time  format
    ------------------------------------------------------------
    84717686  24.02%    2:40.267 m:s  WDF/DECRYPT + RAR
    133009483  37.71%    2:40.254 m:s  WDF/DECRYPT + BZIP2
    330201884  93.62%      25.208 sec  WDF/DECRYPT
    352695380 100.00%       8.939 sec  WDF
    353345496 100.18%    6:24.597 m:s  WDF + RAR
    354162098 100.41%    3:19.670 m:s  WDF + BZIP2
    ------------------------------------------------------------
    
    Since that time I know that decrypting and compressing is the best way to keep Wii discs small. Decrypted images contain all the hash data which takes 3% of the partition data. Hash data looks also like random data and is not compressible. But the hash values are reproducible from the war data. So the best is to throw it away.


    WIA = Wii ISO Archive

    With all this knowledge I have designed a new Wii disc image format: WIA = Wii ISO Archive. My main interest is to find out, what is possible. I don't know If I ever convert my few games into WIA because the handling is much slower than WDF and I have enough space on my drives. The WIT tools already support this new format, long term tests are running and a first beta will be released in the next days.

    A WIA archive allows random access reading and is implemented in the file layer of the WIT tools. The WIA files can be read and written like the other supported formats (ISO,CISO,WDF,WBFS). Only the direct editing and patching is not possible. Converting a WIA into a PLAIN ISO restores all non scrubbed data including wrong hash values.

    I support 2 compression formats: Uncompressed and compressed with bzip2 (default). My decision for bzip2 was easy: Easy to use lib for all systems and good compression ratio for small (2MB) chunks.

    Here is the size statistic:
    Code:
    ------------------------------------------------------------
    size   percent   creation time  format
    ------------------------------------------------------------
    79942145  22.66%    2:41.197 m:s  WIA/NOCOMPRESS + RAR
    84717686  24.02%    2:40.267 m:s  WDF/DECRYPT + RAR
    127436075  36.13%    2:36.284 m:s  WIA/NOCOMPRESS + BZIP2
    127603521  36.17%    4:43.668 m:s  WIA + RAR
    127696363  36.20%    2:31.820 m:s  WIA
    128007670  36.29%    3:40.140 m:s  WIA + BZIP2
    133009483  37.71%    2:40.254 m:s  WDF/DECRYPT + BZIP2
    312976812  88.73%      27.094 sec  WIA/NOCOMPRESS
    330201884  93.62%      25.208 sec  WDF/DECRYPT
    352695380 100.00%       8.939 sec  WDF
    353345496 100.18%    6:24.597 m:s  WDF + RAR
    354162098 100.41%    3:19.670 m:s  WDF + BZIP2
    ------------------------------------------------------------
    
    The WIA compression is really good and an additional compression with BZIP2 or RAR has no positive effect. The smallest images are made by creating uncompressed WIA images and then compressing it with an external tool. You see also that WIA/NOCOMPRESS is better than WDF/DECRYPT because of throwing away unneeded hash values.


    Summary

    Last not least all formats in one table:
    Code:
    Summary of RUUP01, Animal Crossing: Let's Go to the City:
    
    ------------------------------------------------------------
    size   percent   creation time  format
    ------------------------------------------------------------
    79942145  22.66%    2:41.197 m:s  WIA/NOCOMPRESS + RAR
    84717686  24.02%    2:40.267 m:s  WDF/DECRYPT + RAR
    127436075  36.13%    2:36.284 m:s  WIA/NOCOMPRESS + BZIP2
    127603521  36.17%    4:43.668 m:s  WIA + RAR
    127696363  36.20%    2:31.820 m:s  WIA
    128007670  36.29%    3:40.140 m:s  WIA + BZIP2
    133009483  37.71%    2:40.254 m:s  WDF/DECRYPT + BZIP2
    312976812  88.73%      27.094 sec  WIA/NOCOMPRESS
    330201884  93.62%      25.208 sec  WDF/DECRYPT
    352695380 100.00%       8.939 sec  WDF
    353345496 100.18%    6:24.597 m:s  WDF + RAR
    354147564 100.41%    6:04.515 m:s  PLAIN ISO + BZIP2
    354162098 100.41%    3:19.670 m:s  WDF + BZIP2
    355472608 100.78%   12:52.996 m:s  PLAIN ISO + RAR
    358645760 101.68%      14.946 sec  CISO
    360448000 102.19%       8.842 sec  WBFS
    4699979776   1332%       8.798 sec  PLAIN ISO
    ------------------------------------------------------------
    


    Other images

    Animal crossing is an extreme example. The image is reducible to 23%. Other games are not so good for compression. The next table shows the WIA rates of some other games. And remember: 100% is the WDF size of a scrubbed image.

    Code:
    8054276  31.98%      24.213 sec  WIA  RTYP01, Wii Chess
    127696363  36.20%    3:00.483 m:s  WIA  RUUP01, Animal Crossing
    531722793  80.26%    5:25.284 m:s  WIA  RSPP01, Wii Sports
    153673062  86.74%    1:35.435 m:s  WIA  RHAP01, Wii Play
    1010766389  87.80%    9:36.026 m:s  WIA  RZDP01, Twilight Princess
    2567636439  92.60%   25:50.572 m:s  WIA  RMCP01, Mario Kart Wii
    
    tmv_josue likes this.


    • Global Moderator

    TwinRetro Don't start nothin', Won't be nothin'

    Member Since:
    Aug 29, 2008
    Message Count:
    6,177
    Location:
    Las Vegas, NV
    Country:
    United States
    That's nice, but....why?
    1 people like this.
    • Member

    Wiimm Developer

    Member Since:
    Aug 11, 2009
    Message Count:
    1,809
    Location:
    Germany
    Country:
    Germany
    It was a provocation of my subconscious for the thinking part of my brain and I like programming such things.
    (And now I can sleep again [IMG] )

    Why not?
    • Member

    XFlak Considering Wii-tirement...

    Member Since:
    Sep 12, 2009
    Message Count:
    9,095
    Location:
    Ontario
    Country:
    Canada
    Why not indeed! I can't see this hurting anyone... although I doubt it will be used often (but I'm sure u knew that already).

    Is there anything unique about this format? I read your description, so I'm not just blindly asking. Basically, I'm curious if there is a certain app or loader or something that this format could help lead to improvements and/or more possibilities. Or is this format just another way to store/compress Wii ISOs for now?
    • Member

    W hat Rhythm Heaven Fan

    Member Since:
    Feb 28, 2007
    Message Count:
    568
    Country:
    United States
    I guess I'm the only one who thought this was a good idea, if you remember my posts about compressing FST-dumps with 7z. In fact, besides patching, this is the best feature WIT will ever get.

    I almost asked for built-in compressor support, but I didn't ask because I wasn't sure if you would want to. I also wasn't sure how long it would take to code.

    Is 7-zip out of the question? 7zip (Ultra) is somewhat overkill, but maybe it would be worth testing with normal, fast, or fastest.

    [IMG]

    That data is 7z (ultra) and Winrar (best).
    • Member

    Wiimm Developer

    Member Since:
    Aug 11, 2009
    Message Count:
    1,809
    Location:
    Germany
    Country:
    Germany
    @XFlak

    The main goal is to save disc space in an backup archive. Because of the compression reading data is slow. For me it's more an academic feature because I will never use it. The attraction for me was to implement it, because I'm thinking about this since 3-4 month, and to find out the compression ratio of different discs.

    BTW: My beta repository contains the source code: lib-wia.h and llib-wia.c
    • Member

    s3phir0th115 New Member

    Member Since:
    Dec 31, 2008
    Message Count:
    690
    Country:
    United States
    This looks very logical actually. I'd fully support the format so long as it be converted to other formats and be useful as more than just better size, but it looks like you've taken care of that already.
    • Member

    Rydian Resident Furvert™

    Member Since:
    Feb 4, 2010
    Message Count:
    27,886
    Location:
    Cave Entrance, Watching Cyan Write Letters
    Country:
    United States
    Just to clarify, there's not a way to get the wii to directly load these and it's just for archival purposes on a PC, correct?
    • Newcomer

    vexing New Member

    Member Since:
    Nov 16, 2009
    Message Count:
    65
    Country:
    United States
    nice work, but i agree with w hat, you should try 7zip's compression...
    it'd probably end up half the current size
    http://www.7-zip.org/sdk.html

    also, if you need a new project, how about a wbfs to fat32 file system converter.
    • Member

    dodol New Member

    Member Since:
    Sep 24, 2007
    Message Count:
    121
    Country:
    United States
    I thought 7zip compression = good compression, slow extract/compress
    is it change nowadays ?
    • Member

    W hat Rhythm Heaven Fan

    Member Since:
    Feb 28, 2007
    Message Count:
    568
    Country:
    United States
    It's true, but "normal" or "fast" mode in 7zip might be okay.
    • Member

    Wiimm Developer

    Member Since:
    Aug 11, 2009
    Message Count:
    1,809
    Location:
    Germany
    Country:
    Germany
    Interpret it by yourself [IMG]
    • Member

    WiiPower New Member

    Member Since:
    Oct 17, 2008
    Message Count:
    8,165
    Country:
    Germany
    Wiimm do you know how the game loading works in CFG? I mean with the offset list and stuff.

    I think you should find the smallest format that:
    1. Contains all data the scrubbed image contains, and extracting and/or converting it results in a 100% match to the scrubbed image.
    2. Can be read by usb loaders, or to be precise by the cIOS modules of them.
    This could be used for storage and game loading. A solution with good compression for storage is nice, but if you need another format for gaming, you need your games 2x, and the advantage is lost.

    Removing the hashes and using decrypted images should work for both. But what about an easy(implementable with VERY FEW code) compression that allows random access of the .isos? You were speaking of 2 MB chunks, is that the smallest that can be used? I think 2 MB is still too big to be usable from within the cIOS. Or wouldn't that mean if data was at offset 1.8MB of a 2MB chunk, that the cIOS needs to read the whole 2MB, then decompress it, and then it can return the 0.2 MB?

    PS: Sorry for expecting so much from you, but you seem to be the only one working on anything in that direction
    PPS: If you find such a format, i'm sure we can convince team CFG to implement it in CFG
    • Member

    Wiimm Developer

    Member Since:
    Aug 11, 2009
    Message Count:
    1,809
    Location:
    Germany
    Country:
    Germany
    @WiiPower

    First, I know nothing about USB loading, put perhaps this will change in the future.

    Chunk size: If I remember the details of bzip2 it works always with independent chunks between 100k and 900k (options -1 .. -9) and decompression is relative fast while compression is slow. My decision for 2MiB chunks (that bzip2 divides internal in 900KB chunks) is the natural structure of of sector groups (1 group = 64 sectors a 32 KiB), but it is not mandatory. By now the WIA will support any chunk size with only little modifications.

    I like such provocations (good for brain storming) as long as I can decide what I do.
    • Member

    Wiimm Developer

    Member Since:
    Aug 11, 2009
    Message Count:
    1,809
    Location:
    Germany
    Country:
    Germany
    • Member

    WiiPower New Member

    Member Since:
    Oct 17, 2008
    Message Count:
    8,165
    Country:
    Germany
    Do your tools allow to defrag the files? I know it's a complicated task, but it could improve the reading speeds on some setups.

    Also i'm asking because i have a new .iso tool for GC games in mind, and that would require defrag code. Would you be interested in writing tools for GC games?
    • Member

    Wiimm Developer

    Member Since:
    Aug 11, 2009
    Message Count:
    1,809
    Location:
    Germany
    Country:
    Germany
    @WiiPower

    On my personal road map there are some extensions. One of it is adding and replacing files (removing is already implemented). For this I need (partial) remapping of the files. And trimming is also a kind of remapping. And if I implement this it is usual for me to do this with many options. So far I have not yet thought about the details.

    I have never seen a GC disc and I don't know anything about the data structure. I thing that GC disc support is not my goal. But it does not hurt if you can give me technical info about GC discs.
    • Member

    Tagg7 New Member

    Member Since:
    Jun 24, 2007
    Message Count:
    394
    Country:
    United States
    Does this mean that you are able to convert unscrubbed Wii ISOs (aka untouched) into WIA format and back again? This has always been a problem in the Wii scene (that all releases are 4.7GB because of the encrypted filler data).
    • Member

    giantpune New Member

    Member Since:
    Apr 10, 2009
    Message Count:
    2,860
    Country:
    United States
    @ wimm
    the structure for GC games is very similar to a wii partition. you use the same method to find the fst.bin, apploader, dol etc. the main difference in GC games is that there is only 1 partition, there is no ticket/tmd/cert, there is no data encryption, and offsets are exactly as you read them from the disc ( no need to do "
    • Member

    Fat D New Member

    Member Since:
    Nov 18, 2006
    Message Count:
    1,097
    Country:
    Germany
    That would require the garbage data to make any sense when decrypted - but it does not, that is why it is called garbage. Unless some game comes out that uses some of the garbage data for copy-protection, keeping the data, by the way, would not make any technical sense at all and is just some elitist scene rule like using certain distribution methods, selecting the appropriate part size, compression and so on.

SPONSORED LINKS
 

Share This Page