WIA = Wii ISO Archive

Discussion in 'Wii - Hacking' started by Wiimm, Aug 27, 2010.

  1. moosehunter

    moosehunter GBAtemp Regular

    Member
    199
    1
    Nov 26, 2008
    United States
    Do you think that this format would work well if it was implemented in Dolphin?
     


  2. Wiimm
    OP

    Wiimm Developer

    Member
    2,121
    334
    Aug 11, 2009
    Gambia, The
    Germany
    Here are some ratings to compare the old and new WIA implementation. The images compressed with the new algorithm is a little better:

    Code:
    Summary of RUUP01, Animal Crossing: Let's Go to the City:
    
    ÂÂ 123325063ÂÂ34.96%ÂÂÂÂ3:38.056 m:sÂÂ+ WIA/NONE + BZIP2
    ÂÂ 123687704ÂÂ35.06%ÂÂÂÂ3:16.845 m:sÂÂ+ WIA/BZIP2
    ÂÂ 124191993ÂÂ35.21%ÂÂÂÂ4:26.899 m:sÂÂ+ WIA/BZIP2 + BZIP2
    ÂÂ 127243676ÂÂ36.07%ÂÂÂÂ3:23.276 m:sÂÂ+ WIA/PURGE + BZIP2
    ÂÂ 127436075ÂÂ36.13%ÂÂÂÂ2:36.284 m:sÂÂold WIA/NOCOMPRESS + BZIP2
    ÂÂ 127696363ÂÂ36.20%ÂÂÂÂ2:31.820 m:sÂÂold WIA/BZIP2
    ÂÂ 128007670ÂÂ36.29%ÂÂÂÂ3:40.140 m:sÂÂold WIA/BZIP2 + BZIP2
    ÂÂ 312521336ÂÂ88.60%ÂÂÂÂÂÂ52.595 secÂÂ+ WIA/PURGE
    ÂÂ 312976812ÂÂ88.73%ÂÂÂÂÂÂ27.094 secÂÂold WIA/NOCOMPRESS
    ÂÂ 347369520ÂÂ98.48%ÂÂÂÂÂÂ53.021 secÂÂ+ WIA/NONE
    Code:
    Summary of RHAP01, Wii Play:
    
    ÂÂÂÂ52899190ÂÂ52.03%ÂÂÂÂ1:16.932 m:sÂÂ+ WIA/NONE + 7Z
    ÂÂÂÂ52934196ÂÂ52.07%ÂÂÂÂÂÂ56.550 secÂÂold WIA/NOCOMPRESS + 7Z
    ÂÂÂÂ52935121ÂÂ52.07%ÂÂÂÂÂÂ59.450 secÂÂ+ WIA/PURGE + 7Z
    ÂÂÂÂ80179986ÂÂ78.87%ÂÂÂÂÂÂ50.465 secÂÂ+ WIA/BZIP2
    ÂÂÂÂ80299088ÂÂ78.98%ÂÂÂÂ1:31.244 m:sÂÂ+ WIA/NONE + BZIP2
    ÂÂÂÂ80301483ÂÂ78.99%ÂÂÂÂÂÂ50.565 secÂÂold WIA
    ÂÂÂÂ80354061ÂÂ79.04%ÂÂÂÂÂÂ51.282 secÂÂold WIA/NOCOMPRESS + BZIP2
    ÂÂÂÂ80420690ÂÂ79.10%ÂÂÂÂÂÂ54.095 secÂÂ+ WIA/PURGE + BZIP2
    ÂÂÂÂ80531997ÂÂ79.21%ÂÂÂÂ1:34.511 m:sÂÂ+ WIA/BZIP2 + BZIP2
    ÂÂÂÂ80581068ÂÂ79.26%ÂÂÂÂ1:33.696 m:sÂÂold WIA + BZIP2
    ÂÂÂÂ81167196ÂÂ79.84%ÂÂÂÂ1:28.863 m:sÂÂ+ WIA/BZIP2 + 7Z
    ÂÂÂÂ81215012ÂÂ79.88%ÂÂÂÂ1:27.386 m:sÂÂold WIA + 7Z
    ÂÂÂÂ98903128ÂÂ97.28%ÂÂÂÂÂÂ11.849 secÂÂ+ WIA/PURGE
    ÂÂÂÂ99358124ÂÂ97.73%ÂÂÂÂÂÂ 8.297 secÂÂold WIA/NOCOMPRESS
    ÂÂ 101458756ÂÂ99.80%ÂÂÂÂÂÂ29.764 secÂÂ+ WIA/NONE
    * Remark: "+ 7Z" or "+ BZIP2" mans: compress the resulting file with the external tool.

    Reading is ready too. At the moment a test script (takes many hours) is running.



    It will work, but 'well'? Decompression takes time and will slow down the game whenever it loads data drom the dsic.
     
  3. wishmasterf

    wishmasterf GBAtemp Regular

    Member
    167
    0
    Apr 20, 2009
    Australia
    Austria
    The difference between BZIP2 and 7z seems to be not small. Do not need an external tool make the use of the implemented BZIP2 more handy.

    To use 7z - Libary is not a choice?
     
  4. Wiimm
    OP

    Wiimm Developer

    Member
    2,121
    334
    Aug 11, 2009
    Gambia, The
    Germany
     
  5. wishmasterf

    wishmasterf GBAtemp Regular

    Member
    167
    0
    Apr 20, 2009
    Australia
    Austria

    Nice to know it is still on your roadmap. The Number "n" tells me it wont be in near future.
     
  6. Wiimm
    OP

    Wiimm Developer

    Member
    2,121
    334
    Aug 11, 2009
    Gambia, The
    Germany
    n and the ellipses before means: I don't know if some other more important things pop up.
     
  7. Wiimm
    OP

    Wiimm Developer

    Member
    2,121
    334
    Aug 11, 2009
    Gambia, The
    Germany
  8. ssjkakaroto

    ssjkakaroto Advanced Member

    Newcomer
    86
    0
    Dec 14, 2005
    Thanks Wiimm, it's working now [​IMG]

    Edit: Bah, the xdelta patch between the US and JAP versions of the game (using WIA) was >6 GiB :/
    At least it was smaller than the patch using the regular ISOs, which was >8 GiB.
     
  9. Wiimm
    OP

    Wiimm Developer

    Member
    2,121
    334
    Aug 11, 2009
    Gambia, The
    Germany
    Bug found:
    Creation of a WIA file fails if option --psel is set and at least one partition is removed (scrubbed). The result is an invalid disc image.
    • Don't use --psel when creating a WIA file.
    The beta repository contains an update -- no time to produce binaries.

    btw: lzma/lzma2 compressions seem easy ....
     
  10. hdhacker

    hdhacker GBAtemp Fan

    Member
    435
    0
    Apr 6, 2009
    United States
    I guess the lesson here is if you're going to use compression, don't try to help it. [​IMG]
     
  11. Wiimm
    OP

    Wiimm Developer

    Member
    2,121
    334
    Aug 11, 2009
    Gambia, The
    Germany
    Here are some compression rates. All games are scrubbed and all non DATA partitions are removed. 100% is the WDF size of the game:

    Code:
    ÂÂ-------------------------------------------------------------------------------
    ÂÂÂÂÂÂ bytesÂÂÂÂWDF%ÂÂÂÂcompr.timeÂÂÂÂfile typeÂÂid6ÂÂÂÂ game
    ÂÂ-------------------------------------------------------------------------------
    ÂÂÂÂ 7823978ÂÂ31.07%ÂÂÂÂÂÂ23.994 secÂÂWIA/BZIP2ÂÂRTYP01, Wii Chess
    ÂÂ 123687748ÂÂ35.06%ÂÂÂÂ2:43.546 m:sÂÂWIA/BZIP2ÂÂRUUP01, Animal Crossing
    ÂÂ6121472090ÂÂ78.57% 1:09:07.263 hmsÂÂWIA/BZIP2ÂÂR3OP01, Metroid: Other M
    ÂÂ2615893038ÂÂ78.76%ÂÂ 30:27.045 m:sÂÂWIA/BZIP2ÂÂRLRP4F, Tomb Raider: Anniversary
    ÂÂÂÂ80180000ÂÂ78.87%ÂÂÂÂÂÂ48.991 secÂÂWIA/BZIP2ÂÂRHAP01, Wii Play
    ÂÂ 531405683ÂÂ80.21%ÂÂÂÂ5:28.472 m:sÂÂWIA/BZIP2ÂÂRSPP01, Wii Sports
    ÂÂ5898426072ÂÂ81.50% 1:16:52.591 hmsÂÂWIA/BZIP2ÂÂRSBP01, Super Smash Bros. Brawl
    ÂÂ 115261529ÂÂ82.06%ÂÂÂÂ1:15.351 m:sÂÂWIA/BZIP2ÂÂR64P01, Wii Music
    ÂÂ1010092197ÂÂ87.75%ÂÂÂÂ9:37.114 m:sÂÂWIA/BZIP2ÂÂRZDP01, Twilight Princess
    ÂÂ2566905355ÂÂ92.58%ÂÂ 26:04.568 m:sÂÂWIA/BZIP2ÂÂRMCP01, Mario Kart Wii
    ÂÂ-------------------------------------------------------------------------------
     
  12. Wiimm
    OP

    Wiimm Developer

    Member
    2,121
    334
    Aug 11, 2009
    Gambia, The
    Germany
    Yesterday I have implemented LZMA support (was easier than thought) and over night I have run my test script. Here are the ratings of some games (my test suite):

    Warning: Spoilers inside!
    The test scripts includes decryption and comparing and all works fine. The next steps are LZMA2, fine tuning (LZMA has much compression parameters), --raw tests and code cleanup.
     
  13. Wiimm
    OP

    Wiimm Developer

    Member
    2,121
    334
    Aug 11, 2009
    Gambia, The
    Germany
    LZMA and LZMA2 (7zip main compressors) implemented!! See beta release.

    It's now your turn to test all ... [​IMG]
     
  14. moosehunter

    moosehunter GBAtemp Regular

    Member
    199
    1
    Nov 26, 2008
    United States
    Code:
    Wii Play
    ÂÂÂÂ53457608ÂÂ55.86%ÂÂÂÂÂÂWIA+LZMA2
    ÂÂÂÂ53471437ÂÂ55.88%ÂÂÂÂÂÂWIA+LZMA
    ÂÂÂÂ70627126ÂÂ73.80%ÂÂÂÂÂÂWIA/LZMA2
    ÂÂÂÂ70637763ÂÂ73.81%ÂÂÂÂÂÂWIA/LZMA
    ÂÂÂÂ95696044ÂÂ100.00%ÂÂÂÂ WIA/none
     
  15. Wiimm
    OP

    Wiimm Developer

    Member
    2,121
    334
    Aug 11, 2009
    Gambia, The
    Germany
    Quick infos:
    - The endian tests on a big endian machine are positive.
    - I have tested all 5 compression methods my test suite (10 scrubbed images) without failures.
    - I think about changing the format again and adding a field 'chunk_size' to support not only 2MiB.

    @moosehunter
    What will you say to me?
     
  16. moosehunter

    moosehunter GBAtemp Regular

    Member
    199
    1
    Nov 26, 2008
    United States
    Hmm?
     
  17. Richardarkless

    Richardarkless GBAtemp Regular

    Member
    166
    0
    Dec 11, 2009
    I find this incredibly interesting and care about saving as much space as possible but is it possible to implement this into usb loaders and if so do you know any developers that are willing to implement it into their usb loaders once of course it is final and ready for public use
     
  18. moosehunter

    moosehunter GBAtemp Regular

    Member
    199
    1
    Nov 26, 2008
    United States
    Probably not. Either it would create too much overhead to play smoothly, or there wouldn't be enough memory to decompress it.

    And I am very impressed by this. I was able to compress a raw unscrubbed MPT to 66% of it's original size.

    Would I be correct in assuming that compressing a raw game that takes up most of the disk would compress to a smaller using wia than one that only takes up only 100 mb or so? It only seems logical.
     
  19. Wiimm
    OP

    Wiimm Developer

    Member
    2,121
    334
    Aug 11, 2009
    Gambia, The
    Germany
    I have yet not done much experiments with raw copies. I never use them because I'm sure that well1 scrubbed images are the right way. So I don't see the advantage of raw dumps at this moment. The disadvantge is that non scrubbed games consumes allways much disc spase. Is there any known game that doesn't run in well1 scrubbed mode?

    1Remark: I use the word "well" before "scrubbed" because I know, that some old tools works wrong when scrubbing. 1.) There was an error in the old wiidisc lib and 2.) some tools forget to rewrite the partition table.

    --- snip ---

    Here is an other statistic:
    Code:
    Summary of RLRP4F, Tomb Raider: Anniversary:
    
    ÂÂ2131723546ÂÂ64.18%ÂÂ 18:34.390 m:sÂÂWIA/NONE + 7Z
    ÂÂ2366324457ÂÂ71.24%ÂÂ 20:21.218 m:sÂÂWIA/LZMA.9
    ÂÂ2366648574ÂÂ71.25%ÂÂ 20:20.319 m:sÂÂWIA/LZMA2.9
    ÂÂ2366869775ÂÂ71.26%ÂÂ 18:26.365 m:sÂÂWIA/LZMA.5
    ÂÂ2367264288ÂÂ71.27%ÂÂ 18:29.362 m:sÂÂWIA/LZMA2.5
    ÂÂ2464080815ÂÂ74.18%ÂÂÂÂ9:50.797 m:sÂÂWIA/LZMA.1
    ÂÂ2464467418ÂÂ74.20%ÂÂ 10:04.271 m:sÂÂWIA/LZMA2.1
    ÂÂ2610128645ÂÂ78.58%ÂÂ 12:10.654 m:sÂÂWIA/BZIP2.1
    ÂÂ2614198548ÂÂ78.70%ÂÂ 13:35.218 m:sÂÂWIA/BZIP2.5
    ÂÂ2615893189ÂÂ78.76%ÂÂ 15:05.526 m:sÂÂWIA/BZIP2.9
    ÂÂ2616222433ÂÂ78.77%ÂÂ 16:19.569 m:sÂÂWIA/NONE + BZIP2
    ÂÂ3073221032ÂÂ92.53%ÂÂÂÂ2:21.765 m:sÂÂWIA/PURGE
    ÂÂ3219977080ÂÂ96.94%ÂÂÂÂ2:19.033 m:sÂÂWIA/NONE
    ÂÂ3244354612ÂÂ97.68%ÂÂÂÂ2:12.695 m:sÂÂWDF/DECRYPT
    ÂÂ3321312056 100.00%ÂÂÂÂ1:33.369 m:sÂÂWDF
    ÂÂ3333941547 100.38%ÂÂ 19:16.613 m:sÂÂWDF + BZIP2
    ÂÂ3362583983 101.24%ÂÂ 18:59.207 m:sÂÂWDF + 7Z
    I compared compression levels 1 + 5 +9 for BZIP2 and LZMA(2). I have analyzed more games in this way and selected Anniversary as a typical example. My conclusions:[*] In general LZMA is a little bit better than LZMA2 and clear better than BZIP2. Sometimes LZMA2 or BZIP2 beat LZMA, but only minimal. And this means: LZMA is the best (and the default) compression method for WIA.
    .
    [*] Somtetimes a smaller BZIP2 level is the better one. I don't know why?
    .
    [*] With 64-bit binaries on 64-bit linux BZIP2 seems to be faster than LZMA. With 32-bit binaries LZMA beats BZIP2.
    .
    [*] The general design of WIA is to enable random access to the image to support fast reading with the WIT tools. Because of this the data is splitted in chunks of 2 MiB and each chunk is compressed independent of others. In the last release I have added the data field 'chunk_size' into the WIA format. This make it easy to experiment with other chunk sizes.
    .
    [*] If you want the best compression at all than store your file in WIA/NONE and compress it with an external packer (like 7zip). The NONE format is designed for this use and minimize the management info. If you want WIA check sums than use WIA/PURGE instead. The compressed files are minimal larger. Before reading such image must be completely uncompressed.
    .
    [*] WIA is also good for raw copies because the decryption allow much better compression.
     
  20. wishmasterf

    wishmasterf GBAtemp Regular

    Member
    167
    0
    Apr 20, 2009
    Australia
    Austria
    Is there any possibility to implement full 7zip support? If it is done by your tool it is more handy to use.