WIA = Wii ISO Archive

Discussion in 'Wii - Hacking' started by Wiimm, Aug 27, 2010.

  1. Wiimm
    OP

    Wiimm Developer

    Member
    2,117
    332
    Aug 11, 2009
    Gambia, The
    Germany
    At the moment I close the development of WIA. Perhaps I make some tests with greater chunk sizes. I think a complete compression as one chunk will never be part of WIT, because a) the file layer of my tools needs random access (e.g. for patching while copying) and b) this is a job for the packing tools. But it is no problem to write a little batch file that (de)compress an image. This is the old Unix philosophy.
     


  2. wishmasterf

    wishmasterf GBAtemp Regular

    Member
    167
    0
    Apr 20, 2009
    Australia
    Austria
    Ok. I agree with you.
     
  3. W hat

    W hat Rhythm Heaven Fan

    Member
    616
    110
    Feb 28, 2007
    United States
    For a WIA-type format on USB loaders, a less compressed medium might work. It sounds very difficult though.
     
  4. wishmasterf

    wishmasterf GBAtemp Regular

    Member
    167
    0
    Apr 20, 2009
    Australia
    Austria
    i tried:

    ./wwt EXTRACT --wia -p /dev/sdc3 SSDDRV
    ***** wwt: Wiimms WBFS Tool v1.17a.beta6 r1764 i386 - Dirk Clemens - 2010-09-18 *****
    WBFSv1 #1 opened: /dev/sdc3
    - EXTRACT SSDDRV -> WIA:Schlag den Raab [SSDDRV].wia
    ! wwt: *******************************************
    ! *** The WIA support is EXPERIMENTAL! ***
    ! *** The WIA format is in development! ***
    ! *** Don't use WIA files productive! ***
    ! *******************************************
    !! wwt: ERROR #38 [INTERNAL ERROR] in WriteWIA() @ src/lib-wia.c#1524
    !! Internal error

    What i am doing wrong?
     
  5. Wiimm
    OP

    Wiimm Developer

    Member
    2,117
    332
    Aug 11, 2009
    Gambia, The
    Germany
    wwt extract uses the old wbfslib extraction function (sequential writing). But wia needs to look inside the disc before starting writing. wwt extract does not support any patching or scrubbing. Rewriting it is already on my 2do list.

    In other words: "wwt extarct --wia" does not work. (I have to deny this option).
     
  6. wishmasterf

    wishmasterf GBAtemp Regular

    Member
    167
    0
    Apr 20, 2009
    Australia
    Austria
    Whats the best way to extract a game from a wbfs-partition into a wia-file?
     
  7. Wiimm
    OP

    Wiimm Developer

    Member
    2,117
    332
    Aug 11, 2009
    Gambia, The
    Germany
    "wit copy /dev/sdx9/ID6xxx %Y.wia" should work. (never tested)
     
  8. wishmasterf

    wishmasterf GBAtemp Regular

    Member
    167
    0
    Apr 20, 2009
    Australia
    Austria
    The last Version seems to be great! I will do my tests and will report my results. I will see if i have enough time, but maybe i will post my Testresults next weekend.
     
  9. Wiimm
    OP

    Wiimm Developer

    Member
    2,117
    332
    Aug 11, 2009
    Gambia, The
    Germany
    WIA supports now variable chunk sizes, always a multiple of 2 MiB. The 2 MiB base value is set, because is is the size of a Wii sector group. The 64 sectors in a Wii sector group share H2 hash values. Loading of all 64 sectors is necessary to calculate all hash values.

    Since tomorrow the test script ./scrips/test-image-size.sh is running. Here is a sample result of Animal Crossing:

    Code:
    Summary of RUUP01, Animal Crossing: Let's Go to the City:
    
    ÂÂÂÂ60891211ÂÂ17.26%ÂÂÂÂ3:23.487 m:sÂÂWIA/LZMA@50
    ÂÂÂÂ60892381ÂÂ17.26%ÂÂÂÂ3:24.152 m:sÂÂWIA/LZMA2@50
    ÂÂÂÂ62191768ÂÂ17.63%ÂÂÂÂ1:44.277 m:sÂÂWIA/NONE@5 + 7Z
    ÂÂÂÂ62198411ÂÂ17.63%ÂÂÂÂ1:42.936 m:sÂÂWIA/NONE@2 + 7Z
    ÂÂÂÂ62198529ÂÂ17.63%ÂÂÂÂ1:42.429 m:sÂÂWIA/NONE@1 + 7Z
    ÂÂÂÂ62204066ÂÂ17.63%ÂÂÂÂ1:46.096 m:sÂÂWIA/NONE@10 + 7Z
    ÂÂÂÂ62209772ÂÂ17.63%ÂÂÂÂ1:51.934 m:sÂÂWIA/NONE@20 + 7Z
    ÂÂÂÂ62233563ÂÂ17.64%ÂÂÂÂ2:10.734 m:sÂÂWIA/NONE@50 + 7Z
    ÂÂÂÂ64299774ÂÂ18.23%ÂÂÂÂ2:49.430 m:sÂÂWIA/LZMA@20
    ÂÂÂÂ64304336ÂÂ18.23%ÂÂÂÂ2:48.873 m:sÂÂWIA/LZMA2@20
    ÂÂÂÂ67348400ÂÂ19.09%ÂÂÂÂ2:39.179 m:sÂÂWIA/LZMA2@10
    ÂÂÂÂ67354140ÂÂ19.09%ÂÂÂÂ2:38.835 m:sÂÂWIA/LZMA@10
    ÂÂÂÂ73261909ÂÂ20.77%ÂÂÂÂ2:34.270 m:sÂÂWIA/LZMA@5
    ÂÂÂÂ73264290ÂÂ20.77%ÂÂÂÂ2:33.395 m:sÂÂWIA/LZMA2@5
    ÂÂÂÂ86310086ÂÂ24.47%ÂÂÂÂ2:32.665 m:sÂÂWIA/LZMA@2
    ÂÂÂÂ86315021ÂÂ24.47%ÂÂÂÂ2:31.092 m:sÂÂWIA/LZMA2@2
    ÂÂÂÂ95699469ÂÂ27.13%ÂÂÂÂ2:32.221 m:sÂÂWIA/LZMA2@1
    ÂÂÂÂ95705674ÂÂ27.13%ÂÂÂÂ2:33.008 m:sÂÂWIA/LZMA@1
    ÂÂ 347250800ÂÂ98.45%ÂÂÂÂÂÂ10.787 secÂÂWIA/NONE@1
    ÂÂ 349338652ÂÂ99.04%ÂÂÂÂÂÂ10.699 secÂÂWIA/NONE@2
    ÂÂ 352695380 100.00%ÂÂÂÂÂÂ 3.870 secÂÂWDF
    ÂÂ 357217459 101.28%ÂÂÂÂ1:52.452 m:sÂÂWDF + 7Z
    ÂÂ 365782884 103.71%ÂÂÂÂÂÂ11.030 secÂÂWIA/NONE@5
    ÂÂ 386424880 109.56%ÂÂÂÂÂÂ11.253 secÂÂWIA/NONE@10
    ÂÂ 427711712 121.26%ÂÂÂÂÂÂ12.316 secÂÂWIA/NONE@20
    ÂÂ 632839056 179.42%ÂÂÂÂÂÂ17.391 secÂÂWIA/NONE@50
    '@N' is the chunk-size-factor. '@50' means: 50 * base_chunk_size = 100 MiB. For this @50 calculation wit needs about 300 MiB of RAM. All compressions done with level 9. You see that the internal WIA/LZMA@50 beats the best external 7zip compression.

    Some more results:
    Warning: Spoilers inside!

    Remark I:
    The test is done with the test script ./scripts/test-image-size.sh, it's part of the release. The command I used is:
    Code:
    ./scripts/test-image-size.sh --fast --level @1,@2,@5,@10,@20,@50 \
    ÂÂÂÂ--data --diff --dump --7zipÂÂ--compr non,lz,lz2 ./test-suite/*.wdf
    Just call ./scripts/test-image-size.sh to print built in help.

    Remark II:
    The next beta release is coming today or tomorrow. The WIA format has not been changed since the previous beta, so that already generated WIAs can be used furthermore.


    Remark III:
    The new (and beta) tool 'wdf' can show some parameters of a wia. Try 'wdf +dump image'. If you forget the '+dump' is will pack the image to a WDF. You can also rename it to 'wdf-dump' to set dumping as default command. 'wdf --help' prints a help message.
     
  10. vexing

    vexing Advanced Member

    Newcomer
    65
    0
    Nov 16, 2009
    United States
    do you know why the internal compression is taking so much longer than external?
    based on those 4 discs it looks like none@2 or none@5 plus external 7zip is a really good mix for speed and high compression...
    have you tested conversion back to a scrubbed playable format? the speed benefit of external might be lost with the extra step
     
  11. Wiimm
    OP

    Wiimm Developer

    Member
    2,117
    332
    Aug 11, 2009
    Gambia, The
    Germany
    My test script make a diff for each created wia file.
     
  12. wishmasterf

    wishmasterf GBAtemp Regular

    Member
    167
    0
    Apr 20, 2009
    Australia
    Austria
    After your test Results i think LZMA with chunck - size 50 MiB should be the default setting. It seems it is most cases one of the best settings. What do you think wimm?
     
  13. vexing

    vexing Advanced Member

    Newcomer
    65
    0
    Nov 16, 2009
    United States
    @50 was most compressed generally but the speed differences were significant at smaller chunk sizes... comparing @50 and @20 in that test set, at worst @20 was 6% larger final size and 17% faster, at best it was 2x as fast with a very slightly smaller final size.
     
  14. wishmasterf

    wishmasterf GBAtemp Regular

    Member
    167
    0
    Apr 20, 2009
    Australia
    Austria
    Oh i see. So maybe the best setting could be @20. So we have to do more tests to find the best default setting because current setting @5 seems not the best way.

    @wimm what do you think about changing the default chuck - size?
     
  15. Wiimm
    OP

    Wiimm Developer

    Member
    2,117
    332
    Aug 11, 2009
    Gambia, The
    Germany
    In the latest beta @20 is the default. At this moment a script runs with @10,20,30,40,50 for about 10 different games. It will run until tomorrow. After that we can continue the discussion.

    EDIT:
    The first 3 results:

    Code:
    Summary of RUUP01, Animal Crossing: Let's Go to the City:
    
    ÂÂÂÂ59919934ÂÂ16.98%ÂÂÂÂ3:04.094 m:sÂÂWIA/LZMA@40
    ÂÂÂÂ60852341ÂÂ17.25%ÂÂÂÂ2:59.068 m:sÂÂWIA/LZMA@30
    ÂÂÂÂ61577490ÂÂ17.45%ÂÂÂÂ3:13.285 m:sÂÂWIA/LZMA@50
    ÂÂÂÂ64762661ÂÂ18.36%ÂÂÂÂ2:49.431 m:sÂÂWIA/LZMA@20
    ÂÂÂÂ67473743ÂÂ19.13%ÂÂÂÂ2:42.040 m:sÂÂWIA/LZMA@10
    ÂÂ 352695380 100.00%ÂÂÂÂÂÂ 3.746 secÂÂWDF
    
    
    Summary of RMCP01, Mario Kart Wii:
    
    ÂÂ2258986334ÂÂ81.47%ÂÂ 28:37.735 m:sÂÂWIA/LZMA@50
    ÂÂ2269315147ÂÂ81.84%ÂÂ 27:01.235 m:sÂÂWIA/LZMA@40
    ÂÂ2275611253ÂÂ82.07%ÂÂ 21:50.050 m:sÂÂWIA/LZMA@20
    ÂÂ2278355000ÂÂ82.17%ÂÂ 24:42.624 m:sÂÂWIA/LZMA@30
    ÂÂ2311609224ÂÂ83.37%ÂÂ 18:21.345 m:sÂÂWIA/LZMA@10
    ÂÂ2772579124 100.00%ÂÂÂÂ1:19.534 m:sÂÂWDF
    
    
    Summary of R3OP01, Metroid: Other M:
    
    ÂÂ5655712840ÂÂ72.59% 1:32:47.480 hmsÂÂWIA/LZMA@50
    ÂÂ5663710912ÂÂ72.70% 1:28:19.128 hmsÂÂWIA/LZMA@40
    ÂÂ5675384932ÂÂ72.85% 1:22:05.267 hmsÂÂWIA/LZMA@30
    ÂÂ5694298457ÂÂ73.09% 1:13:04.454 hmsÂÂWIA/LZMA@20
    ÂÂ5730003666ÂÂ73.55% 1:02:51.743 hmsÂÂWIA/LZMA@10
    ÂÂ7790477636 100.00%ÂÂÂÂ4:10.745 m:sÂÂWDF
    EDIT II:
    Command is: ./test-image-size.sh --compr lz --level @10,@20,@30,@40,@50 --fast --data *.wdf
    Do it with your games [​IMG]
     
  16. wishmasterf

    wishmasterf GBAtemp Regular

    Member
    167
    0
    Apr 20, 2009
    Australia
    Austria
    ok on the last 3 games we see that on average maybe also @10 could be the best setting. maybe the best choice is between @10 and @20, maybe @16, @14 or anything else.
     
  17. Wiimm
    OP

    Wiimm Developer

    Member
    2,117
    332
    Aug 11, 2009
    Gambia, The
    Germany
    First I complete the previous statistic, games sorted by size:
    Warning: Spoilers inside!



    And now I have bad news: The LZMA functions use a user specified allocation function. So it was easy to count the dynamic memory usage of LZMA/LZMA2. Here are the (bad) results:
    <!--c1--><div class='codetop'>CODE</div><div class='codemain'><!--ec1-->ÂÂmethod levelÂÂÂÂÂÂÂÂ memory usage
    -----------------------------------------
    ÂÂ LZMAÂÂ.1ÂÂÂÂÂÂ 1831198 =ÂÂ 1.75 MiB
    ÂÂ LZMAÂÂ.2ÂÂÂÂÂÂ 3174686 =ÂÂ 3.03 MiB
    ÂÂ LZMAÂÂ.3ÂÂÂÂÂÂ 9072926 =ÂÂ 8.65 MiB
    ÂÂ LZMAÂÂ.4ÂÂÂÂÂÂ32665886 =ÂÂ31.15 MiB
    ÂÂ LZMAÂÂ.5ÂÂÂÂ 194146594 = 185.15 MiBÂÂdefault for 7zip
    ÂÂ LZMAÂÂ.6ÂÂÂÂ 387084578 = 369.15 MiB
    ÂÂ LZMAÂÂ.7-9:ÂÂ705851730 = 673.15 MiB

    ÂÂ LZMA2 .1ÂÂÂÂÂÂ 4938158 =ÂÂ 4.71 MiB
    ÂÂ LZMA2 .2ÂÂÂÂÂÂ 5986734 =ÂÂ 5.71 MiB
    ÂÂ LZMA2 .3ÂÂÂÂÂÂ10705326 =ÂÂ10.21 MiB
    ÂÂ LZMA2 .4ÂÂÂÂÂÂ32731566 =ÂÂ31.22 MiB
    ÂÂ LZMA2 .5ÂÂÂÂ 194212274 = 185.22 MiBÂÂdefault for 7zip
    ÂÂ LZMA2 .6ÂÂÂÂ 387150258 = 369.22 MiB
    ÂÂ LZMA2 .7-9:ÂÂ705917410 = 673.22 MiB<!--c2--></div><!--ec2-->
    Level 1-4 uses an other algorithm than 5-7. Level 7-9 are identically. Additionally WIA uses 2*chunk_size == 4MiB*factor of dynamic memory. BZIP2 level 9 should take 47 MiB.


    Here is a general info about the LZMA encoder:
    Warning: Spoilers inside!


    Here are some statistics for different LZMA compression levels:
    Warning: Spoilers inside!


    And this is the dynamic memory usage of the above methods sorted by usage:
    Warning: Spoilers inside!

    <!--coloro:#800000--><span style="color:#800000"><!--/coloro--><b>Discussion: What is the best LZMA default?</b><!--colorc--></span><!--/colorc-->
    I already offer some keywords like BEST, FAST, DEFAULT. Perhaps I need more like LOWMEM, HIMEM, ....
     
  18. Pippin666

    Pippin666 SSF43DE Master

    Member
    1,778
    227
    Mar 30, 2009
    Canada
    Montreal, Qc
    Is there any loader that supports WIA on fat32 ??

    Pip'
     
  19. Bauldrick

    Bauldrick GBAtemp Regular

    Member
    103
    1
    Jan 22, 2009
    Maybe MEM64,MEM128,MEM256 etc.... (I would use MEM64 for example) - Could wit not get that info (amount of RAM) from the box and use it as default?
     
  20. Blue-K

    Blue-K No right of appeal.

    Member
    2,572
    1
    Jun 21, 2008
    Swaziland
    Helvetica
    Not possible.