WIA = Wii ISO Archive

Discussion in 'Wii - Hacking' started by Wiimm, Aug 27, 2010.

  1. Wiimm
    OP

    Wiimm Developer

    Member
    2,159
    368
    Aug 11, 2009
    Gambia, The
    Germany
    If I remember right some old test of me, the decrypted garbage of some blocks is a zero filled data block. And this is the reason that WDF/DECRYPTED is smaller than standard WDF (see statistics in the first post).
     


  2. Wiimm
    OP

    Wiimm Developer

    Member
    2,159
    368
    Aug 11, 2009
    Gambia, The
    Germany
     
  3. wishmasterf

    wishmasterf GBAtemp Regular

    Member
    167
    0
    Apr 20, 2009
    Australia
    Austria
    First of all many thanks for sharing your great tool.

    I think the goals of the new format should besmallest size possible
    easy to use
    fast as possible
    loadable in USB-LoadersThe first 2 things seems to be done

    @giantpune and wiipower:

    it is great to see that you support wimm! im not a developer
    and do not know all about the disc-formats so i only can help
    with testing. It would be great if you 3 guys could create that
    ultimate format ;-)
     
  4. Wiimm
    OP

    Wiimm Developer

    Member
    2,159
    368
    Aug 11, 2009
    Gambia, The
    Germany
    "smallest size possible" and "fast as possible" together is not possible.
     
  5. Remfin

    Remfin Advanced Member

    Newcomer
    65
    0
    Mar 5, 2010
    United States
    Yet Another Gamecube Documentation. I implemented the "what's actually in the encrypted data" stuff for my Wii program working off only that since Wiibrew didn't talk about what the layout of the encrypted data actually was. It's been a long time but I think the only things that I had to change was the bit-shiftings giantpune mentioned (header's offsets and FST's file offsets), so you probably already have all the code written (except for a few ifs).
     
  6. vexing

    vexing Advanced Member

    Newcomer
    65
    0
    Nov 16, 2009
    United States
     
  7. WiiPower

    WiiPower GBAtemp Guru

    Member
    8,165
    72
    Oct 17, 2008
    Gambia, The
    Loading compressed files with a cIOS isn't easy. The PC .iso tools don't have 2 restrictions the cIOS has:
    - The cIOS has very limited memory, you may be able to use a few KB(
     
  8. Wiimm
    OP

    Wiimm Developer

    Member
    2,159
    368
    Aug 11, 2009
    Gambia, The
    Germany
    Before starting to implement I have made a test:
    I have taken a decrypted image (animal crossing):[*]Test #1: packed with bzip2[*]Test #2: split it into peaces of 2 MB and pack each peace. Contact all together and compare the size with test #1Because bzip2 works in chunks (100KB-900kb, user defined) the splitted files are only 1% larger than the unspllttet one. It seems that bzip2 is good for small chunks, but it needs about 500k memory.

    I have not tested other compressions but perhaps any other (vexig for 7z?) can do this.

    But decryption makes it possible to detect repeated values. Perhaps it is the best to use an old and easy compression method for usb loading.
     
  9. Wiimm
    OP

    Wiimm Developer

    Member
    2,159
    368
    Aug 11, 2009
    Gambia, The
    Germany
    A little uncompressing test.

    Copy and convert a "Wii Sports" WDF, WIA and WIA/UNCOMPRESSED to a new WDF:
    Code:
    ÂÂ 25 secÂÂWDF -> WDF
    ÂÂ 63 secÂÂWIA/UNCOMPRESSED -> WDF
    ÂÂ152 secÂÂWIA/COMPRESSED -> WDF
     
  10. vexing

    vexing Advanced Member

    Newcomer
    65
    0
    Nov 16, 2009
    United States
    that's an interesting point. i was thinking up a level, but looking at waninkoko's fat module for example it just handles direct access to offsets.

    new super mario brothers wii, decrypted.wdf is 347,615 KiB.
    using 7z, lzma compression, on ultra settings reduces it to 214,570 KiB
    using 7z, lzma compression, 64 KiB dictionary reduces it to 236,754 KiB

    now, taking the file and splitting it into chunks and compressing with lzma:
    64 KiB chunks: 237,314 KiB
    8 KiB chunks: 252,601 KiB

    i also took a fairly representative of compression 1mb part from the iso, chunked it up, compressed

    chunk size : combined size, both in KiB
    1: 922
    2: 841
    4: 788
    8: 753
    16: 727
    32: 709
    64: 694

    i'm starting to think this could work.
     
  11. W hat

    W hat Rhythm Heaven Fan

    Member
    617
    112
    Feb 28, 2007
    United States
    Some primitive compression might work for USB Loading, but it's almost completely pointless when you can have far better compression on the PC-side.

    For the "fast as possible" people, Bzip2 is a great choice. In the long term, I don't care about spending 1/2 hour or even 1 full hour compressing a game with 7zip if it's a game I just want to back up to my computer.
     
  12. vexing

    vexing Advanced Member

    Newcomer
    65
    0
    Nov 16, 2009
    United States
    i think the opposite... imagine a scrubbed iso going from 4gb to 2.8gb and still directly playable via usb loading, would you really bother with decompression/compression to archive it on your pc at 2.4gb? i know i wouldn't.
     
  13. W hat

    W hat Rhythm Heaven Fan

    Member
    617
    112
    Feb 28, 2007
    United States
    We all have different needs, and I wouldn't trust my WBFS drive with any data. I hate that format, and I'm just waiting for the day when either
    1. The filesystem dies and I have to re-install my games in fat32 / ntfs
    2. Wiimm, master of Wii, makes a total drive converter (the process would take a lot of HDD space while it was working, on at least two drives. )
     
  14. vexing

    vexing Advanced Member

    Newcomer
    65
    0
    Nov 16, 2009
    United States
    well, a wii compressed file system would probably make this easier to implement, but it's not required. these could possibly be a container like .wbfs files that would be readable on a fat32 drive.

    regarding a drive converter, it seems like it'd be possible to directly convert a wbfs into fat32. that is don't modify any of the iso files, just create the file system that contains the pointers to all the files. i believe the data can be kept 100% the same.
    the only issues i can think of are the wbfs probably doesn't reserve enough space at the beginning of the drive for the fat32 file system, so it might need to move the first iso, and it'd have to handle splitting files > 4gb, which might not be hard.
     
  15. Wiimm
    OP

    Wiimm Developer

    Member
    2,159
    368
    Aug 11, 2009
    Gambia, The
    Germany
    How do you find the "fairly representative of compression 1mb"?
    I have splitted a whole image into peaces and have seen, that the compression ratio is very different for each peace.
     
  16. vexing

    vexing Advanced Member

    Newcomer
    65
    0
    Nov 16, 2009
    United States
    i split the whole image into the 64k chunks, then looked for a set of 16 of those that was around 700k compressed, which nearly matches the ratio of the whole image.

    the whole image in 64k chunks was 68.3%
    in 8k chunks was 72.7%
    the 1mb in 64k chunks was 67.8%
    in 8k chunks it was 73.5%

    the 1mb seems a reasonable enough sample to give approximations of how the chunk size affects compression. anyway, the point was even with pretty small chunks like 16k you can get near 70% of original size, which means this is feasible for maintaining random access.
     
  17. Remfin

    Remfin Advanced Member

    Newcomer
    65
    0
    Mar 5, 2010
    United States
    Until you realize the lookup table for a disc stored in 8k chunks is 4 megs (4 bytes per entry, which may not be feasible).

    Warning: Spoilers inside!
     
  18. WiiPower

    WiiPower GBAtemp Guru

    Member
    8,165
    72
    Oct 17, 2008
    Gambia, The
    Wiimm, how much work would it be for you to write a "Wii Release Checker"?

    Did you see thoese threads about Metroid Other M freezing? It looks like it's because of a broken release. Would you be interested in writing an app that tells:
    - if a .iso is scrubbed or not
    - checks the hashes of the .iso? (i guess the hashes for the scrubbed area don't match, but the rest just HAS to)
    - tells if the update partition is missing

    If you do that, it might be the 1st thing people do on new releases to immediately to post: "The release is incomplete, it's scrubbed", "The release is incomplete, there's no update partition" or "The release is broken, some parts(in the not scrubbed area) do not match the hashes, this WILL result in some kind of error in the game"

    I'm too tired to do the math right now, but i fear that at least the direction is correct and the size for the lookup table is one problem more for compressed .iso loading.
     
  19. Wiimm
    OP

    Wiimm Developer

    Member
    2,159
    368
    Aug 11, 2009
    Gambia, The
    Germany
    Is it?
    The lookup table can be loaded partially. And the controller can always hold for example 4 peaces. I think reloading takes less time as loading and decompressing the real data.
     
  20. vexing

    vexing Advanced Member

    Newcomer
    65
    0
    Nov 16, 2009
    United States
    you could have a two layer table, maybe 4700 entries stored in memory, each entry would cover a 1mb area, then the second lookup table could be stored in the 1mb area because the hard disk is going to be reading that area and caching it anyway.