WIA = Wii ISO Archive

Discussion in 'Wii - Hacking' started by Wiimm, Aug 27, 2010.

Aug 27, 2010

WIA = Wii ISO Archive by Wiimm at 8:03 PM (122,914 Views / 1 Likes) 172 replies

  1. wishmasterf

    Member wishmasterf GBAtemp Regular

    Joined:
    Apr 20, 2009
    Messages:
    167
    Location:
    Austria
    Country:
    Austria
    OK. I understand. I will use it and we will see ;-)
     


  2. ssjkakaroto

    Newcomer ssjkakaroto Advanced Member

    Joined:
    Dec 14, 2005
    Messages:
    86
    Country:
    @Wiimm: I tried using wit (r1645) to convert Metroid Other M (dual-layer) ISO to WIA and back to ISO but the MD5 of the resulting ISO was different from the original. Is this expected?
    The commands I used were:
    Code:
    wit COPY --raw --progress --wia --no-compress file.iso file.wia
    wit COPY --raw --progress --iso --no-compress file.wia file2.iso
     
  3. Wiimm
    OP

    Member Wiimm Developer

    Joined:
    Aug 11, 2009
    Messages:
    2,079
    Location:
    Germany
    Country:
    Germany
    First: You can shorten the commands:
    Code:
    wit COPY --raw --progress --no-compress file.iso file.wia
    wit COPY --raw --progress file.wia file2.iso
    a) If the file extension is well known the output format is clear. The option like --wia overrides the output format.
    b) --no-compress is only relevant for converting *to wia*.

    Your question:
    Have both ISO the same length?
    Try "wit diff --raw file.iso file2.iso --long --long" or "wit diff --raw file.iso file2.iso --long --long". It compares the content and prints the block relevant blocks.

    I will do the same test.
     
  4. ssjkakaroto

    Newcomer ssjkakaroto Advanced Member

    Joined:
    Dec 14, 2005
    Messages:
    86
    Country:
  5. Wiimm
    OP

    Member Wiimm Developer

    Joined:
    Aug 11, 2009
    Messages:
    2,079
    Location:
    Germany
    Country:
    Germany
    I have made the same test with similar result. The ISOs differ in the space behind the last partition (=waste). Here is a disc memory layout of "Other M":

    Warning: Spoilers inside!
    You see: The useful data ends at offset 0x1fa550000 and your diffs (and my too) are behind that point. If you use "wit diff" without --raw only needed/used sectors are compared.

    That's bad for raw mode and I have improve this. It's possible that this is only a problem for double layer.


    EDIT:
    I store all data from 0x0..0x4ffff and all partition data (in scrubbed or raw mode). Data between partitions and behind the last partition is not copied. This can be changed without changing the WIA format, because I have already spend a raw data area in the data structure.

    Another file format independent question is: Is it really needed to store such data?
     
  6. Remfin

    Newcomer Remfin Advanced Member

    Joined:
    Mar 5, 2010
    Messages:
    65
    Country:
    United States
    The archivers/collectors are going to want ALL the data, even the garbage--they want stuff bit-perfect no matter what. So a proper archive format should probably keep it.

    If it helps, it's suspected it's simply an artifact of the encryption process...try running a decryption with the various title keys on the disc (and maybe with different alignments?) and see if one of them gets you zeroes.
     
  7. wishmasterf

    Member wishmasterf GBAtemp Regular

    Joined:
    Apr 20, 2009
    Messages:
    167
    Location:
    Austria
    Country:
    Austria

    Is the ISO still useable? I mean could you play the game after converting to WIA and back?
     
  8. Wiimm
    OP

    Member Wiimm Developer

    Joined:
    Aug 11, 2009
    Messages:
    2,079
    Location:
    Germany
    Country:
    Germany
    Above I told:
    a) Only filling data behind the last partition is missed. The game is still playable and scrubbing will ignore this data area and more.
    b) I'm working to improve the raw mode to store *all* data. But this takes time because I'm very busy in this week.


    @all:
    Don't use WIA permanent because I'm on the way to change the format. After the raw data discussion I see a way to simplify the WIA format. This makes it easier to implement it into other tools. And this test phase is a good time for changing.
     
  9. W hat

    Member W hat Rhythm Heaven Fan

    Joined:
    Feb 28, 2007
    Messages:
    612
    Country:
    United States
    I hope an option will exist for heavy compression - FreeARC or 7zip (LZMA). I know you've already thought about this, and that all WIA development is just for fun for you...

    B - Does this mean you will try to go from Superdump->ISO->WIA->ISO without losing any garbage data?
     
  10. Wiimm
    OP

    Member Wiimm Developer

    Joined:
    Aug 11, 2009
    Messages:
    2,079
    Location:
    Germany
    Country:
    Germany
    The WIA structure supports 4294967296 possible compression methods. At the moment 0=uncompressed and 1=bzip2 are implemented. My road map:
    1.) finish WIA implementation with 2 methods and a better raw support (good for gamecube discs too)
    2.) finish gamecube support (incl. mix command and wia support)
    3.) implement trimming in 3 levels (disc, partition, file system)
    ...
    n) view the 7z lib source and ....


    Yes if using --raw (because scrubbing is the default)
    This is already true for all other conversions with "wit copy --raw ..."
     
  11. W hat

    Member W hat Rhythm Heaven Fan

    Joined:
    Feb 28, 2007
    Messages:
    612
    Country:
    United States
    That's a bold move; I'd expect nothing less from Wiimm.

    I can't wait to see all of it, even if what I'm looking forward to is shoved aside right now for GameCube support and trimming. Trimming means using 00 to scrub, not FF, so you'll get an ISO that is 2.35 GiB in modern filesystems, but has a real size of 4.3GiB?
     
  12. ssjkakaroto

    Newcomer ssjkakaroto Advanced Member

    Joined:
    Dec 14, 2005
    Messages:
    86
    Country:
    I don't know. I don't use USB loaders, so I didn't try burning the disc just to test.


    Thanks for looking into it Wiimm
     
  13. W hat

    Member W hat Rhythm Heaven Fan

    Joined:
    Feb 28, 2007
    Messages:
    612
    Country:
    United States
    1. Garbage data is not needed ever - it may as well be storing Wikipedia archives, or other more useful data. Different people have different needs, of course, but I think the garbage data may be specifically designed to be incompressible.

    2. My hopes for Wii storage formats are as follows:

    a. Store patches for games efficiently (For example, Disney Princess and the Frog regular edition -> deluxe edition, Wii Sports v1.00 -> v1.01) and automate this patching process
    b. Store Wii games that I never play as efficiently as I can, with a maximum compression time of two hours and maximum decompression time of one hour.
    c. Superdumps of the same disc from multiple sources should still be identical after being archived, provided "nodates" and other modes are used to ensure uniform compression.
     
  14. wishmasterf

    Member wishmasterf GBAtemp Regular

    Joined:
    Apr 20, 2009
    Messages:
    167
    Location:
    Austria
    Country:
    Austria

    Do we get patches for games or only if i download the whole ISO on warez sites?
     
  15. W hat

    Member W hat Rhythm Heaven Fan

    Joined:
    Feb 28, 2007
    Messages:
    612
    Country:
    United States
    I'm not going to lie - I have pirated games before. That does not mean I'm filthy scum - I've bought Super Mario Galaxy 2, New Super Mario Bros. Wii, Endless Ocean, Endless Ocean 2, and many more. (I've even bought games I didn't like because I don't believe in the "pirate before you buy" policy.)

    Did I buy "The Princess and the Frog"? No. Did I play it? No. Did I use it as an example, and would I be interested to see a patch from the normal to deluxe edition? From a theoretical standpoint, yes. This is not because I wish to steal the deluxe features of the Deluxe edition. Same thing with Wii Sports 1.00 to 1.01. I'm sure if Nintendo had the means to distribute patches, a Wii Sports update would be free. Even so, I have not downloaded Wii Sports v1.01, merely used it as an example.

    Wiimm seems to be very honest - I don't believe he ever downloads games, even if downloading a JPN or NTSC-U game would help his testing.
     
  16. Wiimm
    OP

    Member Wiimm Developer

    Joined:
    Aug 11, 2009
    Messages:
    2,079
    Location:
    Germany
    Country:
    Germany
    For my tools scrubbing means to leave out unused areas while writing. This is similar like writing zeros, but for plain iso file the OS creates sparse files for the unused areas. WBFS and CISO files are also candidates for sparse files because of the large block size (>=2MB). WDF and WIA manage this holes. If reading the holes seems to be zeroed.

    With trimming I mean moving data. For details read this and this. Wii Scrubber moves also data.
     
  17. wishmasterf

    Member wishmasterf GBAtemp Regular

    Joined:
    Apr 20, 2009
    Messages:
    167
    Location:
    Austria
    Country:
    Austria
    @W hat

    I started a List of "new edition games". Maybe you know more of this games and add them.

    new edition games
     
  18. wishmasterf

    Member wishmasterf GBAtemp Regular

    Joined:
    Apr 20, 2009
    Messages:
    167
    Location:
    Austria
    Country:
    Austria

    When will the next release available?
     
  19. Wiimm
    OP

    Member Wiimm Developer

    Joined:
    Aug 11, 2009
    Messages:
    2,079
    Location:
    Germany
    Country:
    Germany
    I will start the re-implementation this week end.
     
  20. Wiimm
    OP

    Member Wiimm Developer

    Joined:
    Aug 11, 2009
    Messages:
    2,079
    Location:
    Germany
    Country:
    Germany
    I have re-implemented WIA. It support now three compression methods:
    • --compression NONE
      Data of used (=non scrubbed sectors) are stored 1:1 with 0x8000 bytes (32KiB) or 0x7c00 bytes for decrypted data partitions (hash removed). This format is very good for an external packer. No hash values for the chunks available.
      .
    • --compression PURGE
      Data is purged like WDF: Areas with zeros are detected and skipped. The image is smaller than 'NONE' but contains more management info. There is a SHA1 hash for each data chunk so that data errors can be noticed.
      .
    • --compression BZIP2
      Each chunk is compressed with bzip2 level 9. bzip2 has its own verification code.
      .
    Here are some first compression ratings. As expected WIA/NONE+7Z is the smallest. I'm wondering that WIA/BZIP2 is smaller than WIA/NONE+BZIP2

    Code:
    Summary of RZDP01, The Legend of Zelda: Twilight Princess:
    
    ÂÂ 808195658ÂÂ70.21%ÂÂ 10:34.731 m:sÂÂWIA/NONE + 7Z
    ÂÂ 808527639ÂÂ70.24%ÂÂ 11:03.371 m:sÂÂWIA/PURGE + 7Z
    ÂÂ 834192659ÂÂ72.46%ÂÂ 10:35.820 m:sÂÂ WDF/DECRYPT + 7Z
    ÂÂ1010092171ÂÂ87.75%ÂÂÂÂ9:43.185 m:sÂÂWIA/BZIP2
    ÂÂ1010175516ÂÂ87.75%ÂÂÂÂ9:44.945 m:sÂÂWIA/NONE + BZIP2
    ÂÂ1010696787ÂÂ87.80%ÂÂÂÂ9:43.515 m:sÂÂWIA/PURGE + BZIP2
    ÂÂ1014462607ÂÂ88.13%ÂÂ 18:46.240 m:sÂÂWIA/BZIP2 + BZIP2
    ÂÂ1022830826ÂÂ88.85%ÂÂ 17:08.357 m:sÂÂWIA/BZIP2 + 7Z
    ÂÂ1038300726ÂÂ90.20%ÂÂÂÂ9:43.655 m:sÂÂ WDF/DECRYPT + BZIP2
    ÂÂ1089171376ÂÂ94.62%ÂÂÂÂ1:40.799 m:sÂÂWIA/PURGE
    ÂÂ1119341208ÂÂ97.24%ÂÂÂÂ1:38.125 m:sÂÂWIA/NONE
    ÂÂ1124536808ÂÂ97.69%ÂÂÂÂ1:27.135 m:sÂÂWDF/DECRYPT
    ÂÂ1151087416 100.00%ÂÂÂÂÂÂ25.264 secÂÂ WDF
    ÂÂ1156120637 100.43%ÂÂ 10:46.613 m:sÂÂ WDF + BZIP2
    ÂÂ1166488810 101.33%ÂÂÂÂ8:49.884 m:sÂÂ WDF + 7Z
    At the moment I'm working at the reading part.
     

Share This Page