Hacking WIA = Wii ISO Archive

[Wiimm]

Does this mean that you are able to convert unscrubbed Wii ISOs (aka untouched) into WIA format and back again? This has always been a problem in the Wii scene (that all releases are 4.7GB because of the encrypted filler data).
Yes, WIA is only another container like WBFS, CISO or WDF. My tools support scrubbed (default) and non scrubbed copying. If you convert any ISO to WIA and back the result is the same as the original ISO. And this includes also hash errors.

QUOTE(Fat D @ Aug 29 2010, 04:15 PM) That would require the garbage data to make any sense when decrypted - but it does not, that is why it is called garbage. ...

If I remember right some old test of me, the decrypted garbage of some blocks is a zero filled data block. And this is the reason that WDF/DECRYPTED is smaller than standard WDF (see statistics in the first post).

 

[Wiimm]

@ wimm
the structure for GC games is very similar to a wii partition. you use the same method to find the fst.bin, apploader, dol etc. the main difference in GC games is that there is only 1 partition, there is no ticket/tmd/cert, there is no data encryption, and offsets are exactly as you read them from the disc ( no need to do "

 

[wishmasterf]

First of all many thanks for sharing your great tool.

I think the goals of the new format should besmallest size possible
easy to use
fast as possible
loadable in USB-LoadersThe first 2 things seems to be done

@giantpune and wiipower:

it is great to see that you support wimm! im not a developer
and do not know all about the disc-formats so i only can help
with testing. It would be great if you 3 guys could create that
ultimate format ;-)

 

[Wiimm]

"smallest size possible" and "fast as possible" together is not possible.

 

[Remfin]

Thanx giant. I think I can borrow a disc from a friend for analysis. Do you (or any other) know a good reference about GC disc structure (before I talk to google and browse all bad sites)?

Yet Another Gamecube Documentation. I implemented the "what's actually in the encrypted data" stuff for my Wii program working off only that since Wiibrew didn't talk about what the layout of the encrypted data actually was. It's been a long time but I think the only things that I had to change was the bit-shiftings giantpune mentioned (header's offsets and FST's file offsets), so you probably already have all the code written (except for a few ifs).

 

[vexing]

Summary of RHAP01, Wii Play:
52934196  52.07%	  56.550 sec  WIA/NOCOMPRESS + 7Z
...
80354061  79.04%	  51.282 sec  WIA/NOCOMPRESS + BZIP2

10% slower, and 2/3rds the size.
so, any plans to add 7z's lzma compression? i haven't looked at your code, or their sdk, but i can't imagine it'd be too difficult


QUOTE(wishmasterf @ Aug 29 2010, 10:17 AM) loadable in USB-Loaders[/quote]

as wiipower mentioned a cios would have to support decompression... and even if that's done the decompression has to perform fast enough so games are still playable.
it seems like each file would have to be encrypted individually

 

[WiiPower]

[...]

loadable in USB-Loaders

as wiipower mentioned a cios would have to support decompression... and even if that's done the decompression has to perform fast enough so games are still playable.
it seems like each file would have to be encrypted individually

Loading compressed files with a cIOS isn't easy. The PC .iso tools don't have 2 restrictions the cIOS has:
- The cIOS has very limited memory, you may be able to use a few KB(

 

[Wiimm]

- It has to work with random access, something is read @20320KB within the .iso, you can't decompress a 2MB chunk to retieve 2 KB requested data. I think this is the biggest problem and most likely why we will never see this in usb loaders.

Before starting to implement I have made a test:
I have taken a decrypted image (animal crossing):[*]Test #1: packed with bzip2[*]Test #2: split it into peaces of 2 MB and pack each peace. Contact all together and compare the size with test #1Because bzip2 works in chunks (100KB-900kb, user defined) the splitted files are only 1% larger than the unspllttet one. It seems that bzip2 is good for small chunks, but it needs about 500k memory.

I have not tested other compressions but perhaps any other (vexig for 7z?) can do this.

But decryption makes it possible to detect repeated values. Perhaps it is the best to use an old and easy compression method for usb loading.

 

[Wiimm]

A little uncompressing test.

Copy and convert a "Wii Sports" WDF, WIA and WIA/UNCOMPRESSED to a new WDF:

ÂÂ 25 secÂÂWDF -> WDF
ÂÂ 63 secÂÂWIA/UNCOMPRESSED -> WDF
ÂÂ152 secÂÂWIA/COMPRESSED -> WDF

 

[vexing]

- It has to work with random access, something is read @20320KB within the .iso, you can't decompress a 2MB chunk to retieve 2 KB requested data. I think this is the biggest problem and most likely why we will never see this in usb loaders.

that's an interesting point. i was thinking up a level, but looking at waninkoko's fat module for example it just handles direct access to offsets.

new super mario brothers wii, decrypted.wdf is 347,615 KiB.
using 7z, lzma compression, on ultra settings reduces it to 214,570 KiB
using 7z, lzma compression, 64 KiB dictionary reduces it to 236,754 KiB

now, taking the file and splitting it into chunks and compressing with lzma:
64 KiB chunks: 237,314 KiB
8 KiB chunks: 252,601 KiB

i also took a fairly representative of compression 1mb part from the iso, chunked it up, compressed

chunk size : combined size, both in KiB
1: 922
2: 841
4: 788
8: 753
16: 727
32: 709
64: 694

i'm starting to think this could work.

 

[W hat]

Some primitive compression might work for USB Loading, but it's almost completely pointless when you can have far better compression on the PC-side.

"smallest size possible" and "fast as possible" together is not possible.

For the "fast as possible" people, Bzip2 is a great choice. In the long term, I don't care about spending 1/2 hour or even 1 full hour compressing a game with 7zip if it's a game I just want to back up to my computer.

 

[vexing]

Some primitive compression might work for USB Loading, but it's almost completely pointless when you can have far better compression on the PC-side.

i think the opposite... imagine a scrubbed iso going from 4gb to 2.8gb and still directly playable via usb loading, would you really bother with decompression/compression to archive it on your pc at 2.4gb? i know i wouldn't.

 

[W hat]

Some primitive compression might work for USB Loading, but it's almost completely pointless when you can have far better compression on the PC-side.

i think the opposite... imagine a scrubbed iso going from 4gb to 2.8gb and still directly playable via usb loading, would you really bother with decompression/compression to archive it on your pc at 2.4gb? i know i wouldn't.

We all have different needs, and I wouldn't trust my WBFS drive with any data. I hate that format, and I'm just waiting for the day when either
1. The filesystem dies and I have to re-install my games in fat32 / ntfs
2. Wiimm, master of Wii, makes a total drive converter (the process would take a lot of HDD space while it was working, on at least two drives. )

 

[vexing]

We all have different needs, and I wouldn't trust my WBFS drive with any data. I hate that format, and I'm just waiting for the day when either
1. The filesystem dies and I have to re-install my games in fat32 / ntfs
2. Wiimm, master of Wii, makes a total drive converter (the process would take a lot of HDD space while it was working, on at least two drives. )

well, a wii compressed file system would probably make this easier to implement, but it's not required. these could possibly be a container like .wbfs files that would be readable on a fat32 drive.

regarding a drive converter, it seems like it'd be possible to directly convert a wbfs into fat32. that is don't modify any of the iso files, just create the file system that contains the pointers to all the files. i believe the data can be kept 100% the same.
the only issues i can think of are the wbfs probably doesn't reserve enough space at the beginning of the drive for the fat32 file system, so it might need to move the first iso, and it'd have to handle splitting files > 4gb, which might not be hard.

 

[Wiimm]

i also took a fairly representative of compression 1mb part from the iso, chunked it up, compressed

How do you find the "fairly representative of compression 1mb"?
I have splitted a whole image into peaces and have seen, that the compression ratio is very different for each peace.

 

[vexing]

How do you find the "fairly representative of compression 1mb"?
I have splitted a whole image into peaces and have seen, that the compression ratio is very different for each peace.

i split the whole image into the 64k chunks, then looked for a set of 16 of those that was around 700k compressed, which nearly matches the ratio of the whole image.

the whole image in 64k chunks was 68.3%
in 8k chunks was 72.7%
the 1mb in 64k chunks was 67.8%
in 8k chunks it was 73.5%

the 1mb seems a reasonable enough sample to give approximations of how the chunk size affects compression. anyway, the point was even with pretty small chunks like 16k you can get near 70% of original size, which means this is feasible for maintaining random access.

 

[Remfin]

Until you realize the lookup table for a disc stored in 8k chunks is 4 megs (4 bytes per entry, which may not be feasible).

Spoiler

Boy, I hope I did that math right

 

[WiiPower]

Wiimm, how much work would it be for you to write a "Wii Release Checker"?

Did you see thoese threads about Metroid Other M freezing? It looks like it's because of a broken release. Would you be interested in writing an app that tells:
- if a .iso is scrubbed or not
- checks the hashes of the .iso? (i guess the hashes for the scrubbed area don't match, but the rest just HAS to)
- tells if the update partition is missing

If you do that, it might be the 1st thing people do on new releases to immediately to post: "The release is incomplete, it's scrubbed", "The release is incomplete, there's no update partition" or "The release is broken, some parts(in the not scrubbed area) do not match the hashes, this WILL result in some kind of error in the game"

Until you realize the lookup table for a disc stored in 8k chunks is 4 megs (4 bytes per entry, which may not be feasible).

Spoiler

Boy, I hope I did that math right

I'm too tired to do the math right now, but i fear that at least the direction is correct and the size for the lookup table is one problem more for compressed .iso loading.

 

[Wiimm]

Wiimm, how much work would it be for you to write a "Wii Release Checker"? A command within wit is ok!?

- checks the hashes of the .iso? (i guess the hashes for the scrubbed area don't match, but the "wit verify image" does it. And "wit verify image --psel whole" does this in unscrubbed mode.

- tells if the update partition is missing "wit dump image" prints an overview.

Perhaps I should write a new command like "wit status image-list" and perhaps giantpune extents his gui to support it. Let me think about it.

QUOTE(WiiPower @ Aug 30 2010, 04:04 PM) QUOTE(Remfin @ Aug 30 2010, 03:38 PM)

Until you realize the lookup table for a disc stored in 8k chunks is 4 megs (4 bytes per entry, which may not be feasible).

I'm too tired to do the math right now, but i fear that at least the direction is correct and the size for the lookup table is one problem more for compressed .iso loading.

Is it?
The lookup table can be loaded partially. And the controller can always hold for example 4 peaces. I think reloading takes less time as loading and decompressing the real data.

 

[vexing]

Until you realize the lookup table for a disc stored in 8k chunks is 4 megs (4 bytes per entry, which may not be feasible).

you could have a two layer table, maybe 4700 entries stored in memory, each entry would cover a 1mb area, then the second lookup table could be stored in the 1mb area because the hard disk is going to be reading that area and caching it anyway.