Hacking WIA = Wii ISO Archive

Wiimm

Developer
OP
Member
Joined
Aug 11, 2009
Messages
2,286
Trophies
0
Location
Germany
Website
wiimmfi.de
XP
1,442
Country
Germany
Tagg7 said:
Does this mean that you are able to convert unscrubbed Wii ISOs (aka untouched) into WIA format and back again? This has always been a problem in the Wii scene (that all releases are 4.7GB because of the encrypted filler data).
Yes, WIA is only another container like WBFS, CISO or WDF. My tools support scrubbed (default) and non scrubbed copying. If you convert any ISO to WIA and back the result is the same as the original ISO. And this includes also hash errors.

QUOTE(Fat D @ Aug 29 2010, 04:15 PM) That would require the garbage data to make any sense when decrypted - but it does not, that is why it is called garbage. ...
If I remember right some old test of me, the decrypted garbage of some blocks is a zero filled data block. And this is the reason that WDF/DECRYPTED is smaller than standard WDF (see statistics in the first post).
 

Wiimm

Developer
OP
Member
Joined
Aug 11, 2009
Messages
2,286
Trophies
0
Location
Germany
Website
wiimmfi.de
XP
1,442
Country
Germany
giantpune said:
@ wimm
the structure for GC games is very similar to a wii partition. you use the same method to find the fst.bin, apploader, dol etc. the main difference in GC games is that there is only 1 partition, there is no ticket/tmd/cert, there is no data encryption, and offsets are exactly as you read them from the disc ( no need to do "
 

wishmasterf

Well-Known Member
Member
Joined
Apr 20, 2009
Messages
167
Trophies
0
Location
Austria
Website
Visit site
XP
82
Country
Australia
First of all many thanks for sharing your great tool.

I think the goals of the new format should besmallest size possible
easy to use
fast as possible
loadable in USB-LoadersThe first 2 things seems to be done

@giantpune and wiipower:

it is great to see that you support wimm! im not a developer
and do not know all about the disc-formats so i only can help
with testing. It would be great if you 3 guys could create that
ultimate format ;-)
 

Remfin

Well-Known Member
Newcomer
Joined
Mar 5, 2010
Messages
65
Trophies
0
XP
75
Country
United States
Wiimm said:
Thanx giant. I think I can borrow a disc from a friend for analysis. Do you (or any other) know a good reference about GC disc structure (before I talk to google and browse all bad sites)?
Yet Another Gamecube Documentation. I implemented the "what's actually in the encrypted data" stuff for my Wii program working off only that since Wiibrew didn't talk about what the layout of the encrypted data actually was. It's been a long time but I think the only things that I had to change was the bit-shiftings giantpune mentioned (header's offsets and FST's file offsets), so you probably already have all the code written (except for a few ifs).
 

vexing

Well-Known Member
Newcomer
Joined
Nov 16, 2009
Messages
65
Trophies
0
XP
27
Country
United States
Wiimm said:
Code:
Summary of RHAP01, Wii Play:
52934196  52.07%	  56.550 sec  WIA/NOCOMPRESS + 7Z
...
80354061  79.04%	  51.282 sec  WIA/NOCOMPRESS + BZIP2

10% slower, and 2/3rds the size.
so, any plans to add 7z's lzma compression? i haven't looked at your code, or their sdk, but i can't imagine it'd be too difficult


QUOTE(wishmasterf @ Aug 29 2010, 10:17 AM) loadable in USB-Loaders[/quote]

as wiipower mentioned a cios would have to support decompression... and even if that's done the decompression has to perform fast enough so games are still playable.
it seems like each file would have to be encrypted individually
 

WiiPower

Well-Known Member
Member
Joined
Oct 17, 2008
Messages
8,165
Trophies
0
XP
345
Country
Gambia, The
vexing said:
[...]
wishmasterf said:
loadable in USB-Loaders

as wiipower mentioned a cios would have to support decompression... and even if that's done the decompression has to perform fast enough so games are still playable.
it seems like each file would have to be encrypted individually

Loading compressed files with a cIOS isn't easy. The PC .iso tools don't have 2 restrictions the cIOS has:
- The cIOS has very limited memory, you may be able to use a few KB(
 

Wiimm

Developer
OP
Member
Joined
Aug 11, 2009
Messages
2,286
Trophies
0
Location
Germany
Website
wiimmfi.de
XP
1,442
Country
Germany
WiiPower said:
- It has to work with random access, something is read @20320KB within the .iso, you can't decompress a 2MB chunk to retieve 2 KB requested data. I think this is the biggest problem and most likely why we will never see this in usb loaders.
Before starting to implement I have made a test:
I have taken a decrypted image (animal crossing):[*]Test #1: packed with bzip2[*]Test #2: split it into peaces of 2 MB and pack each peace. Contact all together and compare the size with test #1Because bzip2 works in chunks (100KB-900kb, user defined) the splitted files are only 1% larger than the unspllttet one. It seems that bzip2 is good for small chunks, but it needs about 500k memory.

I have not tested other compressions but perhaps any other (vexig for 7z?) can do this.

But decryption makes it possible to detect repeated values. Perhaps it is the best to use an old and easy compression method for usb loading.
 

Wiimm

Developer
OP
Member
Joined
Aug 11, 2009
Messages
2,286
Trophies
0
Location
Germany
Website
wiimmfi.de
XP
1,442
Country
Germany
A little uncompressing test.

Copy and convert a "Wii Sports" WDF, WIA and WIA/UNCOMPRESSED to a new WDF:
Code:
ÂÂ 25 secÂÂWDF -> WDF
ÂÂ 63 secÂÂWIA/UNCOMPRESSED -> WDF
ÂÂ152 secÂÂWIA/COMPRESSED -> WDF
 

vexing

Well-Known Member
Newcomer
Joined
Nov 16, 2009
Messages
65
Trophies
0
XP
27
Country
United States
WiiPower said:
- It has to work with random access, something is read @20320KB within the .iso, you can't decompress a 2MB chunk to retieve 2 KB requested data. I think this is the biggest problem and most likely why we will never see this in usb loaders.

that's an interesting point. i was thinking up a level, but looking at waninkoko's fat module for example it just handles direct access to offsets.

new super mario brothers wii, decrypted.wdf is 347,615 KiB.
using 7z, lzma compression, on ultra settings reduces it to 214,570 KiB
using 7z, lzma compression, 64 KiB dictionary reduces it to 236,754 KiB

now, taking the file and splitting it into chunks and compressing with lzma:
64 KiB chunks: 237,314 KiB
8 KiB chunks: 252,601 KiB

i also took a fairly representative of compression 1mb part from the iso, chunked it up, compressed

chunk size : combined size, both in KiB
1: 922
2: 841
4: 788
8: 753
16: 727
32: 709
64: 694

i'm starting to think this could work.
 

W hat

Rhythm Heaven Fan
Member
Joined
Feb 28, 2007
Messages
630
Trophies
0
XP
675
Country
United States
Some primitive compression might work for USB Loading, but it's almost completely pointless when you can have far better compression on the PC-side.

Wiimm said:
"smallest size possible" and "fast as possible" together is not possible.

For the "fast as possible" people, Bzip2 is a great choice. In the long term, I don't care about spending 1/2 hour or even 1 full hour compressing a game with 7zip if it's a game I just want to back up to my computer.
 

vexing

Well-Known Member
Newcomer
Joined
Nov 16, 2009
Messages
65
Trophies
0
XP
27
Country
United States
W hat said:
Some primitive compression might work for USB Loading, but it's almost completely pointless when you can have far better compression on the PC-side.

i think the opposite... imagine a scrubbed iso going from 4gb to 2.8gb and still directly playable via usb loading, would you really bother with decompression/compression to archive it on your pc at 2.4gb? i know i wouldn't.
 

W hat

Rhythm Heaven Fan
Member
Joined
Feb 28, 2007
Messages
630
Trophies
0
XP
675
Country
United States
vexing said:
W hat said:
Some primitive compression might work for USB Loading, but it's almost completely pointless when you can have far better compression on the PC-side.

i think the opposite... imagine a scrubbed iso going from 4gb to 2.8gb and still directly playable via usb loading, would you really bother with decompression/compression to archive it on your pc at 2.4gb? i know i wouldn't.
We all have different needs, and I wouldn't trust my WBFS drive with any data. I hate that format, and I'm just waiting for the day when either
1. The filesystem dies and I have to re-install my games in fat32 / ntfs
2. Wiimm, master of Wii, makes a total drive converter (the process would take a lot of HDD space while it was working, on at least two drives. )
 

vexing

Well-Known Member
Newcomer
Joined
Nov 16, 2009
Messages
65
Trophies
0
XP
27
Country
United States
W hat said:
We all have different needs, and I wouldn't trust my WBFS drive with any data. I hate that format, and I'm just waiting for the day when either
1. The filesystem dies and I have to re-install my games in fat32 / ntfs
2. Wiimm, master of Wii, makes a total drive converter (the process would take a lot of HDD space while it was working, on at least two drives. )

well, a wii compressed file system would probably make this easier to implement, but it's not required. these could possibly be a container like .wbfs files that would be readable on a fat32 drive.

regarding a drive converter, it seems like it'd be possible to directly convert a wbfs into fat32. that is don't modify any of the iso files, just create the file system that contains the pointers to all the files. i believe the data can be kept 100% the same.
the only issues i can think of are the wbfs probably doesn't reserve enough space at the beginning of the drive for the fat32 file system, so it might need to move the first iso, and it'd have to handle splitting files > 4gb, which might not be hard.
 

Wiimm

Developer
OP
Member
Joined
Aug 11, 2009
Messages
2,286
Trophies
0
Location
Germany
Website
wiimmfi.de
XP
1,442
Country
Germany
vexing said:
i also took a fairly representative of compression 1mb part from the iso, chunked it up, compressed
How do you find the "fairly representative of compression 1mb"?
I have splitted a whole image into peaces and have seen, that the compression ratio is very different for each peace.
 

vexing

Well-Known Member
Newcomer
Joined
Nov 16, 2009
Messages
65
Trophies
0
XP
27
Country
United States
Wiimm said:
How do you find the "fairly representative of compression 1mb"?
I have splitted a whole image into peaces and have seen, that the compression ratio is very different for each peace.

i split the whole image into the 64k chunks, then looked for a set of 16 of those that was around 700k compressed, which nearly matches the ratio of the whole image.

the whole image in 64k chunks was 68.3%
in 8k chunks was 72.7%
the 1mb in 64k chunks was 67.8%
in 8k chunks it was 73.5%

the 1mb seems a reasonable enough sample to give approximations of how the chunk size affects compression. anyway, the point was even with pretty small chunks like 16k you can get near 70% of original size, which means this is feasible for maintaining random access.
 

Remfin

Well-Known Member
Newcomer
Joined
Mar 5, 2010
Messages
65
Trophies
0
XP
75
Country
United States
Until you realize the lookup table for a disc stored in 8k chunks is 4 megs (4 bytes per entry, which may not be feasible).

Boy, I hope I did that math right
 

WiiPower

Well-Known Member
Member
Joined
Oct 17, 2008
Messages
8,165
Trophies
0
XP
345
Country
Gambia, The
Wiimm, how much work would it be for you to write a "Wii Release Checker"?

Did you see thoese threads about Metroid Other M freezing? It looks like it's because of a broken release. Would you be interested in writing an app that tells:
- if a .iso is scrubbed or not
- checks the hashes of the .iso? (i guess the hashes for the scrubbed area don't match, but the rest just HAS to)
- tells if the update partition is missing

If you do that, it might be the 1st thing people do on new releases to immediately to post: "The release is incomplete, it's scrubbed", "The release is incomplete, there's no update partition" or "The release is broken, some parts(in the not scrubbed area) do not match the hashes, this WILL result in some kind of error in the game"

Remfin said:
Until you realize the lookup table for a disc stored in 8k chunks is 4 megs (4 bytes per entry, which may not be feasible).

Boy, I hope I did that math right

I'm too tired to do the math right now, but i fear that at least the direction is correct and the size for the lookup table is one problem more for compressed .iso loading.
 

Wiimm

Developer
OP
Member
Joined
Aug 11, 2009
Messages
2,286
Trophies
0
Location
Germany
Website
wiimmfi.de
XP
1,442
Country
Germany
WiiPower said:
Wiimm, how much work would it be for you to write a "Wii Release Checker"? A command within wit is ok!?

WiiPower said:
- checks the hashes of the .iso? (i guess the hashes for the scrubbed area don't match, but the "wit verify image" does it. And "wit verify image --psel whole" does this in unscrubbed mode.

WiiPower said:
- tells if the update partition is missing "wit dump image" prints an overview.

Perhaps I should write a new command like "wit status image-list" and perhaps giantpune extents his gui to support it. Let me think about it.

QUOTE(WiiPower @ Aug 30 2010, 04:04 PM) QUOTE(Remfin @ Aug 30 2010, 03:38 PM)
Until you realize the lookup table for a disc stored in 8k chunks is 4 megs (4 bytes per entry, which may not be feasible).

I'm too tired to do the math right now, but i fear that at least the direction is correct and the size for the lookup table is one problem more for compressed .iso loading.
Is it?
The lookup table can be loaded partially. And the controller can always hold for example 4 peaces. I think reloading takes less time as loading and decompressing the real data.
 

vexing

Well-Known Member
Newcomer
Joined
Nov 16, 2009
Messages
65
Trophies
0
XP
27
Country
United States
Remfin said:
Until you realize the lookup table for a disc stored in 8k chunks is 4 megs (4 bytes per entry, which may not be feasible).

you could have a two layer table, maybe 4700 entries stored in memory, each entry would cover a 1mb area, then the second lookup table could be stored in the 1mb area because the hard disk is going to be reading that area and caching it anyway.
 
General chit-chat
Help Users
  • No one is chatting at the moment.
  • Dark_Phoras @ Dark_Phoras:
    The younger generation uses mostly TikTok and YouTube, some Instagram. It seems like they don't care so much about social profiles.
    Gift
  • KenniesNewName @ KenniesNewName:
    I just keep it for worthless friends and families that don't do nothing
    Gift
  • Dark_Phoras @ Dark_Phoras:
    I have no idea what's happening with all my friends from other places
    Gift
  • KenniesNewName @ KenniesNewName:
    It's weird people in their mid 50s-20s seems to like social media more than TikTok and YouTube
    Gift
  • KenniesNewName @ KenniesNewName:
    Unless they get paid to upload videos
    Gift
  • Dark_Phoras @ Dark_Phoras:
    I used TikTok a couple of days and got tired
    Gift
  • Dark_Phoras @ Dark_Phoras:
    But it's still better than Facebook
    Gift
  • KenniesNewName @ KenniesNewName:
    One dude here swore I was trolling him on twitch and I never even used it
    Gift
  • Dark_Phoras @ Dark_Phoras:
    But that's a thing with social media; it creates an illusion of proximity, when in truth everyone is overwhelmed with all the information and all the social demands in the platforms
    Gift
  • Dark_Phoras @ Dark_Phoras:
    And a lot of people are addicted to the phone, checking social media or talking with someone. Some exercise in the gym while being glued to the phone, not even activity takes them away from it
    Gift
  • Dark_Phoras @ Dark_Phoras:
    And I was like that too, until I left social media. Now sometimes I'm grabbing my phone, opening the browser and thinking: I've seen everything I wanted to see, I don't have a reason to use the phone. And I put it down
    Gift
  • KenniesNewName @ KenniesNewName:
    I'd rather look at my phone than sit at a computer 24/7
    Gift
  • KenniesNewName @ KenniesNewName:
    Funnily when I'm out I hardly use my phone other than checking a message here and there
    Gift
  • KenniesNewName @ KenniesNewName:
    I'm mostly too busy making sure my neighbor doesn't get himself arrested
    Gift
  • Dark_Phoras @ Dark_Phoras:
    This evening two guys were speeding head first into my car in these:
    bolt-announces-a-custom-built-scooter-model-2-1024x536.jpg
    Gift
  • Dark_Phoras @ Dark_Phoras:
    I didn't make case, I made my manoeuver to park the car. But I caught them off-guard, the one ahead hit the brakes and had to jump from the scooter. For a second I thought he would go against the car. They were trying to race through a gap they perceived, thinking I was going to always drive forward
    Gift
  • Dark_Phoras @ Dark_Phoras:
    I guess they learned a valuable lesson, don't assume that real roads are a playground
    Gift
  • KenniesNewName @ KenniesNewName:
    I had a woman walk in the middle of the intersection once I was debating on flooring it
    Gift
  • Dark_Phoras @ Dark_Phoras:
    In those cases I honk
    Gift
  • KenniesNewName @ KenniesNewName:
    I need to get my uncle to install a cow horn
    Gift
  • Dark_Phoras @ Dark_Phoras:
    Those scooters go 40km/h, and the kids were riding them with no helmets
    Gift
  • Dark_Phoras @ Dark_Phoras:
    And trying to make for the gap like they're Millennium Falcon
    Gift
  • KenniesNewName @ KenniesNewName:
    Annoying can't find any type c to usb adapters in my place at least they're like $3
    Gift
  • KenniesNewName @ KenniesNewName:
    Sweet found one installing emudeck
    Gift
  • KenniesNewName @ KenniesNewName:
    Lol stupid ips is blocking rom sites time for a vpn
    Gift
    KenniesNewName @ KenniesNewName: Lol stupid ips is blocking rom sites time for a vpn