Hacking Thought on making the downloads smaller

[Dack]

Was just wondering if anyone had thought about this.

1. Wii games are encrypted on the disk. The encryption being such that the disk image will not compress - with rare exceptions e.g. Puzzle Quest

2. It is now possible to open up the encrypted discs to see the actual size of the used data.

What would happen if, for example, the ISO image was doctored to blank fill the non-used area of the image with compressable data. It would then be possible to compress it, the uncompressed one still being the correct 'full iso' in size. The game should also still run as no data will be missing and no signing of code required..

Just a thought (and if I didn't have a stinking head cold I'd give it a go)

 

[sidneyyoung]

Puzzle Quest isn't compressed, it was just an error about the size in the nfo, the download is still 4+gigs

Good idea about opening the encryted images and then shrinking them, but you over-looking the fact there are no tools afaik to rebuild and resign them once unpacked.

 

[Dack]

But you don't need to resign them

The download for Puzzle quest (European) was only 300Meg or so, I tried last night, the expanded image was then the usual 4 Gig.

If you know where in the ISO image file the data ends and then basically block fill from there to the end with 0's then the modified ISO file will then compress.

 

[FAST6191]

Wait unsigned discs can run now? news to me if true (and will probably result in me grabbing one of the various chips I have and installing it).

Most compression is based on on the dictionary method ( http://www.cs.ucl.ac.uk/staff/jon/mmbook/book/node101.html ) but it might be possible to adapt the decryption method (I only quickly looked at it and what amounts to little more than gut feeling suggests it could take to it in a halfway acceptable manner), perhaps it would be possible to make it lossy and maybe tack on some specially tailored parity (think par2) stuff.

The main problem is that this would be a completely new standard and trying to get it accepted would be a massive struggle (I still occasionally get PM/emails when I use 7zip (even in these sort of circles) despite it being fairly broadly accepted for a few years now on most (operating) systems and see the albeit minor backlash when various people decided to make multi rar files for DS releases).

 

[sidneyyoung]

Wow that is weird, the Puzzlequest release is 6*50mb after all.

 

[Dack]

The entropy of the scrambled data is such that my gut feeling is it won't compress.

I think there is a bit of confusion here over what I'm suggesting/hypothesizing.

1. If you read in a 'backup' (sic) disc using something like imgeburn you will end up with an 4Gig ISO file on your hard drive.

2. If you then use one of the available tools to check on the actual size of the content of the ISO, you can then add on the size of the update partition to give you the point in the ISO file where the data ends. Respecting the block boundaries - so you don't overwrite half a potentialy signed/crypted block.

3. Modifying the ISO file, by overwriting from the end of the data to the end of the ISO, with blank data would then allow it to be compressed.

The discs themselves would only seem to be signed in respect to the signature check for running (that the mod chip drive controller fix). The actual blank content of the data on the disc itself wouldn't need to be signed as it would never be read/checked. If data is read off the disc by the game then it would/could be checked internally. If not read then there shouldn't be a problem.

As to the compression used - it would still be multipart rar, just that you would actually get some compression instead of just having the RAR as a container

Other game that compressed well was Spyro (USA)

 

[niklasnyfiken]

Yes, this should work because there's no way Nintendo checks the garbage. The discs are 4.7GB big (or twice that for dual layer games), and it would take a LONG time to check all of that before letting the game run. So of course the Wii only checks the signature for the data it actually reads from the disk. I'm surprised no-one has written a utility like this yet.

 

[Dack]

Well only recently was there a method to see inside the iso's.

The only potential problem here would be if the data was not laid out sequentialy (padding data to the outside to increase access speed etc.)

What would really need to be done is a utility to analyse where the files are mapped inside the ISO and then blank the unused areas.

 

[Pikachu025]

Wii discs - like GC discs - have their game data on the most outwards point possible, I think. So it's something like this:

[start of iso] - Header - Update partition - garbage data - Game partition - [end of iso]

 

[Dack]

Just to let people know - This does work

I wrote an application to decrypt the disc and then go through it marking the areas where there is no data present. The modified iso is then created.

I managed to compress the ISO of Nitrobike down to 1 Gig or so.

Interesting project but somewhat pointless in that once the image has been created it never needs to be run on it again - it only needs to be run on each of the images once. These can then be compressed for easier storage.

Slight update:
Tried a few other images I had to hand - Go Diego Go ends up as 3 Gig while Puzzle Quest (which is what started me off on this) ended up at 280Meg. Should make backing up compressed images a bit easier.

 

[Rebound]

I don't understand, so if you put that Nitrobike on the internet I could download it and it would only be 1 gig? then could I just burn it straight off or would i have to do something with it first

 

[FAST6191]

@Rebound. Wii discs are read straight from the disc in a sector by sector copy (which is why you need a special drive if you are going to use a PC).
This copy is encrypted/signed and takes up the whole disc (4 odd gigs).
Recently the encryption was broken when it came to actually getting the files it held but not putting it back together.
These files were often far smaller than the disc (often less than 2 gigabytes) which naturally frustrated people who were forced to store files 4 gigabytes in size.
Normal compression is based on taking a chunk of code and giving a number/letter (4 being far smaller than 1728765376 and if 1728765376 is repeated several times.....), wii isos however do not fare well under this form of compression as the data is essentially random and does not repeat so much.
Dack hypothesised that the decryption algorithm could be turned to describe the files (for example 123456789 can be described by saying 1(+1x8): 7 characters versus 9) and apparently he pulled it off.

 

[Dack]

I don't understand, so if you put that Nitrobike on the internet I could download it and it would only be 1 gig? then could I just burn it straight off or would i have to do something with it first

Yep - If I uploaded the RAR pack for Nitrobike it would take up 1 Gig. When the RAR was then decompressed it would form a 4.5Gig ISO image that could simply be written out. No other operations would need to be carried out.

I called the application Scrubber (because it scrubs the unused data areas out of the encrypted ISO image, and I saw Up Pompeii recently

) - this allows the scrubbed image to then be compressed.

What I was wondering about next was trying to change a data file and see what happens (Manhunt?). If the WIIED application can recreate ISO from decrypted partitions then it must have a method of recrypting the 32k sector data.

 

[mb01]

So would this Nitrobike iso 4.5Gig actually boot? because if it does you are onto something.

 

[Dack]

Yep - it all works as per the 'original' ISO. It would have a different MD5 for the ISO image but the actual content is the same.

Trying to put this as simply as I can:

The ISO image consists of a data blocks showing where the files are located in the image and the files themselves. Due to the encryption on the discs a sector that is not occupied by a file still gets encrypted and as such does not compress when fed through winrar. What my app does is find out where these unused blocks are and then over-writes them in the image with a sector of 0xFFs - this *CAN* be compressed by winrar.

The sectors containing program data and associated data files remain untouched and so it all still functions as a 'real' disc - it's just it can be compressed now.

 

[mb01]

Well done. Are you going to release this app?

 

[FAST6191]

Interesting method Dack, I suppose the only problem would be if Nintendo implemented something similar to what we are seeing with the 360 and corrupt sectors.

Also is the FF region an exact science or did you build in a certain level of protection: I had a few GBA roms stop working (and I am told from reliable sources a similar thing happened for the DS with "safe trimming" although I never confirmed it for myself) because one of the overwritten/deleted sections was actually a file/text section end which then promptly crashed when it could not find an end.

I am for it if it can be considered safe but some scene standards would force a nuke from any "compressed release" using your method and if Nintendo do implement a check of some form there could be some bother (which would occur at some point down the line too leading to a state like the end of Golden Sun Team). Unless of course you decided to also make an unpatcher of some form (whether this would be patches for the sections or if it can be done with an algorithm remains to be seen I guess).
I am not quite sure how you would spread such a method either but can I suggest that you open the spec you use up to everyone (your app can stay closed if you like) and bundle it with all your app.

 

[Dack]

What I normally do when I write apps is get them in a usable state and then release them with the source code. As a lot of the info came from the wiifuse source (under GPL) then that would also only be the decent thing to do.

App is currently in a debug only state (in that there are lots of extra bits that were in there while I was developing e.g. I limited the disc size to 4.5G) - also I don't know what else to add to it.

As to the method of undoing then the simplest way would be to have the app create two images, one of the compressable version and the second with only the bits that were removed to make it compressable. That way you would have two rars, one containing the compressable and the second containing the rest. i.e. in the nitrobike case one rar set would be 1Gig and the second would be 3.5 Gig. A separate app could then merge the two of them recreating the original image.

 

[FAST6191]

I suppose the natural progression would be towards DVD9 which unless something has changed (my apathy towards SSB Brawl has grown over the last few days which has limited my knowledge of the area) people are effectively still using 2 single layer discs courtesy of them setting the layer break by hand. On the other hand it would probably be worth waiting until it all settles down and Wii DVD9 gets finalised for the "scene".

If you are going to go the patching route could I suggest an existing patcher like xdelta, bsdiff or PPF.

As for what to add that one is up to you, I imagine most people using such an app would be reasonably competent users meaning there is little need for a fancy GUI (if you do go the gui route could you leave a command line option too).
If you can stick with having the code using to decrypt (if indeed you use it) outside/added in the command line/app so as to help it be spread (some places do not like the code being posted).
Regarding extra features unless cross platform counts as a feature I would say get the compression/FF replacement sorted, maybe consider having the option to generate the replacement patches, get it fairly fast (although I suspect it is the hard drive that limits speed for the most part, especially if you are creating a new iso) and then consider extra features (extraction of wii isos/compression with rar and whatnot), cube stuff and whatever else you want a disc tool to do.

 

[mb01]

Well best of luck with tweaking your app mate. I would be very impressed to see such an app released soon, as would alot of other people. I have been eagerly awaiting an app like this for a long time now.