Hacking WDF = Wii Disc File

[Wiimm]

WDF = Wii Disc File

Wii ISO images has a size of about 4.4 GiB, but much space is unused and have to be filled with zeros. Some games need less than 100 MiB and that means about 4.3 GiB waste. The linux ext2/3 file system can handle such files very good because it do not allocate disc space for such (zero-)holes. Such files called "sparse files".

For me there is no need to store Wii ISO images in the old way filled with zeros. And the handling with large sparse files is not effective and needs always special options like 'ls -s' or 'cp --sparse=always'. Also it costs much time because the holes have to be scanned again.

One way is to compress the data. But compressing works only for the holes but not for the crypt data. And random access (needed by the WBFS framework) is very difficult for compressed data. The direct support of holes seems to be much better.

Because of this all I have defined a new file format, the WDF format. There are some main issue for me:[*] The WDF format should handles holes effectively to save disk space.[*] It should support splitted files because small files can sometimes be handled more effectively than whole large files.[*] There must be a WBFS tool to handle this kind of files. I have already implemented the WDF support im my 'WWT' (Wiimms WBFS Tool). The WWT package contains some more tools to handle WBFS.[*] Other WBFS manager should support the WDF too. And this means that the implementation must be done by the well known wrapper functions 'rw_sector_callback_t' and 'read_wiidisc_callback_t'. That makes transfering the code very easy.I have already implemented this (but not splitted WDF) in my WWT tools and it works fine. The tools are:

• 'wwt' is a general WBFS manager that may import WDF files directly to a WBFS and may export discs from WBFS directly to WDF files.
• 'wdf-dump' dumps the data structure of a WDF file.
• 'iso2wdf' converts ISO images into WDF files.
• 'wdf2iso' converts WDF files into ISO images.
• 'wdf-cat' is a 'cat' like program with special handling of WDF.

The following table show the size of ISO images and WDF files of some games (my test suite).
CODE---------------------------------------------------------------------
id6 image size WDF size % name of disc
---------------------------------------------------------------------
RTYP01 4482 MiB 27 MiB

 

[tj_cool]

ever heard of cISO ?

 

[HackerOfWii]

Cool.

 

[KirovAir]

ever heard of cISO ?

cISO will keep the 4.4gig size on NTFS file systems?

 

[CasperH]

Isn't this the same as WiiScruber?

 

[pepxl]

@ DarkMystery

No it doesnt cISO's are compressed ISO's so the file sizes are game size only and not full ISO size with all the zeros, plus its NTFS not NTSC

 

[DjoeN]

Never used cISO, but whatever program compresses the best without loss of speed/quality is good for me

I only wish a Windows command tool of your wwt tools came out

, since i don't really like linux.
(Reads, i'm to lazy to work with linux now, after many failed attempts to)

 

[olliepop2000]

Tests need to be made to compare WDF and CISO to compare ripping speeds and file sizes. If there is significant improvement then maybe this is the way to go.

 

[KirovAir]

@ DarkMystery

No it doesnt cISO's are compressed ISO's so the file sizes are game size only and not full ISO size with all the zeros

Thanks, thought it kept the zero's.

QUOTE(pepxl @ Oct 4 2009, 12:11 PM) plus its NTFS not NTSC

common typo of me.

 

[Dack]

Isn't this the same as WiiScruber?

No - though it's had 'shrink' capability built in since about February (try using a resource file editor and enabling the hidden buttons on the 1.40 release

but I refined the format a bit after the release (which was one of the reasons I didn't activate the buttons))

Split files are an issue though as it's a bit pointless unless the IOSs support the split files format

 

[Chaos Punk]

WBFS (Wii Backup File Systeym) keeps the ISO sizes to their absolute minimums.

 

[Dack]

It doesn't actually - it's quite a horrible format when you start looking into it.

(e.g. Due to the 16 bit limitation on number of clusters, for a 1 terrabyte disc you end up having 'clusters' that are 14 Meg in size - this means files are then rounded upto that nearest size).

BTW the Wii Chess size in the test is a bit of a con as thats the version that has the update partition removed, in fact, looking at the results - they all have the updates removed, in the case of Mario Kart you've also stripped the channel.

 

[Wiimm]

ever heard of cISO ?heard? yes. But I never found a specification of it.
And ISO images aren't good candidates for compressing. Or means 'compressed' not something like zip/rar does?

I will make speed test later in the week. Here are the results of an quick test:
CODEAdding Zelda ISO:ÂÂÂÂ 31s
Adding Zelda WDF:ÂÂÂÂ 27s
Extracting Zelda ISO: 30s
Extracting Zelda WDF: 30s

 

[Wiimm]

I have made some test in the last days. Here are the results:

WDF does not compress but handles holes. And it is better in time and space as compression programs. I have compressed scrubbed ISOs (holes filled with 0x00) with gzip, bzi2, 7z and rar. The compression has taken much much time and the files are larger than the WDF files.

CODE----------------------------------------------------------------------------------
ÂÂÂÂÂÂÂÂÂÂÂÂWChÂÂÂÂ WPlÂÂÂÂ WMuÂÂÂÂ WSpÂÂÂÂ TLZÂÂÂÂ MKWÂÂÂÂ TRAÂÂÂÂ SSBÂÂÂÂ MPT
ÂÂÂÂÂÂÂÂÂÂ RTYP01ÂÂRHAP01ÂÂR64P01ÂÂRSPP01ÂÂRZDP01ÂÂRMCP01ÂÂRLRP4FÂÂRSBP01ÂÂR3MP01
----------------------------------------------------------------------------------
iso size:ÂÂ 4482ÂÂÂÂ4482ÂÂÂÂ4482ÂÂÂÂ4482ÂÂÂÂ4482ÂÂÂÂ4482ÂÂÂÂ4482ÂÂÂÂ7590ÂÂÂÂ8102
wdf size:ÂÂÂÂ 27ÂÂÂÂÂÂ99ÂÂÂÂ 137ÂÂÂÂ 635ÂÂÂÂ1101ÂÂÂÂ2647ÂÂÂÂ3171ÂÂÂÂ6907ÂÂÂÂ7563
wdf ratio:ÂÂ

 

[G0dLiKe]

Well, sounds good to me.

We need a OSX and/ or Win XP program. Please.

 

[Omega Knight]

Great to see some linux love. Keep it up :3

 

[wolffangalchemist]

while space is always a concern when storing data i don't see the point when it comes to a WBFS type of storage.
i have a 100gb wbfs partition and it has only 14 or so gigs used while some ppl might have more used if they copy every game know to existence to it(though that is kinda pointless).
simply because the amount if space each disc takes on A FS can be reduced by altering it doesn't mean you will save a considerable enough amount of space to justify it (meaning it's not gonna squeeze out enough space to make it so you can add a considerable amount of extra games.)

though carry on who knows it could do more than i think(increase load times speed and things of that nature) i'm always willing to test for things like this since i used Linux my self.

 

[Wiimm]

while space is always a concern when storing data i don't see the point when it comes to a WBFS type of storage.
i have a 100gb wbfs partition

To avoid misunderstandings: WDF will not replace WBFS. It is only a way to shrink ISO images on other file systems like ext2, FAT or NTFS. And while you are adding/extracting a WDF image to/from WBFS the transformation can be done on the fly without time costs.

 

[bowser]

I think WDF is a great idea. Right now I'm using a lot of disk space on my computer storing ISOs of Wii games. This will help clear up a lot of space. Keep up the good work!

PS: I don't feel like using WiiScrubber and then compressing the ISOs because i'll have to unzip them again before adding them to WBFS.

 

[nknave]

I think this will be good if the software was able to burn into a DVD-R directly from this WDF without any other conversion.