Hacking Inside WBFS

[Wiimm]

If you read this enum ...

typedef enum
{
ÂÂÂÂUPDATE_PARTITION_TYPEÂÂÂÂÂÂÂÂ= 0,
ÂÂÂÂGAME_PARTITION_TYPE,
ÂÂÂÂOTHER_PARTITION_TYPE,

ÂÂÂÂ// value in between selects partition types of that value
ÂÂÂÂALL_PARTITIONSÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ= 0xffffffff-3,
ÂÂÂÂREMOVE_UPDATE_PARTITION,ÂÂÂÂÂÂÂÂ// keeps game + channel installers
ÂÂÂÂONLY_GAME_PARTITION,

} partition_selector_t;

... and this function ....

static int test_partition_skip ( u32 partition_type, partition_selector_t part_sel )
{
ÂÂÂÂswitch (part_sel)
ÂÂÂÂ{
ÂÂÂÂÂÂÂÂcase ALL_PARTITIONS:
ÂÂÂÂÂÂÂÂÂÂÂÂreturn 0;
ÂÂÂÂÂÂÂÂcase REMOVE_UPDATE_PARTITION:
ÂÂÂÂÂÂÂÂÂÂÂÂreturn (partition_type==1);
ÂÂÂÂÂÂÂÂcase ONLY_GAME_PARTITION:
ÂÂÂÂÂÂÂÂÂÂÂÂreturn (partition_type!=0);
ÂÂÂÂÂÂÂÂdefault:
ÂÂÂÂÂÂÂÂÂÂÂÂreturn (partition_type!=part_sel);
ÂÂÂÂ}
}

... you see that the values of the constants 'UPDATE_PARTITION_TYPE' and 'GAME_PARTITION_TYPE' have been confused.

In detail:[*]test_partition_skip() returns 1 if the partition should be skipped (not selected).[*]'ONLY_GAME_PARTITION' checks partition type '0' but 'UPDATE_PARTITION_TYPE' is defined as '0'.[*]'REMOVE_UPDATE_PARTITION' checks partition type '1' but GAME_PARTITION_TYPE is defined as '1'.It seems that the correct definition is:

typedef enum
{
ÂÂÂÂGAME_PARTITION_TYPEÂÂ = 0,
ÂÂÂÂUPDATE_PARTITION_TYPE = 1,
ÂÂÂÂ...

Moreover there is no need for ONLY_GAME_PARTITION because (the corrected) GAME_PARTITION_TYPE does the same.


P.S.: I have already renamed test_parition_skip() to test_partition_skip().

 

[Wiimm]

@dack:
I fully agree.

Also, I've discovered some oddities in libwbfs. And that means that I have lost my last confidence in the stability of WBFS. I recommend to use a better tested file system as backup medium. And with an external backup medium resizing is no longer a problem -> just build the WBFS from scratch.

 

[Wiimm]

The size calculation of 'freeblocks' is wrong!

1.) This macro expands the size to the next full hd sector size. That's ok:

#define ALIGN_LBA(x) (((x)+p->hd_sec_sz-1)&(~(p->hd_sec_sz-1)))

2.) And this is the calculation of the free blocks size

ALIGN_LBA(p->n_wbfs_sec/8) >> p->hd_sec_sz_s

'n_wbfs_sec' is *not* always a multiple of 8. If n_wbfs_sec is for example 4100 than 'n_wbfs_sec/8' is exactily 512.5. And this means that 2 sectors are needed. But the integer calulation of 'n_wbfs_sec/8' is only 512 (exact one logical block).

Assuming a sector size of 512 this error appears for n_wbfs_sec that are multiple of 4096 + (1..7). For other values the LBA aligning is a kind of bug fix.

But it could be that these errors are never noticeable, because the other functions such as block_used() and alloc_block() have similar calculation errors and therefore the missing bits are never accessed.

 

[Wiimm]

It's not really "Inside WBFS" but I won't open a new thread.

While I'm implementing an iso dump function into 'wit' (Wiimms ISO Tool) I dumped SSBB too with curious results. Wiibrew tells me, that there are 4 partition tables and I found 13 partitions in the second partition tables. But they seems invalid because the data is stupid.

After that I modified CFG loader v44 and enable the 1:1 flag so that the whole disc is dumped (the new dump is larger than the old one). And the info dump of this disc dump is:

Dump of file ssbb-full.wdf

ÂÂ...
ÂÂISO file size:ÂÂ230000000/hex = 9395240960 = 8960 MiB

ÂÂ...

ÂÂ2 partition tables with 15 partitions:

ÂÂÂÂ tab.idxÂÂ n(part)ÂÂÂÂÂÂ offset(part.tab) .. end(p.tab)
ÂÂÂÂ--------------------------------------------------------
ÂÂÂÂÂÂÂÂ0ÂÂÂÂÂÂÂÂ2ÂÂÂÂÂÂ 10008*4 =ÂÂÂÂÂÂ40020 ..ÂÂÂÂÂÂ40030
ÂÂÂÂÂÂÂÂ1ÂÂÂÂÂÂ 13ÂÂÂÂÂÂ 10010*4 =ÂÂÂÂÂÂ40040 ..ÂÂÂÂÂÂ400a8

ÂÂ15 partitions:

ÂÂÂÂ indexÂÂÂÂÂÂtypeÂÂÂÂÂÂoffset .. end offsetÂÂ size/hex =ÂÂ size/dec =ÂÂMiB
ÂÂÂÂ--------------------------------------------------------------------------
ÂÂÂÂÂÂ0.0ÂÂ UPDATE 1ÂÂÂÂÂÂ 50000 ..ÂÂÂÂaea8000ÂÂÂÂae58000 =ÂÂ182812672 =ÂÂ174
ÂÂÂÂÂÂ0.1ÂÂÂÂ GAME 0ÂÂÂÂ f800000 ..ÂÂ1da550000ÂÂ1cad50000 = 7697924096 = 7341
ÂÂÂÂÂÂ1.0ÂÂ 48413850ÂÂ 1da550000ÂÂÂÂÂÂÂÂ ** INVALID PARTITION **
ÂÂÂÂÂÂ1.1ÂÂ 48413950ÂÂ 1daff0000ÂÂÂÂÂÂÂÂ ** INVALID PARTITION **
ÂÂÂÂÂÂ1.2ÂÂ 48424150ÂÂ 1dbaa0000ÂÂÂÂÂÂÂÂ ** INVALID PARTITION **
ÂÂÂÂÂÂ1.3ÂÂ 48424246ÂÂ 1dc570000ÂÂÂÂÂÂÂÂ ** INVALID PARTITION **
ÂÂÂÂÂÂ1.4ÂÂ 48424250ÂÂ 1dd0e0000ÂÂÂÂÂÂÂÂ ** INVALID PARTITION **
ÂÂÂÂÂÂ1.5ÂÂ 48424350ÂÂ 1ddc50000ÂÂÂÂÂÂÂÂ ** INVALID PARTITION **
ÂÂÂÂÂÂ1.6ÂÂ 48424450ÂÂ 1de720000ÂÂÂÂÂÂÂÂ ** INVALID PARTITION **
ÂÂÂÂÂÂ1.7ÂÂ 48424550ÂÂ 1df1d0000ÂÂÂÂÂÂÂÂ ** INVALID PARTITION **
ÂÂÂÂÂÂ1.8ÂÂ 48424650ÂÂ 1dfcc0000ÂÂÂÂÂÂÂÂ ** INVALID PARTITION **
ÂÂÂÂÂÂ1.9ÂÂ 48424750ÂÂ 1e08b0000ÂÂÂÂÂÂÂÂ ** INVALID PARTITION **
ÂÂÂÂÂÂ1.10ÂÂ48424950ÂÂ 1e1490000ÂÂÂÂÂÂÂÂ ** INVALID PARTITION **
ÂÂÂÂÂÂ1.11ÂÂ48424b50ÂÂ 1e2490000ÂÂÂÂÂÂÂÂ ** INVALID PARTITION **
ÂÂÂÂÂÂ1.12ÂÂ48424c50ÂÂ 1e5d40000ÂÂÂÂÂÂÂÂ ** INVALID PARTITION **

...

What does this mean?[*] Wiibrew is wrong[*] The disc dump is wrong[*] My tool is wrong (I don't believe that)[*] Are this the reasons for the problems while playing the game with an USB loader?

 

[Dr. Clipper]

By 4, are you referring to the fact it can't play the Masterpieces (i.e., the time-limited VC games)? Given the number of them, there could very well be one game per extra partition. What might help prove this is testing the Japanese version as it has two Masterpieces (Fire Emblem and Earthbound, IIRC) that aren't in the US/PAL versions.

 

[Wiimm]

By 4, are you referring to the fact it can't play the Masterpieces (i.e., the time-limited VC games)?

No, i read many times thas SSBB don't work because of this or that. And while I'm testing my dump function I found the above curiosity. And maybe these things are related somehow.

QUOTE(Dr. Clipper @ Oct 31 2009, 11:15 AM) What might help prove this is testing the Japanese version as it has two Masterpieces (Fire Emblem and Earthbound, IIRC) that aren't in the US/PAL versions.

The new WWT is online. Just say "wit dump iso_file..." to get dumps of that iso files. perhaps anyone can do that for me.

 

[noobwarrior7]

I may not be asking this right due to lack of technical knowledge, but does the layer break, new partition table, and new partition all begin at the same area of the dual-layer discs?

 

[Blue-K]

I may not be asking this right due to lack of technical knowledge, but does the layer break, new partition table, and new partition all begin at the same area of the dual-layer discs?

Layer Break is for every Wii Game the same (like with the XBox360 games). Dunno about the rest...

@Wiimm: Brawl doesn't work with cIOS r14 because the Dual Layer Support is broken in this cIOS. That's the official information. Proof for that: Brawl works with the cIOSes from Hermes and some cIOS before r14. So it's the cIOS, not more.

 

[Wiimm]

The second partition table resists at address 40040..400a8 and thats far away from the layer break. And the partition types and partition addresses looks curios and not like all other partitions.

 

[Dr. Clipper]

Wiimm, when you were testing the 1:1 option, were you using IOS249 rev 14 or something else? That particular IOS does have trouble with ripping both MPT and SSBB, so you should try the test again while using IOS222 or a different revision of IOS249 to see if that does change anything.

 

[Wiimm]

There is a function in libwbfs called wbfs_trim(). It is used by several programs (my wwt, oggzees wbfs_file, usb loaders) to trim a wbfs (=truncate the wbfs to only necessaries blocks).
I have found 3 (three!) bugs in it. Here is the original source:

CODEu32 wbfs_trim(wbfs_t*p)
{
ÂÂÂÂu32 maxbl;
ÂÂÂÂload_freeblocks(p);
ÂÂÂÂmaxbl = alloc_block(p);
ÂÂÂÂp->n_hd_sec = maxblhd_sec_sz_s);
ÂÂÂÂp->head->n_hd_sec = wbfs_htonl(p->n_hd_sec);
ÂÂÂÂ// make all block full
ÂÂÂÂmemset(p->freeblks,0,p->n_wbfs_sec/8);
ÂÂÂÂwbfs_sync(p);
ÂÂÂÂ// os layer will truncate the file.
ÂÂÂÂreturn maxbl;
}

BUGS:[*] To determine the last block the function alloc_block() is used. If there are already delete blocks alloc_blocks() returns NOT the last block.[*] If there are not any free blocks alloc_block() return ~0. Then the WBFS is not shrinked but enlarged to about 2 tera bytes (assuming sector size 512).[*] The calculation of 'p->n_hd_sec' is wrong. The value '1' for the wbfs header must be added. Without +1 the files becomes 1 sector (usual 512 bytes) short.My solution: 1.) implement the new function find_last_used_block() and 2.) modify wbfs_trim():

CODEstatic u32 find_last_used_block ( wbfs_t * p )
{
ÂÂÂÂint i;
ÂÂÂÂ// the division is a rounding error!
ÂÂÂÂfor ( i = p->n_wbfs_sec/(8*4) - 1; i >= 0; i-- )
ÂÂÂÂ{
ÂÂÂÂÂÂÂÂu32 v = wbfs_ntohl(p->freeblks);
ÂÂÂÂÂÂÂÂif ( v != ~(u32)0 )
ÂÂÂÂÂÂÂÂ{
ÂÂÂÂÂÂÂÂÂÂÂÂint j;
ÂÂÂÂÂÂÂÂÂÂÂÂfor ( j = 31; j >= 0; j-- )
ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂif ( !(v & 1hd_sec_sz_s;
ÂÂÂÂp->head->n_hd_sec = wbfs_htonl(p->n_hd_sec);

ÂÂÂÂ// make all block full
ÂÂÂÂmemset(p->freeblks,0,p->n_wbfs_sec/8);
ÂÂÂÂwbfs_sync(p);

ÂÂÂÂ// os layer will truncate the file.
ÂÂÂÂreturn max_block;
}

Cutting the file looks then like:

ÂÂÂÂwbfs_t * w = ...
ÂÂÂÂ...
ÂÂÂÂwbfs_trim(w);
ÂÂÂÂconst u64 cut = (u64)w->n_hd_sec * w->hd_sec_sz;
ÂÂÂÂset_file_size(file,cut);

Now it's time to test ...


P.S.:
You can use wwt (see signature) for dumping a memory map of any WBFS and controlling the results of trimming and other actions:

wwt dump -lll wbfs_file

EDIT: a little correction: part_lba+1 is not needed for size calculations.

 

[oggzee]

BUGS:[*] To determine the last block the function alloc_block() is used. If there are already delete blocks alloc_blocks() returns NOT the last block.[*] If there are not any free blocks alloc_block() return ~0. Then the WBFS is not shrinked but enlarged to about 2 tera bytes (assuming sector size 512).[*] The calculation of 'p->n_hd_sec' is wrong. The value 'p->part_lba + 1' must be added. Because 'p->part_lba' is mostly zero the partition is 1 sector to short.My solution: 1.) implement the new function find_last_used_block() and 2.) modify wbfs_trim():

It seems to me that the first 2 points can never happen when just adding 1 iso to a fresh wbfs file, which is the only scenario that wbfs_file does. If you want to use that to trim and resize a real partition then yes there are problems with it, but wbfs_file would never use it for that.
As for the third point, i don't understand why would you add part_lba? - If it is a real partition, then it's wrong, if it's a file it's always zero. So the only question remains is, if the +1 is needed? I seems to me that it isn't, since the just allocated block has the position past all data... and also all the converted iso to wbfs files seem to work fine... but i could be mistaken.

 

[Wiimm]

BUGS:[*] To determine the last block the function alloc_block() is used. If there are already delete blocks alloc_blocks() returns NOT the last block.[*] If there are not any free blocks alloc_block() return ~0. Then the WBFS is not shrinked but enlarged to about 2 tera bytes (assuming sector size 512).[*] The calculation of 'p->n_hd_sec' is wrong. The value 'p->part_lba + 1' must be added. Because 'p->part_lba' is mostly zero the partition is 1 sector to short.My solution: 1.) implement the new function find_last_used_block() and 2.) modify wbfs_trim():

It seems to me that the first 2 points can never happen when just adding 1 iso to a fresh wbfs file, which is the only scenario that wbfs_file does. If you want to use that to trim and resize a real partition then yes there are problems with it, but wbfs_file would never use it for that.
As for the third point, i don't understand why would you add part_lba? - If it is a real partition, then it's wrong, if it's a file it's always zero. So the only question remains is, if the +1 is needed? I seems to me that it isn't, since the just allocated block has the position past all data... and also all the converted iso to wbfs files seem to work fine... but i could be mistaken.

The second point can happen if the WBFS is full.

And you are right: p->part_lba is only needed for offsets but not for size. I change the post.

 

[Wiimm]

I'm not sure if bug 3 really happens. I will work on that later.
oggzee: thanx for brain storming

 

[Wiimm]

Bugs 1. and 2. are still there but does not appear in newly created wbfs that are not full.

Bug 3. is an interpretation error of me: At the beginning of a WBFS all WBFS sectors (multiple of 512, on 500G partitions about 8 MB) has an delta offset of 1 hd sector (typical 512 bytes) but in the data area that offset is 0. That has confused me.

 

[Wiimm]

I have made some more small tests. With sector size 512 (default) a WBFS must be smaller as 2 TiB. But when increasing the sector size larger WBFS partitions seems possible. To create a test partition use WWT and type for example:

wwt init --size 4t --sector-size 1024 --split-size 2t --force a.wbfs

Splitting is needed because the maximal single file size.

The created empty WBFS uses only 16 KiB of the ext3 harddisk:

# ls -lsh a.wbf*
ÂÂ0 -rw-r--r-- 1 root m7 1.9T 2009-11-06 17:40 a.wbf1
16K -rw-r--r-- 1 root m7 1.9T 2009-11-06 17:40 a.wbfs

And this is the WBFS dump:

# wwt dump -lll a.wbfs

DUMP of a.wbfs

ÂÂWBFS-Header:
ÂÂÂÂMAGIC:ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ'WBFS' = 57 42 46 53
ÂÂÂÂnumber of sectors:ÂÂ 279303400
ÂÂÂÂhd sector size:ÂÂÂÂÂÂÂÂÂÂÂÂ 10 ->ÂÂÂÂÂÂ 1024
ÂÂÂÂWBFS sector size:ÂÂÂÂÂÂÂÂÂÂ 26 ->ÂÂ 67108864

ÂÂÂÂDisc table:
ÂÂÂÂÂÂÂÂ0.. 59: .......... .......... .......... .......... .......... ..........
ÂÂÂÂÂÂ 60..119: .......... .......... .......... .......... .......... ..........
ÂÂÂÂÂÂ....
ÂÂÂÂÂÂ960..1011: .......... .......... .......... .......... .......... ..

ÂÂhd:ÂÂ sector size:ÂÂÂÂÂÂÂÂÂÂ1024 = 2^10
ÂÂhd:ÂÂ num of sectors: 3906250000
ÂÂhd:ÂÂÂÂtotal size:ÂÂÂÂÂÂ 3814697 MiB

ÂÂwii:ÂÂsector size:ÂÂÂÂÂÂÂÂ 32768 = 2^15
ÂÂwii:ÂÂnum of sectors:ÂÂ122070016
ÂÂwii:ÂÂsectors/disc:ÂÂÂÂÂÂ 286864
ÂÂwii:ÂÂ total size:ÂÂÂÂÂÂ 3814688 MiB

ÂÂwbfs: sector size:ÂÂÂÂÂÂ67108864 = 2^26
ÂÂwbfs: num of sectors:ÂÂÂÂÂÂ59604
ÂÂwbfs: sectors/disc:ÂÂÂÂÂÂÂÂÂÂ140
ÂÂwbfs:ÂÂtotal size:ÂÂÂÂÂÂ 3814656 MiB

ÂÂpartition lba:ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ 0
ÂÂfree blocks lba:ÂÂÂÂÂÂÂÂÂÂ 65528
ÂÂdisc info size:ÂÂÂÂÂÂÂÂÂÂÂÂ 1024

ÂÂused disk space:ÂÂÂÂÂÂÂÂÂÂÂÂ1280 MiB =ÂÂ 1%
ÂÂfree disk space:ÂÂÂÂÂÂÂÂ 3813376 MiB =ÂÂ99%
ÂÂtotal disk space:ÂÂÂÂÂÂÂÂ3814656 MiB = 100%

ÂÂnumber of wii discs:ÂÂÂÂÂÂÂÂÂÂ 0ÂÂÂÂ =ÂÂ 0%
ÂÂmax disc:ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ 1012
ÂÂdisc open:ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ 0


ÂÂWBFS Memory Map:

ÂÂÂÂÂÂÂÂunused :ÂÂoff(beg) ..ÂÂoff(end) :ÂÂÂÂ size : info
ÂÂ --------------------------------------------------------------------------------
ÂÂÂÂÂÂÂÂÂÂÂÂÂÂ :ÂÂÂÂÂÂÂÂ 0 ..ÂÂÂÂÂÂÂÂ c :ÂÂÂÂÂÂÂÂc : WBFS header
ÂÂÂÂÂÂÂÂÂÂÂÂÂÂ :ÂÂÂÂÂÂÂÂ c ..ÂÂÂÂÂÂ 400 :ÂÂÂÂÂÂ3f4 : Disc table
ÂÂÂÂÂÂÂÂÂÂÂÂÂÂ :ÂÂÂÂÂÂ 400 ..ÂÂÂÂ fd400 :ÂÂÂÂfd000 : Disc info
ÂÂÂÂÂÂ 3f00c00 :ÂÂ 3ffe000 ..ÂÂ 3fffd1a :ÂÂÂÂ 1d1a : Free blocks table
ÂÂÂÂ 3a34e9442e6 :3a352944000 ..3a352944000 :ÂÂÂÂÂÂÂÂ0 : -- end of WBFS device/file --

Partitions larger than 4 TiB are not possible. There seems a limit in "wbfs sector size".

 

[tj_cool]

Partitions larger than 4 TiB are not possible. There seems a limit in "wbfs sector size".

You have to be a really big pirate if you can fill 4 TebiByte

 

[Wiimm]

I'm really new in the WBFS business. When i take a look into the Wiki/WBFS managers I see "Sorgs's Mod" and other things. Can anybody please tell me:

• What is "Sorgs's Mod"?
  .
• What is the "Delete Bug Fix" in technical details?
  (Do I have them in WWT? Tests says: NO / I use kwiirk wbfs as base for my wwt.)
  .
• And btw: Where can I found the CISO specification.
  .

 

[Wiimm]

[*] What is the "Delete Bug Fix" in technical details?
(Do I have them in WWT? Tests says: NO / I use kwiirk wbfs as base for my wwt.)
.

I answer myself with a link: http://github.com/kwiirk/wbfs/commit/ce1d7...79a7ab77c77befd
I have the "Delete Bug Fix" (puuh...)

 

[Wiimm]

A good message for Win/NTFS users:
I have played with the newest cygwin version to prove the functionality. And what did I found?

NTFS supports sparse files too!

When creating a WBFS or extracting a raw ISO file from a small game the disc usages is much less than the file size. But when copying the ISO this advantage is lost. This means: WDF is still the best solution for storing ISO images.