Inside WBFS

Wiimm · Oct 13, 2009

Inside WBFS
A few truths about WBFS

While I'm working with my WBFS tool (WWT) I take some deeper looks into the WBFS subsystem and found some interest things.

Overall in the internet you can read that the maximal number of discs is that a WBFS partition can contain is about 500. This is true but not the whole truth. The value depends of the sector size parameter. It defines the number of bytes of 1 hd sector. The first sector of a WBFS partition is reserved for the WBFS header (12 bytes) and for a disc table (all other bytes, 1 byte per disc).

The user/tool can set the sector size while formatting a WBFS partition. The default sector size is 512. WWT knows the option --sector-size and may format a WBFS partition with different sector sizes. This is nice for testing.

The following dump shows the specific parameters of a WBFS partition with about 2 tera bytes (a file with exact 2.000.000.000.000 bytes). WWT creates this as sparse file so that the WBFS partition only needs 16 KiB disc space after formatting:
Code:
DUMP of a.wbfs

WBFS-Header:
MAGIC:				  'WBFS' = 57 42 46 53
number of sectors:   279303400
hd sector size:			  9 ->		512
WBFS sector size:		   26 ->   67108864

hd:   sector size:		   512 = 2^9
hd:   num of sectors: 3906250000
hd:	total size:	   1907349 MiB

wii:  sector size:		 32768 = 2^15
wii:  num of sectors:   61035008
wii:  sectors/disc:	   286864
wii:   total size:	   1907344 MiB

wbfs: sector size:	  67108864 = 2^26
wbfs: num of sectors:	  29802
wbfs: sectors/disc:		  140
wbfs:  total size:	   1907328 MiB

partition lba:				 0
free blocks lba:		  131064
disc info size:			 1024

used disk space:			 640 MiB =   1%
free disk space:		 1906688 MiB =  99%
total disk space:		1907328 MiB = 100%

number of wii discs:		   0	 =   0%
max disc:					500
disc open:					 0
With sector size 512 the maximal number of discs is set to 500 (=512-12). But when a WBFS is formatted with sector size 2048 than it may contain up to 2036 (2048-12) discs. See the dump:
Code:
DUMP of a.wbfs

WBFS-Header:
MAGIC:				  'WBFS' = 57 42 46 53
number of sectors:  1143551290
hd sector size:			 11 ->	   2048
WBFS sector size:		   25 ->   33554432

hd:   sector size:		  2048 = 2^11
hd:   num of sectors:  976562500
hd:	total size:	   1907349 MiB

wii:  sector size:		 32768 = 2^15
wii:  num of sectors:   61034496
wii:  sectors/disc:	   286864
wii:   total size:	   1907328 MiB

wbfs: sector size:	  33554432 = 2^25
wbfs: num of sectors:	  59604
wbfs: sectors/disc:		  280
wbfs:  total size:	   1907328 MiB

partition lba:				 0
free blocks lba:		   16380
disc info size:			 2048

used disk space:			 640 MiB =   1%
free disk space:		 1906688 MiB =  99%
total disk space:		1907328 MiB = 100%

number of wii discs:		   0	 =   0%
max disc:				   2036
disc open:					 0
I don't know if other tools and the usb loaders support other sector sizes (this is subject of later tests, perhaps others will do that) but the solution is very easy: Read the WBFS sectors before calling the function wbfs_open_partition() and calculate the sector size:

CODEwbfs_head_t whead;
stat = ReadAt(file,0,&whead,sizeof(whead));
if (stat) { ERROR_HANDLING; }
sector_size = 1

Skizzo · Oct 14, 2009

See this http://gbatemp.net/index.php?showtopic=180935

I believe the current WBFS code uses the most efficient (for storage) sector size that would still allow the number of sectors to fit into a 16-bit value.

nubecoder · Oct 14, 2009

Quoted from Skizzo:

See this http://gbatemp.net/index.php?showtopic=180935

I believe the current WBFS code uses the most efficient (for storage) sector size that would still allow the number of sectors to fit into a 16-bit value.
Click to expand...

This is exactly why all of this is so confusing..

HD Sector Size:
All hard disk drives have a specific sector size, it cannot be modified. For most (all?) drives it is 512.

WBFS Sector Size:
All WBFS Formatted drives (partitions) have a dynamically set WBFS sector size. This seems to be calculated by making sure that the WBFS sector count will fit into 16 bits.

What Wiimm is talking about is telling the libWBFS code that the hard drives sector size is larger than it actually is.

It seems that this might be a workaround for the max game count limit.

But there is a limitation to take note of, you can only read data from a drive in multiples of the real HD sector size.

Also notice that the WBFS sector size is doubled in his example above (I'm not sure what effects this might have, if any).

Interesting stuff, nonetheless.

=]

Wiimm · Oct 14, 2009

Quoted from nubecoder:

HD Sector Size:
All hard disk drives have a specific sector size, it cannot be modified. For most (all?) drives it is 512.Yes that's the physical layer. But logical you can use a multiple of the physical sector size and many filesystems do that.

Quoted from nubecoder:

What Wiimm is talking about is telling the libWBFS code that the hard drives sector size is larger than it actually is.

It seems that this might be a workaround for the max game count limit.

But there is a limitation to take note of, you can only read data from a drive in multiples of the real HD sector size.
Not only a multiple: It must be a power of 2 because the sector size will be stored in the wbfs header as exponent with the default value "9" (2^9 == 512).

QUOTE(nubecoder @ Oct 14 2009, 03:10 AM)
Click to expand...

Also notice that the WBFS sector size is doubled in his example above (I'm not sure what effects this might have, if any).
Click to expand...

I have used factor 4 (2048 bytes). Now I have imported and exported about 20 games with sector size 2048 without any errors. It seems that the WBFS subsystems can handle other sectors sizes well.

Skizzo · Oct 14, 2009

It seems the two of you have a pretty good grasp of WBFS, so I'll ask you both, would a tool to resize a WBFS partition be possible or not? Just wondering if it is at all possible...before I take the time to start poking around. Sorry if this strays a bit OT.

Wiimm · Oct 14, 2009

Yes, resizing is possible (and hard work)

ChokeD · Oct 14, 2009

Finally a good read among the childish crap in other threads.

Thanks

LOTG · Oct 14, 2009

This is interesting.

Is there a specification somewhere about WBFS? Or is it all trial and error with the available sourcecode?

nIxx · Oct 14, 2009

There where very little infos about the specs in the header file anyway i think 500 Games are more as enough

KirovAir · Oct 14, 2009

Quoted from LOTG:

This is interesting.

Is there a specification somewhere about WBFS? Or is it all trial and error with the available sourcecode?
Click to expand...

I second this, a documentation about the WBFS subsystem would be a nice read, and resource.

Wiimm · Oct 15, 2009

I have tested sector-size = 2048 with USB loader cfg and Waninkokos USB load 1.5:
As expected, it does not work.

r-win · Oct 16, 2009

Wiimm,

allright, you've got us (USB Loader GX) partially convinced, so we implemented this change in the loader. However, I'm not sure if games will load, since you've also got cios, which contain a libwbfs on their own. If they are not patched, is this gonna work anyway?
Fastest way to try is using USB Loader GX rev 793+ with a changed sector size

Anyway, excellent find and excellent job!

Wiimm · Oct 16, 2009

Quoted from r-win:

Wiimm,

allright, you've got us (USB Loader GX) partially convinced, so we implemented this change in the loader. However, I'm not sure if games will load, since you've also got cios, which contain a libwbfs on their own. If they are not patched, is this gonna work anyway?
Fastest way to try is using USB Loader GX rev 793+ with a changed sector size

Anyway, excellent find and excellent job!
Click to expand...

Fine!

I have used WWT to add+extract about 50 games in 2 series (sector size 1024 and 2048) without any errors. The extracted games are identically to the sources. And this means: WWT is your friend for creating and filling WBFS partitions with other sector sizes.

Skizzo · Oct 20, 2009

Wiimm, if you have the time, could you explain what makes a resizing tool for WBFS such hard work?

Wiimm · Oct 20, 2009

It's hard (for me) to learn the details of wbfs. There is nearly none inline or other documentation. And before checking or resizing a wbfs partition the author must know precisely the impact of each parameter.

Wiimm · Oct 22, 2009

I have found a possible buffer overrun. Perhaps it never happens because there is no disc with more than 32 partitions but it's still a bug.
Code:
static void do_disc ( wiidisc_t * d )
{
ÂÂÂÂ...
ÂÂÂÂu64 partition_offset[32]; // XXX: don't know the real maximum
ÂÂÂÂu64 partition_type[32]; // XXX: don't know the real maximum
There is space only for 32 partitions.
u64 is not necessary because source data is only u32.
Code:
ÂÂÂÂ...
ÂÂÂÂn_partitions = _be32(b);
ÂÂÂÂdisc_read(d,_be32(b + 4), b, 0x100);
missing a check of 'n_partitions'. Also only data for max 32 partitions are read.

CODEÂÂÂÂfor (i = 0; i < n_partitions; i++)
ÂÂÂÂ{
ÂÂÂÂÂÂÂÂpartition_offset = _be32(b + 8 * i);
ÂÂÂÂÂÂÂÂpartition_type = _be32(b + 8 * i+4);
ÂÂÂÂÂÂÂÂ**BUMMM**

Dack · Oct 22, 2009

If you have a disc with 4 or more partitions then the Wii will refuse to read it anyways - it's a problem I encountered when I did the multidisc.

However......

The standard code included in the libwfs does not handle the second partition table (or third and fourth) - this means the VC games on SSBB are not seen for example. There doesn't seem to be limits on those tables.

You also need the u64 as for a Wii disc the stored values need to be multiplied by 4 to get actual - taking you over the 32 bit limit.

Wiimm · Oct 22, 2009

Quoted from Dack:

You also need the u64 as for a Wii disc the stored values need to be multiplied by 4 to get actual - taking you over the 32 bit limit.
Click to expand...

Not in this function: The data read is u32. And when the variables used the content will be copied back into u32 variables (function call). Calculations are not done in this function.

P.S.: Anything but important.

Dack · Oct 23, 2009

The problems with resizing come down to the field labeled "WBFS_SECTOR_SIZE" up above (and it's companions)

When a partition is initialised it effectively splits the storage space down into 64k chunks - these chunks are where the data is stored in the WBFS partition and represent a cluster of data. Each chunk is the size of WBFS_SECTOR_SIZE and starts at location partition start + wbfs header + game header table + usage table + WBFS_SECTOR_SIZE * cluster_number.

If you look at the header for a game placed on a WBFS partition you'll see the usual game name etc. and then a cluster usage table of 16 bit values. This table is created by down-converting the Wii's 32k usage map.
i.e. if the Wii disc showed data at 0x40000 to 0x50000 and 0xf800000 to 0x10000000 then 38000000 to 390000000 the cluster table would allocate data from the disc pool to cover those areas. The size of the games cluster table is derived from the size of a Wii disc divided by the WBFS_SECTOR_SIZE. If you look at the WBFS disc data with a hex editor you'll likely see some random data after each games cluster table as the routines don't usually pad the table out to the disc sector size.

This has the overall effect of each cluster containing WBFS_SECTOR_SIZE bytes of Wii disc data even if the original disc usage table showed only 32k as used.

When you resize the partition, the value of WBFS_SECTOR_SIZE etc. will be altered and data will need to be moved on the disc so that the pointers inside the games are corrected.

The limit on clusters is still 64k but each games cluster table entries will now be based around the old sized cluster/sector size and so data will be read incorrectly.

It's a lot of tweaking of data required as you have to modify each games cluster usage table entries that are already on the disc as well as the overall usage table..

The easiest way to resize would seem to be keep two copies of the usage table in memory. One representing the old size and the second representing the new size and copy the complete partition from one copy to the other, one game at a time. This is of course ensuring you don't overlap data blocks during the copy and you maintain disk usage tables correctly etc.

It's a lot of work for minimal real benefit.

arodey · Oct 23, 2009

This is all fascinating. (you all lost me before I started reading) But this is what this whole scene is about. Keep up the good work!

Useful Searches

OP Wiimm Developer Member Level 7

Skizzo Banned Banned Level 0

nubecoder Advanced Member Newcomer Level 1

OP Wiimm Developer Member Level 7

Skizzo Banned Banned Level 0

OP Wiimm Developer Member Level 7

ChokeD The Contributor Member Level 2

LOTG Advanced Member Newcomer Level 2

nIxx GBAtemp Advanced Maniac Member Level 3

KirovAir Dutch Alcoholic Programmer Member Level 8

OP Wiimm Developer Member Level 7

r-win GBAtemp Fan Member Level 1

OP Wiimm Developer Member Level 7

Skizzo Banned Banned Level 0

OP Wiimm Developer Member Level 7

OP Wiimm Developer Member Level 7

Dack GBAtemp Advanced Fan Member Level 1

OP Wiimm Developer Member Level 7

Dack GBAtemp Advanced Fan Member Level 1

arodey GBAtemp Regular Member Level 1

Hide similar threads Similar threads with keywords - Inside,

Share this thread:

OP Wiimm Developer
Member
Level 7

Skizzo Banned
Banned
Level 0

nubecoder Advanced Member
Newcomer
Level 1

OP Wiimm Developer
Member
Level 7

Skizzo Banned
Banned
Level 0

OP Wiimm Developer
Member
Level 7

ChokeD The Contributor
Member
Level 2

LOTG Advanced Member
Newcomer
Level 2

nIxx GBAtemp Advanced Maniac
Member
Level 3

KirovAir Dutch Alcoholic Programmer
Member
Level 8

OP Wiimm Developer
Member
Level 7

r-win GBAtemp Fan
Member
Level 1

OP Wiimm Developer
Member
Level 7

Skizzo Banned
Banned
Level 0

OP Wiimm Developer
Member
Level 7

OP Wiimm Developer
Member
Level 7

Dack GBAtemp Advanced Fan
Member
Level 1

OP Wiimm Developer
Member
Level 7

Dack GBAtemp Advanced Fan
Member
Level 1

arodey GBAtemp Regular
Member
Level 1