Inside WBFS

By Wiimm, Oct 13, 2009 41,041 84 0

Page 1 of 5
  1. Wiimm

    OP Wiimm Developer
    Member
    Level 7

    Joined:
    Aug 11, 2009
    Messages:
    2,266
    Country:
    Germany
    Inside WBFS
    A few truths about WBFS

    While I'm working with my WBFS tool (WWT) I take some deeper looks into the WBFS subsystem and found some interest things.

    Overall in the internet you can read that the maximal number of discs is that a WBFS partition can contain is about 500. This is true but not the whole truth. The value depends of the sector size parameter. It defines the number of bytes of 1 hd sector. The first sector of a WBFS partition is reserved for the WBFS header (12 bytes) and for a disc table (all other bytes, 1 byte per disc).

    The user/tool can set the sector size while formatting a WBFS partition. The default sector size is 512. WWT knows the option --sector-size and may format a WBFS partition with different sector sizes. This is nice for testing.

    The following dump shows the specific parameters of a WBFS partition with about 2 tera bytes (a file with exact 2.000.000.000.000 bytes). WWT creates this as sparse file so that the WBFS partition only needs 16 KiB disc space after formatting:

    Code:
    DUMP of a.wbfs

WBFS-Header:
MAGIC:				  'WBFS' = 57 42 46 53
number of sectors:   279303400
hd sector size:			  9 ->		512
WBFS sector size:		   26 ->   67108864

hd:   sector size:		   512 = 2^9
hd:   num of sectors: 3906250000
hd:	total size:	   1907349 MiB

wii:  sector size:		 32768 = 2^15
wii:  num of sectors:   61035008
wii:  sectors/disc:	   286864
wii:   total size:	   1907344 MiB

wbfs: sector size:	  67108864 = 2^26
wbfs: num of sectors:	  29802
wbfs: sectors/disc:		  140
wbfs:  total size:	   1907328 MiB

partition lba:				 0
free blocks lba:		  131064
disc info size:			 1024

used disk space:			 640 MiB =   1%
free disk space:		 1906688 MiB =  99%
total disk space:		1907328 MiB = 100%

number of wii discs:		   0	 =   0%
max disc:					500
disc open:					 0
    With sector size 512 the maximal number of discs is set to 500 (=512-12). But when a WBFS is formatted with sector size 2048 than it may contain up to 2036 (2048-12) discs. See the dump:

    Code:
    DUMP of a.wbfs

WBFS-Header:
MAGIC:				  'WBFS' = 57 42 46 53
number of sectors:  1143551290
hd sector size:			 11 ->	   2048
WBFS sector size:		   25 ->   33554432

hd:   sector size:		  2048 = 2^11
hd:   num of sectors:  976562500
hd:	total size:	   1907349 MiB

wii:  sector size:		 32768 = 2^15
wii:  num of sectors:   61034496
wii:  sectors/disc:	   286864
wii:   total size:	   1907328 MiB

wbfs: sector size:	  33554432 = 2^25
wbfs: num of sectors:	  59604
wbfs: sectors/disc:		  280
wbfs:  total size:	   1907328 MiB

partition lba:				 0
free blocks lba:		   16380
disc info size:			 2048

used disk space:			 640 MiB =   1%
free disk space:		 1906688 MiB =  99%
total disk space:		1907328 MiB = 100%

number of wii discs:		   0	 =   0%
max disc:				   2036
disc open:					 0
    I don't know if other tools and the usb loaders support other sector sizes (this is subject of later tests, perhaps others will do that) but the solution is very easy: Read the WBFS sectors before calling the function wbfs_open_partition() and calculate the sector size:

    CODEwbfs_head_t whead;
    stat = ReadAt(file,0,&whead,sizeof(whead));
    if (stat) { ERROR_HANDLING; }
    sector_size = 1
     
  2. Skizzo

    Skizzo Banned
    Banned
    Level 0

    Joined:
    May 1, 2009
    Messages:
    475
    Country:
    United States
  3. nubecoder

    nubecoder Advanced Member
    Newcomer
    Level 1

    Joined:
    May 20, 2009
    Messages:
    82
    Country:
    United States
    This is exactly why all of this is so confusing..

    HD Sector Size:
    All hard disk drives have a specific sector size, it cannot be modified. For most (all?) drives it is 512.

    WBFS Sector Size:
    All WBFS Formatted drives (partitions) have a dynamically set WBFS sector size. This seems to be calculated by making sure that the WBFS sector count will fit into 16 bits.

    What Wiimm is talking about is telling the libWBFS code that the hard drives sector size is larger than it actually is.

    It seems that this might be a workaround for the max game count limit.

    But there is a limitation to take note of, you can only read data from a drive in multiples of the real HD sector size.

    Also notice that the WBFS sector size is doubled in his example above (I'm not sure what effects this might have, if any).

    Interesting stuff, nonetheless.

    =]
     
  4. Wiimm

    OP Wiimm Developer
    Member
    Level 7

    Joined:
    Aug 11, 2009
    Messages:
    2,266
    Country:
    Germany
    I have used factor 4 (2048 bytes). Now I have imported and exported about 20 games with sector size 2048 without any errors. It seems that the WBFS subsystems can handle other sectors sizes well.
     
  5. Skizzo

    Skizzo Banned
    Banned
    Level 0

    Joined:
    May 1, 2009
    Messages:
    475
    Country:
    United States
    It seems the two of you have a pretty good grasp of WBFS, so I'll ask you both, would a tool to resize a WBFS partition be possible or not? Just wondering if it is at all possible...before I take the time to start poking around. [​IMG] Sorry if this strays a bit OT.
     
  6. Wiimm

    OP Wiimm Developer
    Member
    Level 7

    Joined:
    Aug 11, 2009
    Messages:
    2,266
    Country:
    Germany
    Yes, resizing is possible (and hard work)
     
  7. ChokeD

    ChokeD The Contributor
    Member
    Level 2

    Joined:
    Jul 18, 2009
    Messages:
    968
    Country:
    United States
    Finally a good read among the childish crap in other threads.

    Thanks
     
  8. LOTG

    LOTG Advanced Member
    Newcomer
    Level 1

    Joined:
    May 11, 2006
    Messages:
    51
    Country:
    Netherlands
    This is interesting.

    Is there a specification somewhere about WBFS? Or is it all trial and error with the available sourcecode?
     
  9. nIxx

    nIxx GBAtemp Advanced Maniac
    Member
    Level 3

    Joined:
    Sep 30, 2007
    Messages:
    1,544
    Country:
    Gambia, The
    There where very little infos about the specs in the header file anyway i think 500 Games are more as enough
     
  10. KirovAir

    KirovAir Dutch Alcoholic Programmer
    Member
    Level 7

    Joined:
    Dec 7, 2006
    Messages:
    689
    Country:
    Netherlands
    I second this, a documentation about the WBFS subsystem would be a nice read, and resource.
     
  11. Wiimm

    OP Wiimm Developer
    Member
    Level 7

    Joined:
    Aug 11, 2009
    Messages:
    2,266
    Country:
    Germany
    I have tested sector-size = 2048 with USB loader cfg and Waninkokos USB load 1.5:
    As expected, it does not work.
     
  12. r-win

    r-win GBAtemp Fan
    Member
    Level 1

    Joined:
    Oct 10, 2009
    Messages:
    453
    Country:
    Netherlands
    Wiimm,

    allright, you've got us (USB Loader GX) partially convinced, so we implemented this change in the loader. However, I'm not sure if games will load, since you've also got cios, which contain a libwbfs on their own. If they are not patched, is this gonna work anyway?
    Fastest way to try is using USB Loader GX rev 793+ with a changed sector size [​IMG]

    Anyway, excellent find and excellent job!
     
  13. Wiimm

    OP Wiimm Developer
    Member
    Level 7

    Joined:
    Aug 11, 2009
    Messages:
    2,266
    Country:
    Germany
    Fine!

    I have used WWT to add+extract about 50 games in 2 series (sector size 1024 and 2048) without any errors. The extracted games are identically to the sources. And this means: WWT is your friend for creating and filling WBFS partitions with other sector sizes.
     
  14. Skizzo

    Skizzo Banned
    Banned
    Level 0

    Joined:
    May 1, 2009
    Messages:
    475
    Country:
    United States
    Wiimm, if you have the time, could you explain what makes a resizing tool for WBFS such hard work?
     
  15. Wiimm

    OP Wiimm Developer
    Member
    Level 7

    Joined:
    Aug 11, 2009
    Messages:
    2,266
    Country:
    Germany
    It's hard (for me) to learn the details of wbfs. There is nearly none inline or other documentation. And before checking or resizing a wbfs partition the author must know precisely the impact of each parameter.
     
  16. Wiimm

    OP Wiimm Developer
    Member
    Level 7

    Joined:
    Aug 11, 2009
    Messages:
    2,266
    Country:
    Germany
    I have found a possible buffer overrun. Perhaps it never happens because there is no disc with more than 32 partitions but it's still a bug.

    Code:
    static void do_disc ( wiidisc_t * d )
{
ÂÂÂÂ...
ÂÂÂÂu64 partition_offset[32]; // XXX: don't know the real maximum
ÂÂÂÂu64 partition_type[32]; // XXX: don't know the real maximum
    There is space only for 32 partitions.
    u64 is not necessary because source data is only u32.

    Code:
    ÂÂÂÂ...
ÂÂÂÂn_partitions = _be32(b);
ÂÂÂÂdisc_read(d,_be32(b + 4), b, 0x100);
    missing a check of 'n_partitions'. Also only data for max 32 partitions are read.

    CODEÂÂÂÂfor (i = 0; i < n_partitions; i++)
    ÂÂÂÂ{
    ÂÂÂÂÂÂÂÂpartition_offset = _be32(b + 8 * i);
    ÂÂÂÂÂÂÂÂpartition_type = _be32(b + 8 * i+4);
    ÂÂÂÂÂÂÂÂ**BUMMM**
     
  17. Dack

    Dack GBAtemp Advanced Fan
    Member
    Level 1

    Joined:
    Aug 26, 2007
    Messages:
    603
    Country:
    If you have a disc with 4 or more partitions then the Wii will refuse to read it anyways - it's a problem I encountered when I did the multidisc.

    However......

    The standard code included in the libwfs does not handle the second partition table (or third and fourth) - this means the VC games on SSBB are not seen for example. There doesn't seem to be limits on those tables.

    You also need the u64 as for a Wii disc the stored values need to be multiplied by 4 to get actual - taking you over the 32 bit limit.
     
  18. Wiimm

    OP Wiimm Developer
    Member
    Level 7

    Joined:
    Aug 11, 2009
    Messages:
    2,266
    Country:
    Germany
    Not in this function: The data read is u32. And when the variables used the content will be copied back into u32 variables (function call). Calculations are not done in this function.

    P.S.: Anything but important.
     
  19. Dack

    Dack GBAtemp Advanced Fan
    Member
    Level 1

    Joined:
    Aug 26, 2007
    Messages:
    603
    Country:
    The problems with resizing come down to the field labeled "WBFS_SECTOR_SIZE" up above (and it's companions)

    When a partition is initialised it effectively splits the storage space down into 64k chunks - these chunks are where the data is stored in the WBFS partition and represent a cluster of data. Each chunk is the size of WBFS_SECTOR_SIZE and starts at location partition start + wbfs header + game header table + usage table + WBFS_SECTOR_SIZE * cluster_number.

    If you look at the header for a game placed on a WBFS partition you'll see the usual game name etc. and then a cluster usage table of 16 bit values. This table is created by down-converting the Wii's 32k usage map.
    i.e. if the Wii disc showed data at 0x40000 to 0x50000 and 0xf800000 to 0x10000000 then 38000000 to 390000000 the cluster table would allocate data from the disc pool to cover those areas. The size of the games cluster table is derived from the size of a Wii disc divided by the WBFS_SECTOR_SIZE. If you look at the WBFS disc data with a hex editor you'll likely see some random data after each games cluster table as the routines don't usually pad the table out to the disc sector size.

    This has the overall effect of each cluster containing WBFS_SECTOR_SIZE bytes of Wii disc data even if the original disc usage table showed only 32k as used.

    When you resize the partition, the value of WBFS_SECTOR_SIZE etc. will be altered and data will need to be moved on the disc so that the pointers inside the games are corrected.

    The limit on clusters is still 64k but each games cluster table entries will now be based around the old sized cluster/sector size and so data will be read incorrectly.

    It's a lot of tweaking of data required as you have to modify each games cluster usage table entries that are already on the disc as well as the overall usage table..

    The easiest way to resize would seem to be keep two copies of the usage table in memory. One representing the old size and the second representing the new size and copy the complete partition from one copy to the other, one game at a time. This is of course ensuring you don't overlap data blocks during the copy and you maintain disk usage tables correctly etc.

    It's a lot of work for minimal real benefit.
     
  20. arodey

    arodey GBAtemp Regular
    Member
    Level 1

    Joined:
    Aug 19, 2009
    Messages:
    208
    Country:
    United States
    This is all fascinating. (you all lost me before I started reading) But this is what this whole scene is about. Keep up the good work!
     
(You must log in or sign up to post here.)
Page 1 of 5
Draft saved Draft deleted
Loading...

Hide similar threads Similar threads with keywords - Inside,

  1. ladygugu

    [Released] Mario & Luigi: Bowser's Inside Story + Bowser Jr's Journey SaveEditor

    ladygugu, Feb 2, 2019, in forum: 3DS - Hacks, Translations and Utilities
    Replies:
    41
    Views:
    9,213
    ladygugu
    Jan 5, 2020
  2. Honkinghard

    RetroArch Wii: Garbled text when playing certain MAME games like TMNT (pics inside)

    Honkinghard, Dec 29, 2019, in forum: Wii - Emulation and Homebrew
    Replies:
    8
    Views:
    378
    Tetsuo Shima
    Dec 29, 2019
  3. PabloMK7

    SARC File Manager - A tool for managing files inside SARC archives

    PabloMK7, Aug 30, 2017, in forum: 3DS - Hacks, Translations and Utilities
    Replies:
    5
    Views:
    1,329
    MarioandSonicfan04
    Dec 15, 2019
  4. Pataxu

    Inject a custom game inside a 3DS cartridge

    Pataxu, Nov 10, 2019, in forum: 3DS - Console, Accessories and Hardware
    Replies:
    3
    Views:
    319
    SomeGamer
    Nov 10, 2019
  5. jifri

    i have a 3DS XL hacked with 100 games as a gift, how do i copy the things inside?

    jifri, Oct 26, 2019, in forum: 3DS - Homebrew Development and Emulators
    Replies:
    2
    Views:
    477
    jifri
    Oct 26, 2019