Inside WBFS

Discussion in 'Wii - Backup Loaders' started by Wiimm, Oct 13, 2009.

  1. Wiimm
    OP

    Wiimm Developer

    Member
    2,150
    364
    Aug 11, 2009
    Gambia, The
    Germany
    Inside WBFS
    A few truths about WBFS


    While I'm working with my WBFS tool (WWT) I take some deeper looks into the WBFS subsystem and found some interest things.

    Overall in the internet you can read that the maximal number of discs is that a WBFS partition can contain is about 500. This is true but not the whole truth. The value depends of the sector size parameter. It defines the number of bytes of 1 hd sector. The first sector of a WBFS partition is reserved for the WBFS header (12 bytes) and for a disc table (all other bytes, 1 byte per disc).

    The user/tool can set the sector size while formatting a WBFS partition. The default sector size is 512. WWT knows the option --sector-size and may format a WBFS partition with different sector sizes. This is nice for testing.

    The following dump shows the specific parameters of a WBFS partition with about 2 tera bytes (a file with exact 2.000.000.000.000 bytes). WWT creates this as sparse file so that the WBFS partition only needs 16 KiB disc space after formatting:

    Code:
    DUMP of a.wbfs
    
    WBFS-Header:
    MAGIC:				  'WBFS' = 57 42 46 53
    number of sectors:   279303400
    hd sector size:			  9 ->		512
    WBFS sector size:		   26 ->   67108864
    
    hd:   sector size:		   512 = 2^9
    hd:   num of sectors: 3906250000
    hd:	total size:	   1907349 MiB
    
    wii:  sector size:		 32768 = 2^15
    wii:  num of sectors:   61035008
    wii:  sectors/disc:	   286864
    wii:   total size:	   1907344 MiB
    
    wbfs: sector size:	  67108864 = 2^26
    wbfs: num of sectors:	  29802
    wbfs: sectors/disc:		  140
    wbfs:  total size:	   1907328 MiB
    
    partition lba:				 0
    free blocks lba:		  131064
    disc info size:			 1024
    
    used disk space:			 640 MiB =   1%
    free disk space:		 1906688 MiB =  99%
    total disk space:		1907328 MiB = 100%
    
    number of wii discs:		   0	 =   0%
    max disc:					500
    disc open:					 0
    With sector size 512 the maximal number of discs is set to 500 (=512-12). But when a WBFS is formatted with sector size 2048 than it may contain up to 2036 (2048-12) discs. See the dump:

    Code:
    DUMP of a.wbfs
    
    WBFS-Header:
    MAGIC:				  'WBFS' = 57 42 46 53
    number of sectors:  1143551290
    hd sector size:			 11 ->	   2048
    WBFS sector size:		   25 ->   33554432
    
    hd:   sector size:		  2048 = 2^11
    hd:   num of sectors:  976562500
    hd:	total size:	   1907349 MiB
    
    wii:  sector size:		 32768 = 2^15
    wii:  num of sectors:   61034496
    wii:  sectors/disc:	   286864
    wii:   total size:	   1907328 MiB
    
    wbfs: sector size:	  33554432 = 2^25
    wbfs: num of sectors:	  59604
    wbfs: sectors/disc:		  280
    wbfs:  total size:	   1907328 MiB
    
    partition lba:				 0
    free blocks lba:		   16380
    disc info size:			 2048
    
    used disk space:			 640 MiB =   1%
    free disk space:		 1906688 MiB =  99%
    total disk space:		1907328 MiB = 100%
    
    number of wii discs:		   0	 =   0%
    max disc:				   2036
    disc open:					 0
    I don't know if other tools and the usb loaders support other sector sizes (this is subject of later tests, perhaps others will do that) but the solution is very easy: Read the WBFS sectors before calling the function wbfs_open_partition() and calculate the sector size:

    CODEwbfs_head_t whead;
    stat = ReadAt(file,0,&whead,sizeof(whead));
    if (stat) { ERROR_HANDLING; }
    sector_size = 1
     


  2. Skizzo

    Skizzo Banned

    Banned
    475
    0
    May 1, 2009
    United States
  3. nubecoder

    nubecoder Advanced Member

    Newcomer
    82
    0
    May 20, 2009
    United States
    This is exactly why all of this is so confusing..

    HD Sector Size:
    All hard disk drives have a specific sector size, it cannot be modified. For most (all?) drives it is 512.

    WBFS Sector Size:
    All WBFS Formatted drives (partitions) have a dynamically set WBFS sector size. This seems to be calculated by making sure that the WBFS sector count will fit into 16 bits.

    What Wiimm is talking about is telling the libWBFS code that the hard drives sector size is larger than it actually is.

    It seems that this might be a workaround for the max game count limit.

    But there is a limitation to take note of, you can only read data from a drive in multiples of the real HD sector size.

    Also notice that the WBFS sector size is doubled in his example above (I'm not sure what effects this might have, if any).

    Interesting stuff, nonetheless.

    =]
     
  4. Wiimm
    OP

    Wiimm Developer

    Member
    2,150
    364
    Aug 11, 2009
    Gambia, The
    Germany
    I have used factor 4 (2048 bytes). Now I have imported and exported about 20 games with sector size 2048 without any errors. It seems that the WBFS subsystems can handle other sectors sizes well.
     
  5. Skizzo

    Skizzo Banned

    Banned
    475
    0
    May 1, 2009
    United States
    It seems the two of you have a pretty good grasp of WBFS, so I'll ask you both, would a tool to resize a WBFS partition be possible or not? Just wondering if it is at all possible...before I take the time to start poking around. [​IMG] Sorry if this strays a bit OT.
     
  6. Wiimm
    OP

    Wiimm Developer

    Member
    2,150
    364
    Aug 11, 2009
    Gambia, The
    Germany
    Yes, resizing is possible (and hard work)
     
  7. ChokeD

    ChokeD The Contributor

    Member
    968
    5
    Jul 18, 2009
    United States
    Finally a good read among the childish crap in other threads.

    Thanks
     
  8. LOTG

    LOTG Advanced Member

    Newcomer
    51
    0
    May 11, 2006
    Netherlands
    Maastricht
    This is interesting.

    Is there a specification somewhere about WBFS? Or is it all trial and error with the available sourcecode?
     
  9. nIxx

    nIxx GBAtemp Advanced Maniac

    Member
    1,528
    78
    Sep 30, 2007
    Gambia, The
    Germany
    There where very little infos about the specs in the header file anyway i think 500 Games are more as enough
     
  10. KirovAir

    KirovAir Dutch Alcoholic Programmer

    Member
    587
    24
    Dec 7, 2006
    Netherlands
    Netherlands
    I second this, a documentation about the WBFS subsystem would be a nice read, and resource.
     
  11. Wiimm
    OP

    Wiimm Developer

    Member
    2,150
    364
    Aug 11, 2009
    Gambia, The
    Germany
    I have tested sector-size = 2048 with USB loader cfg and Waninkokos USB load 1.5:
    As expected, it does not work.
     
  12. r-win

    r-win GBAtemp Fan

    Member
    453
    0
    Oct 10, 2009
    Netherlands
    Wiimm,

    allright, you've got us (USB Loader GX) partially convinced, so we implemented this change in the loader. However, I'm not sure if games will load, since you've also got cios, which contain a libwbfs on their own. If they are not patched, is this gonna work anyway?
    Fastest way to try is using USB Loader GX rev 793+ with a changed sector size [​IMG]

    Anyway, excellent find and excellent job!
     
  13. Wiimm
    OP

    Wiimm Developer

    Member
    2,150
    364
    Aug 11, 2009
    Gambia, The
    Germany
    Fine!

    I have used WWT to add+extract about 50 games in 2 series (sector size 1024 and 2048) without any errors. The extracted games are identically to the sources. And this means: WWT is your friend for creating and filling WBFS partitions with other sector sizes.
     
  14. Skizzo

    Skizzo Banned

    Banned
    475
    0
    May 1, 2009
    United States
    Wiimm, if you have the time, could you explain what makes a resizing tool for WBFS such hard work?
     
  15. Wiimm
    OP

    Wiimm Developer

    Member
    2,150
    364
    Aug 11, 2009
    Gambia, The
    Germany
    It's hard (for me) to learn the details of wbfs. There is nearly none inline or other documentation. And before checking or resizing a wbfs partition the author must know precisely the impact of each parameter.
     
  16. Wiimm
    OP

    Wiimm Developer

    Member
    2,150
    364
    Aug 11, 2009
    Gambia, The
    Germany
    I have found a possible buffer overrun. Perhaps it never happens because there is no disc with more than 32 partitions but it's still a bug.

    Code:
    static void do_disc ( wiidisc_t * d )
    {
    ÂÂÂÂ...
    ÂÂÂÂu64 partition_offset[32]; // XXX: don't know the real maximum
    ÂÂÂÂu64 partition_type[32]; // XXX: don't know the real maximum
    There is space only for 32 partitions.
    u64 is not necessary because source data is only u32.

    Code:
    ÂÂÂÂ...
    ÂÂÂÂn_partitions = _be32(b);
    ÂÂÂÂdisc_read(d,_be32(b + 4), b, 0x100);
    missing a check of 'n_partitions'. Also only data for max 32 partitions are read.

    CODEÂÂÂÂfor (i = 0; i < n_partitions; i++)
    ÂÂÂÂ{
    ÂÂÂÂÂÂÂÂpartition_offset = _be32(b + 8 * i);
    ÂÂÂÂÂÂÂÂpartition_type = _be32(b + 8 * i+4);
    ÂÂÂÂÂÂÂÂ**BUMMM**
     
  17. Dack

    Dack GBAtemp Advanced Fan

    Member
    603
    0
    Aug 26, 2007
    UK
    If you have a disc with 4 or more partitions then the Wii will refuse to read it anyways - it's a problem I encountered when I did the multidisc.

    However......

    The standard code included in the libwfs does not handle the second partition table (or third and fourth) - this means the VC games on SSBB are not seen for example. There doesn't seem to be limits on those tables.

    You also need the u64 as for a Wii disc the stored values need to be multiplied by 4 to get actual - taking you over the 32 bit limit.
     
  18. Wiimm
    OP

    Wiimm Developer

    Member
    2,150
    364
    Aug 11, 2009
    Gambia, The
    Germany
    Not in this function: The data read is u32. And when the variables used the content will be copied back into u32 variables (function call). Calculations are not done in this function.

    P.S.: Anything but important.
     
  19. Dack

    Dack GBAtemp Advanced Fan

    Member
    603
    0
    Aug 26, 2007
    UK
    The problems with resizing come down to the field labeled "WBFS_SECTOR_SIZE" up above (and it's companions)

    When a partition is initialised it effectively splits the storage space down into 64k chunks - these chunks are where the data is stored in the WBFS partition and represent a cluster of data. Each chunk is the size of WBFS_SECTOR_SIZE and starts at location partition start + wbfs header + game header table + usage table + WBFS_SECTOR_SIZE * cluster_number.

    If you look at the header for a game placed on a WBFS partition you'll see the usual game name etc. and then a cluster usage table of 16 bit values. This table is created by down-converting the Wii's 32k usage map.
    i.e. if the Wii disc showed data at 0x40000 to 0x50000 and 0xf800000 to 0x10000000 then 38000000 to 390000000 the cluster table would allocate data from the disc pool to cover those areas. The size of the games cluster table is derived from the size of a Wii disc divided by the WBFS_SECTOR_SIZE. If you look at the WBFS disc data with a hex editor you'll likely see some random data after each games cluster table as the routines don't usually pad the table out to the disc sector size.

    This has the overall effect of each cluster containing WBFS_SECTOR_SIZE bytes of Wii disc data even if the original disc usage table showed only 32k as used.

    When you resize the partition, the value of WBFS_SECTOR_SIZE etc. will be altered and data will need to be moved on the disc so that the pointers inside the games are corrected.

    The limit on clusters is still 64k but each games cluster table entries will now be based around the old sized cluster/sector size and so data will be read incorrectly.

    It's a lot of tweaking of data required as you have to modify each games cluster usage table entries that are already on the disc as well as the overall usage table..

    The easiest way to resize would seem to be keep two copies of the usage table in memory. One representing the old size and the second representing the new size and copy the complete partition from one copy to the other, one game at a time. This is of course ensuring you don't overlap data blocks during the copy and you maintain disk usage tables correctly etc.

    It's a lot of work for minimal real benefit.
     
  20. arodey

    arodey GBAtemp Regular

    Member
    208
    0
    Aug 19, 2009
    United States
    Cincinnati, OH
    This is all fascinating. (you all lost me before I started reading) But this is what this whole scene is about. Keep up the good work!