Inside WBFS

Discussion in 'Wii - Backup Loaders' started by Wiimm, Oct 13, 2009.

Oct 13, 2009

Inside WBFS by Wiimm at 6:55 PM (34,216 Views / 0 Likes) 84 replies

  1. Wiimm
    OP

    Member Wiimm Developer

    Joined:
    Aug 11, 2009
    Messages:
    2,052
    Location:
    Germany
    Country:
    Germany
    Inside WBFS
    A few truths about WBFS


    While I'm working with my WBFS tool (WWT) I take some deeper looks into the WBFS subsystem and found some interest things.

    Overall in the internet you can read that the maximal number of discs is that a WBFS partition can contain is about 500. This is true but not the whole truth. The value depends of the sector size parameter. It defines the number of bytes of 1 hd sector. The first sector of a WBFS partition is reserved for the WBFS header (12 bytes) and for a disc table (all other bytes, 1 byte per disc).

    The user/tool can set the sector size while formatting a WBFS partition. The default sector size is 512. WWT knows the option --sector-size and may format a WBFS partition with different sector sizes. This is nice for testing.

    The following dump shows the specific parameters of a WBFS partition with about 2 tera bytes (a file with exact 2.000.000.000.000 bytes). WWT creates this as sparse file so that the WBFS partition only needs 16 KiB disc space after formatting:

    Code:
    DUMP of a.wbfs
    
    WBFS-Header:
    MAGIC:				  'WBFS' = 57 42 46 53
    number of sectors:   279303400
    hd sector size:			  9 ->		512
    WBFS sector size:		   26 ->   67108864
    
    hd:   sector size:		   512 = 2^9
    hd:   num of sectors: 3906250000
    hd:	total size:	   1907349 MiB
    
    wii:  sector size:		 32768 = 2^15
    wii:  num of sectors:   61035008
    wii:  sectors/disc:	   286864
    wii:   total size:	   1907344 MiB
    
    wbfs: sector size:	  67108864 = 2^26
    wbfs: num of sectors:	  29802
    wbfs: sectors/disc:		  140
    wbfs:  total size:	   1907328 MiB
    
    partition lba:				 0
    free blocks lba:		  131064
    disc info size:			 1024
    
    used disk space:			 640 MiB =   1%
    free disk space:		 1906688 MiB =  99%
    total disk space:		1907328 MiB = 100%
    
    number of wii discs:		   0	 =   0%
    max disc:					500
    disc open:					 0
    With sector size 512 the maximal number of discs is set to 500 (=512-12). But when a WBFS is formatted with sector size 2048 than it may contain up to 2036 (2048-12) discs. See the dump:

    Code:
    DUMP of a.wbfs
    
    WBFS-Header:
    MAGIC:				  'WBFS' = 57 42 46 53
    number of sectors:  1143551290
    hd sector size:			 11 ->	   2048
    WBFS sector size:		   25 ->   33554432
    
    hd:   sector size:		  2048 = 2^11
    hd:   num of sectors:  976562500
    hd:	total size:	   1907349 MiB
    
    wii:  sector size:		 32768 = 2^15
    wii:  num of sectors:   61034496
    wii:  sectors/disc:	   286864
    wii:   total size:	   1907328 MiB
    
    wbfs: sector size:	  33554432 = 2^25
    wbfs: num of sectors:	  59604
    wbfs: sectors/disc:		  280
    wbfs:  total size:	   1907328 MiB
    
    partition lba:				 0
    free blocks lba:		   16380
    disc info size:			 2048
    
    used disk space:			 640 MiB =   1%
    free disk space:		 1906688 MiB =  99%
    total disk space:		1907328 MiB = 100%
    
    number of wii discs:		   0	 =   0%
    max disc:				   2036
    disc open:					 0
    I don't know if other tools and the usb loaders support other sector sizes (this is subject of later tests, perhaps others will do that) but the solution is very easy: Read the WBFS sectors before calling the function wbfs_open_partition() and calculate the sector size:

    CODEwbfs_head_t whead;
    stat = ReadAt(file,0,&whead,sizeof(whead));
    if (stat) { ERROR_HANDLING; }
    sector_size = 1
     


  2. Skizzo

    Banned Skizzo Banned

    Joined:
    May 1, 2009
    Messages:
    475
    Country:
    United States
  3. nubecoder

    Newcomer nubecoder Advanced Member

    Joined:
    May 20, 2009
    Messages:
    82
    Country:
    United States
    This is exactly why all of this is so confusing..

    HD Sector Size:
    All hard disk drives have a specific sector size, it cannot be modified. For most (all?) drives it is 512.

    WBFS Sector Size:
    All WBFS Formatted drives (partitions) have a dynamically set WBFS sector size. This seems to be calculated by making sure that the WBFS sector count will fit into 16 bits.

    What Wiimm is talking about is telling the libWBFS code that the hard drives sector size is larger than it actually is.

    It seems that this might be a workaround for the max game count limit.

    But there is a limitation to take note of, you can only read data from a drive in multiples of the real HD sector size.

    Also notice that the WBFS sector size is doubled in his example above (I'm not sure what effects this might have, if any).

    Interesting stuff, nonetheless.

    =]
     
  4. Wiimm
    OP

    Member Wiimm Developer

    Joined:
    Aug 11, 2009
    Messages:
    2,052
    Location:
    Germany
    Country:
    Germany
    I have used factor 4 (2048 bytes). Now I have imported and exported about 20 games with sector size 2048 without any errors. It seems that the WBFS subsystems can handle other sectors sizes well.
     
  5. Skizzo

    Banned Skizzo Banned

    Joined:
    May 1, 2009
    Messages:
    475
    Country:
    United States
    It seems the two of you have a pretty good grasp of WBFS, so I'll ask you both, would a tool to resize a WBFS partition be possible or not? Just wondering if it is at all possible...before I take the time to start poking around. [​IMG] Sorry if this strays a bit OT.
     
  6. Wiimm
    OP

    Member Wiimm Developer

    Joined:
    Aug 11, 2009
    Messages:
    2,052
    Location:
    Germany
    Country:
    Germany
    Yes, resizing is possible (and hard work)
     
  7. ChokeD

    Member ChokeD The Contributor

    Joined:
    Jul 18, 2009
    Messages:
    967
    Country:
    United States
    Finally a good read among the childish crap in other threads.

    Thanks
     
  8. LOTG

    Newcomer LOTG Advanced Member

    Joined:
    May 11, 2006
    Messages:
    51
    Location:
    Maastricht
    Country:
    Netherlands
    This is interesting.

    Is there a specification somewhere about WBFS? Or is it all trial and error with the available sourcecode?
     
  9. nIxx

    Member nIxx GBAtemp Maniac

    Joined:
    Sep 30, 2007
    Messages:
    1,459
    Location:
    Germany
    Country:
    Germany
    There where very little infos about the specs in the header file anyway i think 500 Games are more as enough
     
  10. KirovAir

    Member KirovAir Dutch Alcoholic Programmer

    Joined:
    Dec 7, 2006
    Messages:
    587
    Location:
    Netherlands
    Country:
    Netherlands
    I second this, a documentation about the WBFS subsystem would be a nice read, and resource.
     
  11. Wiimm
    OP

    Member Wiimm Developer

    Joined:
    Aug 11, 2009
    Messages:
    2,052
    Location:
    Germany
    Country:
    Germany
    I have tested sector-size = 2048 with USB loader cfg and Waninkokos USB load 1.5:
    As expected, it does not work.
     
  12. r-win

    Member r-win GBAtemp Fan

    Joined:
    Oct 10, 2009
    Messages:
    453
    Country:
    Netherlands
    Wiimm,

    allright, you've got us (USB Loader GX) partially convinced, so we implemented this change in the loader. However, I'm not sure if games will load, since you've also got cios, which contain a libwbfs on their own. If they are not patched, is this gonna work anyway?
    Fastest way to try is using USB Loader GX rev 793+ with a changed sector size [​IMG]

    Anyway, excellent find and excellent job!
     
  13. Wiimm
    OP

    Member Wiimm Developer

    Joined:
    Aug 11, 2009
    Messages:
    2,052
    Location:
    Germany
    Country:
    Germany
    Fine!

    I have used WWT to add+extract about 50 games in 2 series (sector size 1024 and 2048) without any errors. The extracted games are identically to the sources. And this means: WWT is your friend for creating and filling WBFS partitions with other sector sizes.
     
  14. Skizzo

    Banned Skizzo Banned

    Joined:
    May 1, 2009
    Messages:
    475
    Country:
    United States
    Wiimm, if you have the time, could you explain what makes a resizing tool for WBFS such hard work?
     
  15. Wiimm
    OP

    Member Wiimm Developer

    Joined:
    Aug 11, 2009
    Messages:
    2,052
    Location:
    Germany
    Country:
    Germany
    It's hard (for me) to learn the details of wbfs. There is nearly none inline or other documentation. And before checking or resizing a wbfs partition the author must know precisely the impact of each parameter.
     
  16. Wiimm
    OP

    Member Wiimm Developer

    Joined:
    Aug 11, 2009
    Messages:
    2,052
    Location:
    Germany
    Country:
    Germany
    I have found a possible buffer overrun. Perhaps it never happens because there is no disc with more than 32 partitions but it's still a bug.

    Code:
    static void do_disc ( wiidisc_t * d )
    {
    ÂÂÂÂ...
    ÂÂÂÂu64 partition_offset[32]; // XXX: don't know the real maximum
    ÂÂÂÂu64 partition_type[32]; // XXX: don't know the real maximum
    There is space only for 32 partitions.
    u64 is not necessary because source data is only u32.

    Code:
    ÂÂÂÂ...
    ÂÂÂÂn_partitions = _be32(b);
    ÂÂÂÂdisc_read(d,_be32(b + 4), b, 0x100);
    missing a check of 'n_partitions'. Also only data for max 32 partitions are read.

    CODEÂÂÂÂfor (i = 0; i < n_partitions; i++)
    ÂÂÂÂ{
    ÂÂÂÂÂÂÂÂpartition_offset = _be32(b + 8 * i);
    ÂÂÂÂÂÂÂÂpartition_type = _be32(b + 8 * i+4);
    ÂÂÂÂÂÂÂÂ**BUMMM**
     
  17. Dack

    Member Dack GBAtemp Advanced Fan

    Joined:
    Aug 26, 2007
    Messages:
    603
    Location:
    UK
    Country:
    United Kingdom
    If you have a disc with 4 or more partitions then the Wii will refuse to read it anyways - it's a problem I encountered when I did the multidisc.

    However......

    The standard code included in the libwfs does not handle the second partition table (or third and fourth) - this means the VC games on SSBB are not seen for example. There doesn't seem to be limits on those tables.

    You also need the u64 as for a Wii disc the stored values need to be multiplied by 4 to get actual - taking you over the 32 bit limit.
     
  18. Wiimm
    OP

    Member Wiimm Developer

    Joined:
    Aug 11, 2009
    Messages:
    2,052
    Location:
    Germany
    Country:
    Germany
    Not in this function: The data read is u32. And when the variables used the content will be copied back into u32 variables (function call). Calculations are not done in this function.

    P.S.: Anything but important.
     
  19. Dack

    Member Dack GBAtemp Advanced Fan

    Joined:
    Aug 26, 2007
    Messages:
    603
    Location:
    UK
    Country:
    United Kingdom
    The problems with resizing come down to the field labeled "WBFS_SECTOR_SIZE" up above (and it's companions)

    When a partition is initialised it effectively splits the storage space down into 64k chunks - these chunks are where the data is stored in the WBFS partition and represent a cluster of data. Each chunk is the size of WBFS_SECTOR_SIZE and starts at location partition start + wbfs header + game header table + usage table + WBFS_SECTOR_SIZE * cluster_number.

    If you look at the header for a game placed on a WBFS partition you'll see the usual game name etc. and then a cluster usage table of 16 bit values. This table is created by down-converting the Wii's 32k usage map.
    i.e. if the Wii disc showed data at 0x40000 to 0x50000 and 0xf800000 to 0x10000000 then 38000000 to 390000000 the cluster table would allocate data from the disc pool to cover those areas. The size of the games cluster table is derived from the size of a Wii disc divided by the WBFS_SECTOR_SIZE. If you look at the WBFS disc data with a hex editor you'll likely see some random data after each games cluster table as the routines don't usually pad the table out to the disc sector size.

    This has the overall effect of each cluster containing WBFS_SECTOR_SIZE bytes of Wii disc data even if the original disc usage table showed only 32k as used.

    When you resize the partition, the value of WBFS_SECTOR_SIZE etc. will be altered and data will need to be moved on the disc so that the pointers inside the games are corrected.

    The limit on clusters is still 64k but each games cluster table entries will now be based around the old sized cluster/sector size and so data will be read incorrectly.

    It's a lot of tweaking of data required as you have to modify each games cluster usage table entries that are already on the disc as well as the overall usage table..

    The easiest way to resize would seem to be keep two copies of the usage table in memory. One representing the old size and the second representing the new size and copy the complete partition from one copy to the other, one game at a time. This is of course ensuring you don't overlap data blocks during the copy and you maintain disk usage tables correctly etc.

    It's a lot of work for minimal real benefit.
     
  20. arodey

    Member arodey GBAtemp Regular

    Joined:
    Aug 19, 2009
    Messages:
    208
    Location:
    Cincinnati, OH
    Country:
    United States
    This is all fascinating. (you all lost me before I started reading) But this is what this whole scene is about. Keep up the good work!
     

Share This Page