Inside WBFS

Discussion in 'Wii - Backup Loaders' started by Wiimm, Oct 13, 2009.

  1. charroux

    charroux Advanced Member

    Newcomer
    69
    0
    May 23, 2009
    United States
    Amazing stuff!

    1) Is this currently compatible with any USB Loader?
    2) What is the command line to format/init to allow more than 500 games on a partition? And is that compatible with any USB Loader?
     


  2. Wiimm
    OP

    Wiimm Developer

    Member
    2,117
    332
    Aug 11, 2009
    Gambia, The
    Germany
    @charroux

    1.) No, because most (all) usb loader have the default sector size 512 hard coded.

    2.) sector size > 512 needed. See the first post for details.
     
  3. charroux

    charroux Advanced Member

    Newcomer
    69
    0
    May 23, 2009
    United States
    Thanks. 1st post didn't help directly, but I get the idea that it will require a code change in a loader before it will work for most end users like me.
     
  4. r-win

    r-win GBAtemp Fan

    Member
    453
    0
    Oct 10, 2009
    Netherlands
    You could try USB Loader GX, since we applied the code change posted by Wiimm in his first post.
     
  5. Wiimm
    OP

    Wiimm Developer

    Member
    2,117
    332
    Aug 11, 2009
    Gambia, The
    Germany
    Great! I never noticed this before. I will play with it.
     
  6. Elfish

    Elfish GBAtemp Fan

    Member
    447
    3
    Sep 20, 2004
    Gambia, The
    that's what i'M planning to do.
    sadly there's no loader that loads wdf files directly [​IMG]
     
  7. Wiimm
    OP

    Wiimm Developer

    Member
    2,117
    332
    Aug 11, 2009
    Gambia, The
    Germany
    Free Blocks Table and Bugs

    Introduction

    A WBFS is divided into blocks with equal size. The block size is a power of 2. The calculation of of the block size is made when formatting a WBFS:[*]Use the minimal block size so that there are less than 2^16 blocks.[*]Adjust the block size so that all management data fits into one block. The block size power is stored into the WBFS header (struct wbfs_head) as member 'wbfs_sec_sz_s'.Based on this calculation only whole multiple of the sector size of the WBFS file or device can be used. The rest is not used. "wwt DUMP" shows the end of the used are in the memory map.

    After formatting we have N blocks, addressed by 0..N-1. Block #0 contains the management data. All other blocks (1..N-1) are used to store the discs. A block is not used or exactly assigned to 1 disc.

    The "free block table" (FBT) is located at the end of block #0, beginning at a hd sector (parameter "sector size"). For each block there is exact one bit reserved. Is the bit set (1), the block is free (not assigned to a disc). If the bit is unset (0), the block is used by a disc.

    Each byte of the FBT contains the status of 8 blocks. The first byte is to manage the blocks 1..8, the next byte for blocks 9..15 and so on. As you see this assignments are NOT zero based! In the beginning of the WBFS there was a "delete bug": The function free_block() used a zero based calculation of the FBT and marked the wrong blocks for free (e.g. marked block 2 instead of block 1 for free). But there are some more bugs in libwbfs. And the more bugs are the reason of this article.

    I plan to publish a patch for all discussed bugs later after some tests.


    Formatting bug

    While formatting the WBFS only the first N/8 bytes (N=number of WBFS blocks) of the FBT are set with the value 0xff. Remember: A set bit means "free block".
    [*]If N is a multiple of 8 (N%8 == 0) then 1 block to much is marked as free. But because of the allocation bug (see below) this bug appears only, if N is a multiple of 32.[*]If N%8 > 1 then to less blocks are marked as free. But this means only that not the whole WBFS is used for discs.[*]If N%8 == 1 then the correct number of blocks is marked as free.
    This formatting bugs can be fixed very easy:[*]Just unset the wrong bit when opening the WBFS.[*]When formatting or checking the disc then mark all blocks free. This can not be done if there is no space for the additional needed byte in block #0.

    Allocation bug


    libwbfs uses a run time optimization and scans the FBT in u32 (32 bits) steps. Because of this optimization only the first (N/32)*4 bytes of the FBT are used. And this means that up to to 4 bytes of the FBT are ignored. And this means that up to 30 WBFS blocks will not be used for storing discs. On a WBFS with about 500 GiB the WBFS block size is 2^23 -> up to 240 MiB wasted space.

    A patch must be compatible to already existing WBFS. And this means, that only (N/8)*8 blocks can be used, because they are set correctly by the old formatting. The last upto 7 blocks should never be used.


    Free-Block bug


    The free block function free_block() does not make a check of the block number. If the block number is =0 or >=N wrong bits anywhere in the memory are set. No problem if the WBFS is valid.

    But because of other errors in the WBFS file/device it is possible that there are wrong values in the disc memory maps. I have already seen dumps with such errors.


    Add-Disc bug


    If the WBFS becomes full while adding a disc the disc will not added to the WBFS, that's ok. But the already allocated blocks are not freed.
     
  8. charroux

    charroux Advanced Member

    Newcomer
    69
    0
    May 23, 2009
    United States
    I had to give up, since Cygwin is denying me permission to wwt, and I have no idea how to proceed. Thanks for the info though.
     
  9. Wiimm
    OP

    Wiimm Developer

    Member
    2,117
    332
    Aug 11, 2009
    Gambia, The
    Germany
    I found a new small libwbfs bug:

    In function wbfs_open_partition() the number of wii sectors is calculated by
    Code:
    ÂÂÂÂp->n_wii_sec = ( p->n_hd_sec / p->wii_sec_sz ) * p->hd_sec_sz;
    ÂÂÂÂ// -> num_of_hd_sectors / wii_sector_size * hd_sector_size
    If num_of_hd_sectors is not a multiple of wii_sector_size then the calulated result is to small because of rounding down.

    This code looks better (multiply first):
    Code:
    ÂÂÂÂp->n_wii_sec = (u64)p->n_hd_sec * p->hd_sec_sz / p->wii_sec_sz;
    To avoid a 64 bit calculation use this (it works until hd_sector_size is > wii_sec_sz because both are power of 2):
    Code:
    ÂÂÂÂp->n_wii_sec = p->n_hd_sec / ( p->wii_sec_sz / p->hd_sec_sz );
     
  10. Elfish

    Elfish GBAtemp Fan

    Member
    447
    3
    Sep 20, 2004
    Gambia, The
    hmmm interessting. seems like there are tons of bugs :/
     
  11. Wiimm
    OP

    Wiimm Developer

    Member
    2,117
    332
    Aug 11, 2009
    Gambia, The
    Germany
    Yes, I playing with WBFS since only 6 weeks and I have found a dozend bugs, most harmless but some harmfull. I have fixed all of them in my WWT.

    My conclusions:[*] Do not trust WBFS manager or USB loaders that haven't fixed these errors.
    .
    [*] Do not trust me and my bug fixes because nobody has controlled me.
    .
    [*] Do not trust WWT because 2.) and because there are perhaps other bugs.
    .
    [*] Do not use a WBFS for backup, NEVER!
     
  12. mousex

    mousex GBAtemp Advanced Fan

    Member
    987
    0
    Jan 23, 2009
    United States
    This irony, don't use the Wii Backup File System for Backups [​IMG]
    If you wouldn't care for compatibilty you could fix these errors, right?
    What about a WBFS2 from you, tested until hell, developed in months and not some days, with real documentation and you send it to the USBLoader devs?
     
  13. sotra

    sotra Newbie

    Newcomer
    9
    0
    Oct 9, 2009
    United States
    Hi,

    I had an old 250GB Western Digital (model # WD2500B007 - caviar with neon blue and pink lights on the sides) that I used for Wii backup games exclusively, but then I got a 1TB Fantom Drives GreenDrive (model # GDP1000EU) that I chose to partition using EASEUS partition master (25GB FAT32, 25GB NTFS and the rest WBFS). No problems with either of these drives really, but one thing I noticed was that the sizes of the games shows up bigger in most cases on my new HDD.
    Some examples: Wii Sports (324MB old drive vs. 512MB on new drive), Wii Fit Plus (1240Mb vs. 1472MB), The Beatles: Rock Band (3524MB vs. 3568MB), Wii Chess (32MB vs. 80MB) etc.

    I've tried hooking the new HDD up to my Wii and loaded the games from my original discs through various USB loaders, used WBFS manager and WBFS intelligent GUI on my PC and copied from my old HDD directly to my new HDD using ncWBFSTool. Every single game came out bigger on my new HDD.
    Any ideas why? I'd hate to take them all off my new HDD and reformat it a certain way to get them to come up smaller in size.
    Thanks in advance for any replies.

    Edit: If there's any sort of info about each HDD you want me to post then let me know through which program and how and I will do it ASAP. Thanks!
     
  14. Wiimm
    OP

    Wiimm Developer

    Member
    2,117
    332
    Aug 11, 2009
    Gambia, The
    Germany
    I do only fixes that are compatible to WBFS. The idea of WBFS is good but it seems that nobody had made a code view.
     
  15. Wiimm
    OP

    Wiimm Developer

    Member
    2,117
    332
    Aug 11, 2009
    Gambia, The
    Germany
    The function alloc_block() have a serious bug: Depending of the value of 'n_wbfs_sec' a non existend block will be allocated. This patch fixes the bug:
    CODE--- libwbfs-old.cÂÂÂÂÂÂ 2009-11-14 13:12:58.000000000 +0100
    +++ libwbfs.cÂÂ 2009-11-14 13:14:12.000000000 +0100
    @@ -450,7 +450,8 @@
    ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ if (v & (1
     
  16. Wiimm
    OP

    Wiimm Developer

    Member
    2,117
    332
    Aug 11, 2009
    Gambia, The
    Germany
    There is another bug in libwbfs:

    If the WBFS becomes full while adding a game the WBFS becomes an invalid state:[*] The disc slot is marked as used, but the info header was not written and may contain illegal data of a previous removed disc or because there is any data on the harddisk.

    Deleting this disc may free blocks of other discs because of the undefined data. Because of the free_block() bug, any data in the main memory can be modified == undefined behavior of the program.
    .[*] The already alloced blocks are not freed.This is the original code:
    Code:
    bl = alloc_block(p);
    if (bl == 0xffff)
    {
    ÂÂÂÂERROR("no space left on device (disc full)");
    }
    And this is my version. It's implemented in wwt since v.0.16a:
    CODEbl = alloc_block(p);
    if ( bl == WBFS_NO_BLOCK )
    {
    ÂÂÂÂ// free disc slot
    ÂÂÂÂp->head->disc_table[discn] = 0;

    ÂÂÂÂ// free already allocated blocks
    ÂÂÂÂint j;
    ÂÂÂÂfor ( j = 0; j < i; j++ )
    ÂÂÂÂ{
    ÂÂÂÂÂÂÂÂbl = wbfs_ntohs(info->wlba_table[j]);
    ÂÂÂÂÂÂÂÂif (bl)
    ÂÂÂÂÂÂÂÂÂÂÂÂfree_block(p,bl);
    ÂÂÂÂ}
    ÂÂÂÂwbfs_sync(p);
    ÂÂÂÂWBFS_ERROR("no space left on device (disc full)");
    }
     
  17. mousex

    mousex GBAtemp Advanced Fan

    Member
    987
    0
    Jan 23, 2009
    United States
    So, with all these WBFS bugs. DO you think it is safer to store games in .wbfs files on a FAT partition and not on a WBFS drive? I guess if only one game is in a WBFS container most of the bugs can't happen, right?
     
  18. Wiimm
    OP

    Wiimm Developer

    Member
    2,117
    332
    Aug 11, 2009
    Gambia, The
    Germany
    yes
    or
    use a good wbfs manager
    (no idea if there is anyone, haha)
     
  19. Elfish

    Elfish GBAtemp Fan

    Member
    447
    3
    Sep 20, 2004
    Gambia, The
    i guess your tool is good huh?

    Posts merged

    i guess your tool is good huh?
     
  20. mousex

    mousex GBAtemp Advanced Fan

    Member
    987
    0
    Jan 23, 2009
    United States
    It's the best. I don't know if it is already good (in the sense of bug free, sure it is good [​IMG]) due to so many bugs in libwbfs which are maybe not yet found but it's the best if you want a sane WBFS.
    I had 4 corrupt games because I deleted MP3 and installed the same game after this and 3 games later (in USBLoader 1.2). I thought MP3 would install at the same place as before but it caused three other games to use blocks at the same places (wwt deleted them during repair as I have the original discs).