Inside WBFS

Discussion in 'Wii - Backup Loaders' started by Wiimm, Oct 13, 2009.

  1. Wiimm
    OP

    Wiimm Developer

    Member
    2,129
    336
    Aug 11, 2009
    Gambia, The
    Germany
    If you read this enum ...
    Code:
    typedef enum
    {
    ÂÂÂÂUPDATE_PARTITION_TYPEÂÂÂÂÂÂÂÂ= 0,
    ÂÂÂÂGAME_PARTITION_TYPE,
    ÂÂÂÂOTHER_PARTITION_TYPE,
    
    ÂÂÂÂ// value in between selects partition types of that value
    ÂÂÂÂALL_PARTITIONSÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ= 0xffffffff-3,
    ÂÂÂÂREMOVE_UPDATE_PARTITION,ÂÂÂÂÂÂÂÂ// keeps game + channel installers
    ÂÂÂÂONLY_GAME_PARTITION,
    
    } partition_selector_t;
    ... and this function ....
    Code:
    static int test_partition_skip ( u32 partition_type, partition_selector_t part_sel )
    {
    ÂÂÂÂswitch (part_sel)
    ÂÂÂÂ{
    ÂÂÂÂÂÂÂÂcase ALL_PARTITIONS:
    ÂÂÂÂÂÂÂÂÂÂÂÂreturn 0;
    ÂÂÂÂÂÂÂÂcase REMOVE_UPDATE_PARTITION:
    ÂÂÂÂÂÂÂÂÂÂÂÂreturn (partition_type==1);
    ÂÂÂÂÂÂÂÂcase ONLY_GAME_PARTITION:
    ÂÂÂÂÂÂÂÂÂÂÂÂreturn (partition_type!=0);
    ÂÂÂÂÂÂÂÂdefault:
    ÂÂÂÂÂÂÂÂÂÂÂÂreturn (partition_type!=part_sel);
    ÂÂÂÂ}
    }
    ... you see that the values of the constants 'UPDATE_PARTITION_TYPE' and 'GAME_PARTITION_TYPE' have been confused.

    In detail:[*]test_partition_skip() returns 1 if the partition should be skipped (not selected).[*]'ONLY_GAME_PARTITION' checks partition type '0' but 'UPDATE_PARTITION_TYPE' is defined as '0'.[*]'REMOVE_UPDATE_PARTITION' checks partition type '1' but GAME_PARTITION_TYPE is defined as '1'.It seems that the correct definition is:
    Code:
    typedef enum
    {
    ÂÂÂÂGAME_PARTITION_TYPEÂÂ = 0,
    ÂÂÂÂUPDATE_PARTITION_TYPE = 1,
    ÂÂÂÂ...
    Moreover there is no need for ONLY_GAME_PARTITION because (the corrected) GAME_PARTITION_TYPE does the same.


    P.S.: I have already renamed test_parition_skip() to test_partition_skip().
     


  2. Wiimm
    OP

    Wiimm Developer

    Member
    2,129
    336
    Aug 11, 2009
    Gambia, The
    Germany
    @dack:
    I fully agree.

    Also, I've discovered some oddities in libwbfs. And that means that I have lost my last confidence in the stability of WBFS. I recommend to use a better tested file system as backup medium. And with an external backup medium resizing is no longer a problem -> just build the WBFS from scratch.
     
  3. Wiimm
    OP

    Wiimm Developer

    Member
    2,129
    336
    Aug 11, 2009
    Gambia, The
    Germany
    The size calculation of 'freeblocks' is wrong!

    1.) This macro expands the size to the next full hd sector size. That's ok:
    Code:
    #define ALIGN_LBA(x) (((x)+p->hd_sec_sz-1)&(~(p->hd_sec_sz-1)))
    2.) And this is the calculation of the free blocks size
    Code:
    ALIGN_LBA(p->n_wbfs_sec/8) >> p->hd_sec_sz_s
    'n_wbfs_sec' is *not* always a multiple of 8. If n_wbfs_sec is for example 4100 than 'n_wbfs_sec/8' is exactily 512.5. And this means that 2 sectors are needed. But the integer calulation of 'n_wbfs_sec/8' is only 512 (exact one logical block).

    Assuming a sector size of 512 this error appears for n_wbfs_sec that are multiple of 4096 + (1..7). For other values the LBA aligning is a kind of bug fix.

    But it could be that these errors are never noticeable, because the other functions such as block_used() and alloc_block() have similar calculation errors and therefore the missing bits are never accessed.
     
  4. Wiimm
    OP

    Wiimm Developer

    Member
    2,129
    336
    Aug 11, 2009
    Gambia, The
    Germany
    It's not really "Inside WBFS" but I won't open a new thread.

    While I'm implementing an iso dump function into 'wit' (Wiimms ISO Tool) I dumped SSBB too with curious results. Wiibrew tells me, that there are 4 partition tables and I found 13 partitions in the second partition tables. But they seems invalid because the data is stupid.

    After that I modified CFG loader v44 and enable the 1:1 flag so that the whole disc is dumped (the new dump is larger than the old one). And the info dump of this disc dump is:

    Code:
    Dump of file ssbb-full.wdf
    
    ÂÂ...
    ÂÂISO file size:ÂÂ230000000/hex = 9395240960 = 8960 MiB
    
    ÂÂ...
    
    ÂÂ2 partition tables with 15 partitions:
    
    ÂÂÂÂ tab.idxÂÂ n(part)ÂÂÂÂÂÂ offset(part.tab) .. end(p.tab)
    ÂÂÂÂ--------------------------------------------------------
    ÂÂÂÂÂÂÂÂ0ÂÂÂÂÂÂÂÂ2ÂÂÂÂÂÂ 10008*4 =ÂÂÂÂÂÂ40020 ..ÂÂÂÂÂÂ40030
    ÂÂÂÂÂÂÂÂ1ÂÂÂÂÂÂ 13ÂÂÂÂÂÂ 10010*4 =ÂÂÂÂÂÂ40040 ..ÂÂÂÂÂÂ400a8
    
    ÂÂ15 partitions:
    
    ÂÂÂÂ indexÂÂÂÂÂÂtypeÂÂÂÂÂÂoffset .. end offsetÂÂ size/hex =ÂÂ size/dec =ÂÂMiB
    ÂÂÂÂ--------------------------------------------------------------------------
    ÂÂÂÂÂÂ0.0ÂÂ UPDATE 1ÂÂÂÂÂÂ 50000 ..ÂÂÂÂaea8000ÂÂÂÂae58000 =ÂÂ182812672 =ÂÂ174
    ÂÂÂÂÂÂ0.1ÂÂÂÂ GAME 0ÂÂÂÂ f800000 ..ÂÂ1da550000ÂÂ1cad50000 = 7697924096 = 7341
    ÂÂÂÂÂÂ1.0ÂÂ 48413850ÂÂ 1da550000ÂÂÂÂÂÂÂÂ ** INVALID PARTITION **
    ÂÂÂÂÂÂ1.1ÂÂ 48413950ÂÂ 1daff0000ÂÂÂÂÂÂÂÂ ** INVALID PARTITION **
    ÂÂÂÂÂÂ1.2ÂÂ 48424150ÂÂ 1dbaa0000ÂÂÂÂÂÂÂÂ ** INVALID PARTITION **
    ÂÂÂÂÂÂ1.3ÂÂ 48424246ÂÂ 1dc570000ÂÂÂÂÂÂÂÂ ** INVALID PARTITION **
    ÂÂÂÂÂÂ1.4ÂÂ 48424250ÂÂ 1dd0e0000ÂÂÂÂÂÂÂÂ ** INVALID PARTITION **
    ÂÂÂÂÂÂ1.5ÂÂ 48424350ÂÂ 1ddc50000ÂÂÂÂÂÂÂÂ ** INVALID PARTITION **
    ÂÂÂÂÂÂ1.6ÂÂ 48424450ÂÂ 1de720000ÂÂÂÂÂÂÂÂ ** INVALID PARTITION **
    ÂÂÂÂÂÂ1.7ÂÂ 48424550ÂÂ 1df1d0000ÂÂÂÂÂÂÂÂ ** INVALID PARTITION **
    ÂÂÂÂÂÂ1.8ÂÂ 48424650ÂÂ 1dfcc0000ÂÂÂÂÂÂÂÂ ** INVALID PARTITION **
    ÂÂÂÂÂÂ1.9ÂÂ 48424750ÂÂ 1e08b0000ÂÂÂÂÂÂÂÂ ** INVALID PARTITION **
    ÂÂÂÂÂÂ1.10ÂÂ48424950ÂÂ 1e1490000ÂÂÂÂÂÂÂÂ ** INVALID PARTITION **
    ÂÂÂÂÂÂ1.11ÂÂ48424b50ÂÂ 1e2490000ÂÂÂÂÂÂÂÂ ** INVALID PARTITION **
    ÂÂÂÂÂÂ1.12ÂÂ48424c50ÂÂ 1e5d40000ÂÂÂÂÂÂÂÂ ** INVALID PARTITION **
    
    ...
    What does this mean?[*] Wiibrew is wrong[*] The disc dump is wrong[*] My tool is wrong (I don't believe that)[*] Are this the reasons for the problems while playing the game with an USB loader?
     
  5. Dr. Clipper

    Dr. Clipper GBAtemp Addict

    Member
    2,487
    0
    Aug 28, 2007
    By 4, are you referring to the fact it can't play the Masterpieces (i.e., the time-limited VC games)? Given the number of them, there could very well be one game per extra partition. What might help prove this is testing the Japanese version as it has two Masterpieces (Fire Emblem and Earthbound, IIRC) that aren't in the US/PAL versions.
     
  6. Wiimm
    OP

    Wiimm Developer

    Member
    2,129
    336
    Aug 11, 2009
    Gambia, The
    Germany
    The new WWT is online. Just say "wit dump iso_file..." to get dumps of that iso files. perhaps anyone can do that for me.
     
  7. noobwarrior7

    noobwarrior7 GBAtemp Advanced Maniac

    Member
    1,594
    1
    Aug 2, 2008
    United States
    USA
    I may not be asking this right due to lack of technical knowledge, but does the layer break, new partition table, and new partition all begin at the same area of the dual-layer discs?
     
  8. Blue-K

    Blue-K No right of appeal.

    Member
    2,572
    1
    Jun 21, 2008
    Swaziland
    Helvetica
    Layer Break is for every Wii Game the same (like with the XBox360 games). Dunno about the rest...

    @Wiimm: Brawl doesn't work with cIOS r14 because the Dual Layer Support is broken in this cIOS. That's the official information. Proof for that: Brawl works with the cIOSes from Hermes and some cIOS before r14. So it's the cIOS, not more.
     
  9. Wiimm
    OP

    Wiimm Developer

    Member
    2,129
    336
    Aug 11, 2009
    Gambia, The
    Germany
    The second partition table resists at address 40040..400a8 and thats far away from the layer break. And the partition types and partition addresses looks curios and not like all other partitions.
     
  10. Dr. Clipper

    Dr. Clipper GBAtemp Addict

    Member
    2,487
    0
    Aug 28, 2007
    Wiimm, when you were testing the 1:1 option, were you using IOS249 rev 14 or something else? That particular IOS does have trouble with ripping both MPT and SSBB, so you should try the test again while using IOS222 or a different revision of IOS249 to see if that does change anything.
     
  11. Wiimm
    OP

    Wiimm Developer

    Member
    2,129
    336
    Aug 11, 2009
    Gambia, The
    Germany
    There is a function in libwbfs called wbfs_trim(). It is used by several programs (my wwt, oggzees wbfs_file, usb loaders) to trim a wbfs (=truncate the wbfs to only necessaries blocks).
    I have found 3 (three!) bugs in it. Here is the original source:

    CODEu32 wbfs_trim(wbfs_t*p)
    {
    ÂÂÂÂu32 maxbl;
    ÂÂÂÂload_freeblocks(p);
    ÂÂÂÂmaxbl = alloc_block(p);
    ÂÂÂÂp->n_hd_sec = maxblhd_sec_sz_s);
    ÂÂÂÂp->head->n_hd_sec = wbfs_htonl(p->n_hd_sec);
    ÂÂÂÂ// make all block full
    ÂÂÂÂmemset(p->freeblks,0,p->n_wbfs_sec/8);
    ÂÂÂÂwbfs_sync(p);
    ÂÂÂÂ// os layer will truncate the file.
    ÂÂÂÂreturn maxbl;
    }

    BUGS:[*] To determine the last block the function alloc_block() is used. If there are already delete blocks alloc_blocks() returns NOT the last block.[*] If there are not any free blocks alloc_block() return ~0. Then the WBFS is not shrinked but enlarged to about 2 tera bytes (assuming sector size 512).[*] The calculation of 'p->n_hd_sec' is wrong. The value '1' for the wbfs header must be added. Without +1 the files becomes 1 sector (usual 512 bytes) short.My solution: 1.) implement the new function find_last_used_block() and 2.) modify wbfs_trim():

    CODEstatic u32 find_last_used_block ( wbfs_t * p )
    {
    ÂÂÂÂint i;
    ÂÂÂÂ// the division is a rounding error!
    ÂÂÂÂfor ( i = p->n_wbfs_sec/(8*4) - 1; i >= 0; i-- )
    ÂÂÂÂ{
    ÂÂÂÂÂÂÂÂu32 v = wbfs_ntohl(p->freeblks);
    ÂÂÂÂÂÂÂÂif ( v != ~(u32)0 )
    ÂÂÂÂÂÂÂÂ{
    ÂÂÂÂÂÂÂÂÂÂÂÂint j;
    ÂÂÂÂÂÂÂÂÂÂÂÂfor ( j = 31; j >= 0; j-- )
    ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂif ( !(v & 1hd_sec_sz_s;
    ÂÂÂÂp->head->n_hd_sec = wbfs_htonl(p->n_hd_sec);

    ÂÂÂÂ// make all block full
    ÂÂÂÂmemset(p->freeblks,0,p->n_wbfs_sec/8);
    ÂÂÂÂwbfs_sync(p);

    ÂÂÂÂ// os layer will truncate the file.
    ÂÂÂÂreturn max_block;
    }

    Cutting the file looks then like:

    Code:
    ÂÂÂÂwbfs_t * w = ...
    ÂÂÂÂ...
    ÂÂÂÂwbfs_trim(w);
    ÂÂÂÂconst u64 cut = (u64)w->n_hd_sec * w->hd_sec_sz;
    ÂÂÂÂset_file_size(file,cut);
    Now it's time to test ...


    P.S.:
    You can use wwt (see signature) for dumping a memory map of any WBFS and controlling the results of trimming and other actions:

    Code:
    wwt dump -lll wbfs_file
    EDIT: a little correction: part_lba+1 is not needed for size calculations.
     
  12. oggzee

    oggzee GBAtemp Addict

    Member
    2,335
    16
    Apr 11, 2009
    Slovenia
    It seems to me that the first 2 points can never happen when just adding 1 iso to a fresh wbfs file, which is the only scenario that wbfs_file does. If you want to use that to trim and resize a real partition then yes there are problems with it, but wbfs_file would never use it for that.
    As for the third point, i don't understand why would you add part_lba? - If it is a real partition, then it's wrong, if it's a file it's always zero. So the only question remains is, if the +1 is needed? I seems to me that it isn't, since the just allocated block has the position past all data... and also all the converted iso to wbfs files seem to work fine... but i could be mistaken.
     
  13. Wiimm
    OP

    Wiimm Developer

    Member
    2,129
    336
    Aug 11, 2009
    Gambia, The
    Germany
    The second point can happen if the WBFS is full.

    And you are right: p->part_lba is only needed for offsets but not for size. I change the post.
     
  14. Wiimm
    OP

    Wiimm Developer

    Member
    2,129
    336
    Aug 11, 2009
    Gambia, The
    Germany
    I'm not sure if bug 3 really happens. I will work on that later.
    oggzee: thanx for brain storming
     
  15. Wiimm
    OP

    Wiimm Developer

    Member
    2,129
    336
    Aug 11, 2009
    Gambia, The
    Germany
    Bugs 1. and 2. are still there but does not appear in newly created wbfs that are not full.

    Bug 3. is an interpretation error of me: At the beginning of a WBFS all WBFS sectors (multiple of 512, on 500G partitions about 8 MB) has an delta offset of 1 hd sector (typical 512 bytes) but in the data area that offset is 0. That has confused me.
     
  16. Wiimm
    OP

    Wiimm Developer

    Member
    2,129
    336
    Aug 11, 2009
    Gambia, The
    Germany
    I have made some more small tests. With sector size 512 (default) a WBFS must be smaller as 2 TiB. But when increasing the sector size larger WBFS partitions seems possible. To create a test partition use WWT and type for example:

    Code:
    wwt init --size 4t --sector-size 1024 --split-size 2t --force a.wbfs
    Splitting is needed because the maximal single file size.

    The created empty WBFS uses only 16 KiB of the ext3 harddisk:
    Code:
    # ls -lsh a.wbf*
    ÂÂ0 -rw-r--r-- 1 root m7 1.9T 2009-11-06 17:40 a.wbf1
    16K -rw-r--r-- 1 root m7 1.9T 2009-11-06 17:40 a.wbfs
    And this is the WBFS dump:
    Code:
    # wwt dump -lll a.wbfs
    
    DUMP of a.wbfs
    
    ÂÂWBFS-Header:
    ÂÂÂÂMAGIC:ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ'WBFS' = 57 42 46 53
    ÂÂÂÂnumber of sectors:ÂÂ 279303400
    ÂÂÂÂhd sector size:ÂÂÂÂÂÂÂÂÂÂÂÂ 10 ->ÂÂÂÂÂÂ 1024
    ÂÂÂÂWBFS sector size:ÂÂÂÂÂÂÂÂÂÂ 26 ->ÂÂ 67108864
    
    ÂÂÂÂDisc table:
    ÂÂÂÂÂÂÂÂ0.. 59: .......... .......... .......... .......... .......... ..........
    ÂÂÂÂÂÂ 60..119: .......... .......... .......... .......... .......... ..........
    ÂÂÂÂÂÂ....
    ÂÂÂÂÂÂ960..1011: .......... .......... .......... .......... .......... ..
    
    ÂÂhd:ÂÂ sector size:ÂÂÂÂÂÂÂÂÂÂ1024 = 2^10
    ÂÂhd:ÂÂ num of sectors: 3906250000
    ÂÂhd:ÂÂÂÂtotal size:ÂÂÂÂÂÂ 3814697 MiB
    
    ÂÂwii:ÂÂsector size:ÂÂÂÂÂÂÂÂ 32768 = 2^15
    ÂÂwii:ÂÂnum of sectors:ÂÂ122070016
    ÂÂwii:ÂÂsectors/disc:ÂÂÂÂÂÂ 286864
    ÂÂwii:ÂÂ total size:ÂÂÂÂÂÂ 3814688 MiB
    
    ÂÂwbfs: sector size:ÂÂÂÂÂÂ67108864 = 2^26
    ÂÂwbfs: num of sectors:ÂÂÂÂÂÂ59604
    ÂÂwbfs: sectors/disc:ÂÂÂÂÂÂÂÂÂÂ140
    ÂÂwbfs:ÂÂtotal size:ÂÂÂÂÂÂ 3814656 MiB
    
    ÂÂpartition lba:ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ 0
    ÂÂfree blocks lba:ÂÂÂÂÂÂÂÂÂÂ 65528
    ÂÂdisc info size:ÂÂÂÂÂÂÂÂÂÂÂÂ 1024
    
    ÂÂused disk space:ÂÂÂÂÂÂÂÂÂÂÂÂ1280 MiB =ÂÂ 1%
    ÂÂfree disk space:ÂÂÂÂÂÂÂÂ 3813376 MiB =ÂÂ99%
    ÂÂtotal disk space:ÂÂÂÂÂÂÂÂ3814656 MiB = 100%
    
    ÂÂnumber of wii discs:ÂÂÂÂÂÂÂÂÂÂ 0ÂÂÂÂ =ÂÂ 0%
    ÂÂmax disc:ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ 1012
    ÂÂdisc open:ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ 0
    
    
    ÂÂWBFS Memory Map:
    
    ÂÂÂÂÂÂÂÂunused :ÂÂoff(beg) ..ÂÂoff(end) :ÂÂÂÂ size : info
    ÂÂ --------------------------------------------------------------------------------
    ÂÂÂÂÂÂÂÂÂÂÂÂÂÂ :ÂÂÂÂÂÂÂÂ 0 ..ÂÂÂÂÂÂÂÂ c :ÂÂÂÂÂÂÂÂc : WBFS header
    ÂÂÂÂÂÂÂÂÂÂÂÂÂÂ :ÂÂÂÂÂÂÂÂ c ..ÂÂÂÂÂÂ 400 :ÂÂÂÂÂÂ3f4 : Disc table
    ÂÂÂÂÂÂÂÂÂÂÂÂÂÂ :ÂÂÂÂÂÂ 400 ..ÂÂÂÂ fd400 :ÂÂÂÂfd000 : Disc info
    ÂÂÂÂÂÂ 3f00c00 :ÂÂ 3ffe000 ..ÂÂ 3fffd1a :ÂÂÂÂ 1d1a : Free blocks table
    ÂÂÂÂ 3a34e9442e6 :3a352944000 ..3a352944000 :ÂÂÂÂÂÂÂÂ0 : -- end of WBFS device/file --
    Partitions larger than 4 TiB are not possible. There seems a limit in "wbfs sector size".
     
  17. tj_cool

    tj_cool Site dev

    Supervisor
    9,985
    -1
    Jan 7, 2009
    Belgium
    This planet
    You have to be a really big pirate if you can fill 4 TebiByte
     
  18. Wiimm
    OP

    Wiimm Developer

    Member
    2,129
    336
    Aug 11, 2009
    Gambia, The
    Germany
    I'm really new in the WBFS business. When i take a look into the Wiki/WBFS managers I see "Sorgs's Mod" and other things. Can anybody please tell me:
    • What is "Sorgs's Mod"?
      .
    • What is the "Delete Bug Fix" in technical details?
      (Do I have them in WWT? Tests says: NO / I use kwiirk wbfs as base for my wwt.)
      .
    • And btw: Where can I found the CISO specification.
      .
     
  19. Wiimm
    OP

    Wiimm Developer

    Member
    2,129
    336
    Aug 11, 2009
    Gambia, The
    Germany
  20. Wiimm
    OP

    Wiimm Developer

    Member
    2,129
    336
    Aug 11, 2009
    Gambia, The
    Germany
    A good message for Win/NTFS users:
    I have played with the newest cygwin version to prove the functionality. And what did I found?

    NTFS supports sparse files too!

    When creating a WBFS or extracting a raw ISO file from a small game the disc usages is much less than the file size. But when copying the ISO this advantage is lost. This means: WDF is still the best solution for storing ISO images.