Hacking WBFS->WBFS Most Reliable App?

[Skizzo]

There's been a few new WBFS tools popping up lately. So, do any of the tools, old or new, have a RELIABLE direct, on-the-fly WBFS->WBFS ability? I'm getting ready to do my friend's HD, who also just happens to own the exact same 150+ games I own! Figured it would be easier and save her some wear and tear on her Wii if I just copied them over from my HD.

We've got the exact same enclosures/drives, if that makes any difference to the tools. So, what's the best, with 'best' meaning the most reliable as I'm not too concerned with fancy graphics and whatnot? Are there any reliable ones that I can set and forget, and preferably ones not relying on the .net framework, but that isn't an absolute requirement. And just out of curiousity, what kind of time can I expect it to take to copy over ~450GB of games?

On another note, my enclosure is a USB/eSATA one. Do the WBFS tools only work if it's connected to USB, or will they work when connected to eSATA too? I haven't hooked up an eSATA connection on my computer yet, otherwise I'd test myself, but if someone confirms they'll work through eSATA I may just have to do so.

Thanks for any info.

 

[KiiNG]

I just dumped about 250 games (literally, just finished) from WBFS to WBFS using WBFS Intelligent GUI. Everything went great. Lengthy, but that's expected. >> Thread Here

 

[CasperH]

I would go for WBFS Manager but it relies on .NET Framework

 

[laurozza]

I would go for WBFS Manager but it relies on .NET Framework

It's not raccomandable ?

 

[Skizzo]

I just dumped about 250 games (literally, just finished) from WBFS to WBFS using WBFS Intelligent GUI. Everything went great. Lengthy, but that's expected. >> Thread Here

 

[KiiNG]

I tested about 20 or so random games, and they all loaded as they should. I stopped testing after that, but everything seems to run smoothly.

My target drive didn't increase in size, period. It's exactly the same size, down to the byte. I've never deleted a game from my main drive, and if I read correctly in a few forums -- the increase is normally found if you do/have delete games from your main partition.

I'm on a Mac, so I used my native tools to create a FAT partition, then used WBFS I.GUI to format the drive to WBFS. I actually did the entire WBFS WBFS transfer in a virtual environment of XP-SP3.

And, yes, I run the exact copy of the program found in the thread. I didn't change or rename anything. I can't comment on what the new version would offer or do, but I'd only assume if it contains different code, it might not jive too well.

 

[Skizzo]

Thanks again for the reply KiiNG. I'm getting ready to start this process soon and a few more questions popped up.

Although I've not really had any problems with my WBFS drive, for shits and giggles I ran the Check Integrity function of Intelligent GUI and it came back with 4 Crossed LBA errors.

2 of the 4 games affected needed replacing anyways (one has a PROPER out and the other was a PAL release I grabbed before the NTSC release was out) so I just removed them, but the other 2 I'll just have to redo (luckily I keep archives of all the original release files, although this is beginning to get costly so I think I'll just start keeping cISO's if research deems that reliable). Anyways, I've seen mention of 'cloning' drives. Does that do exactly what it sounds like, including cloning any fragmentation that might be on the drives, and if so, does copying them over individually avoid that? I mean, in Windows, if I have a heavily fragged drive and copy the contents to another new drive just using Explorer, the files will be defragged in the process. Do you know if it's the same with WBFS? I'd assume so, but you never know...

I just got a WD 1TB MBE last night so my plan is to copy the games off my current 1TB drive which has 450GB partitions for WBFS and NTFS with the rest as FAT32 onto my friend's 1TB drive which is 900GB WBFS and the rest FAT32. After testing the results, I'll copy the NTFS portion of my drive onto the new WD which I'll just be using for PC storage. Finally I'll repartition my WBFS drive to give me the same 900GB WBFS partition and then copy all the games back from her drive onto mine.

I think if all of that goes well, I'll be sold on WBFS Intelligent GUI!

BTW, can any of the WBFS tools export a list of the games (preferably not truncated) as well as their size?

EDIT: Forgot to add, I usually don't do anything else on my computer when I'm burning anything, but I wonder if it's safe to use it while doing these transfers? I know it's not the same thing as burning, but I wonder what kind of verification/error checking, if any, is used for WBFS? Guess I'll just have to try it and see.

EDIT2: FUCKITY FUCK FUCK! It's not working at all! Gotta run but will post details when I get back home.

OK, well I gave this a go and no luck. I've got 187 games on my current WBFS drive totaling about 407GB. I tried the Check All option and clicked the >> and it appeared like it was working, with the progress bar moving along for the first game but then it got to the next one and it just started going through all the games, 'processing' them really quickly, roughly 2-3 seconds each. Of course, when it finished, it said all games were copied successfully, but the next box to pop up immediately after clicking OK was the same one you get right after formatting, telling me my partition was empty. Tried again, only selecting a single game, but still no go. Tried formatting to WBFS with a few different tools as well. I don't know what the problem could be. I tried wbfsGUI v14.2 and it appeared to be working, but seemed to be going really slow? Do the following times sounds reasonable, or is something not right (size of game/time/copied game's size on new drive)?

1648MB/02:24/1680MB
3136MB/04:33/3168MB
1288MB/01:53/1312MB
112MB/00:11/144MB
4232MB/06:08/4256MB
2128MB/03:06/2160MB

And what's up with the 176MB increase in the total size of the games? I'm almost positive I set the drives up the same way using the default values. I'm going to do a new game on both from an ISO and see what sizes I get.

EDIT3: Well, I put Muramasa on both drives using PrixM's tool. The tool reports the game being 0.63GB. When put on my drive it grows to 0.65GB and on the new, identical drive it goes up to 0.67GB?

If anyone could shed any light on any of this it would be greatly appreciated. I've got to get this done by Sunday evening!

 

[sidishere]

i would too like to know

 

[Benjay]

@Skizzo : try ncWBFSTool , it worked like a charm for me to copy 102 games from a HDD to another , on the fly.
Can't tell you how much time it took , because i left home at the beginning , and when i came back (7 hours later) , it was finished ^^

If you wanna give it a try , look at the last page (or previous , don't remember) of the ncWBFSTool thread , to get the latest build.

 

[Pakatus]

Personally i've been standing with WBFS Manager.
Might be old, but its stable...

ncWBFSTool is a great tool and i've been using it to backup my "unwanted" games to cISO in another HD. However it still returns some exceptions sometimes, in simple actions like "Add Games".

But in fact, im looking forward the new release of WBFSSync. It's awesome, stable and really easy to use. One of my fav things about this program, is that you can parallelize tasks.
Looking forward a new release, where i expect to have the cISO support and direct DISK to WBFS functionality.

 

[Skizzo]

@Benjay - Thanks for the suggestion. I'd already downloaded the original version, but I'm a bit leery using apps when they first come out. I'll check out the latest version. I guess instead of leaving my new 1TB MBE drive entirely as NTFS, I'll make a small WBFS partition to be able to keep testing these apps as they develop, because I'm sure this won't be the last time I have to go through this.

@Pakatus - I did give WBFS Manager a quick go, but as I said, I'm looking for on-the-fly copying as the other method just seems like a big waste of time, creating a temp. 4.3GB file on my drive to copy over every single game, regardless of its size?! No thanks. Why hasn't he implemented on-the-fly copying?

@ Everyone - I'm still wondering why the games are increasing in size. I'll have to see what I can find out about this WBFS. Is there a place where the creator discussed it in detail?

Oh, I don't know what happened, but last night after playing a little TFC (I'm old-skool like that

) I went back and tried a few things with formatting and whatnot, and then went to put Muramasa back on and now the tools were all showing the ISO as being 0.66GB instead of the 0.63GB all of them were showing previously? Strange. I've only briefly followed all of this and I know I recall seeing mention of some/all tools not reporting the correct size, but you'd think they would at least be consistent in that regards.

Anyways, last night before going to bed I started up wbfsGUI v14.2 and let it do its thing and when I got up it was done. Now to hand the drive off to the kids to give a good test to a sampling of the games, before I go and reformat/repartition my drive.

If I find out any useful information regarding any of the questions I've had, I'll be sure to add it here.

Thanks for the replies Benjay & Pakatus.

EDIT: Forgot to mention, in wbfsGUI my drive shows 408.54/450.00GB used, and after copying all the games onto her drive, her's shows 415.27/900.00GB used. So, that's a 6.73GB increase in size. All of the games appear to have gone up in size by either 24 or 32 MB?

I ran just the command line tool wbfs_win.exe on the drives to check the integrity and get their 'info'. First thing I noticed is that the integrity check goes much faster on the newly copied to drive. I don't know if any of you have ever had to watch a bag of popcorn popping in the microwave, standing there counting between each kernel pop, but if this were the same, the bag would have fully popped.

Whereas with my current drive I'm not sure I would have made it past five kernels.

Guess we can attribute that to fragmentation? Or...

From the 'info' command...I get these figures.

MY DRIVE/NEW DRIVE

Blocks 5307/31023
Total 450.00/900.00
Used 408.54/415.27
Free 41.46/484.73

I assumed those blocks represent FREE blocks, which was confirmed by WBFS Manager which is the only tool I've seen so far that even mentions blocks (would be nice if it showed game size in blocks as well as MB). So, if you do the math, this means that for my 450GB partition, each GB is equal to 128 blocks (or 8MB per block), whereas for the new 900GB partition each GB is equal to only 64 blocks (or 16MB per block), which if equivalent to Windows allocation units (and I can't see why it wouldn't be), would mean a less efficient storage scenario for the latter, but with the advantage of increased speed. Unless I'm just totally mistaken about all of this, which is quite possible.

If I'm not mistaken, who/what determines the block (or allocation unit) size in WBFS? Seems kind of strange that it would vary and based upon what? The size of the drive has nothing to do with the content being put on said drive, at least not in this case. Well, I'm off to do some more research to see if I can find any answers to these questions. In the meantime, if anyone has any insight into these matters, please share.

One last thing. Is it possible to pipe the output of the command line tool wbfs_win.exe? I've tried > info.txt with no success and that's about the limit of my knowledge.

EDIT: OK, well I did some poking around and let me just preface the following with the disclaimer that I know next to nothing about coding unless you need something in BASIC.

Anyways, I was looking in libwbfs.h and came across the following, which I'm wondering if it might explain the difference in WBFS 'sector' (block) sizes.

typedef struct wbfs_s
{
ÂÂÂÂÂÂÂÂwbfs_head_t *head;

ÂÂÂÂÂÂÂÂ/* hdsectors, the size of the sector provided by the hosting hard drive */
ÂÂÂÂÂÂÂÂu32 hd_sec_sz;
ÂÂÂÂÂÂÂÂu8ÂÂhd_sec_sz_s; // the power of two of the last number
ÂÂÂÂÂÂÂÂu32 n_hd_sec;ÂÂÂÂ // the number of hd sector in the wbfs partition

ÂÂÂÂÂÂÂÂ/* standard wii sector (0x8000 bytes) */
ÂÂÂÂÂÂÂÂu32 wii_sec_sz; 
ÂÂÂÂÂÂÂÂu8ÂÂwii_sec_sz_s;
ÂÂÂÂÂÂÂÂu32 n_wii_sec;
ÂÂÂÂÂÂÂÂu32 n_wii_sec_per_disc;
ÂÂÂÂÂÂÂÂ
ÂÂÂÂÂÂÂÂ/* The size of a wbfs sector */
ÂÂÂÂÂÂÂÂu32 wbfs_sec_sz;
ÂÂÂÂÂÂÂÂu32 wbfs_sec_sz_s; 
ÂÂÂÂÂÂÂÂu16 n_wbfs_sec;ÂÂ // this must fit in 16 bit!
ÂÂÂÂÂÂÂÂu16 n_wbfs_sec_per_disc;ÂÂ // size of the lookup table

ÂÂÂÂÂÂÂÂu32 part_lba;
ÂÂÂÂÂÂÂÂ/* virtual methods to read write the partition */
ÂÂÂÂÂÂÂÂrw_sector_callback_t read_hdsector;
ÂÂÂÂÂÂÂÂrw_sector_callback_t write_hdsector;
#ifdef WIN32
ÂÂÂÂÂÂÂÂclose_callback_t close_hd;
#endif

ÂÂÂÂÂÂÂÂvoid *callback_data;

ÂÂÂÂÂÂÂÂu16 max_disc;
ÂÂÂÂÂÂÂÂu32 freeblks_lba;
ÂÂÂÂÂÂÂÂu32 *freeblks;
ÂÂÂÂÂÂÂÂu16 disc_info_sz;

ÂÂÂÂÂÂÂÂu8ÂÂ*tmp_buffer;ÂÂ// pre-allocated buffer for unaligned read
ÂÂÂÂÂÂÂÂ
ÂÂÂÂÂÂÂÂu32 n_disc_open;
ÂÂÂÂÂÂ 
}wbfs_t;

In particular, the line u16 n_wbfs_sec; // this must fit in 16 bit!. If this refers to the total number of sectors then that would explain the need for larger sectors on the larger partition. The 450GB partition @ 128 blocks per GB comes out to 57,600 blocks, which as everyone should know, fits into 16 bits. However, doubling that number for a 900GB partition would give us 115,200 blocks, which does not. If that is indeed the explanation, then it looks like it would be impossible to get the more efficient storage of the 450GB partition, at least without modifying the code, if that is even possible. I guess the maximum partition size possible with the more efficient block size would be 65,535/128 or 511.9921875GB. I wonder if it uses an even more efficient size than the 128 blocks per GB if the partition size is small enough. All in all, no big deal but it's nice to understand why things are working as they are.

Apologies if this was all well known amongst the more informed. Guess some of us learn something new every day.

 

[nubecoder]

From Sorg's 1.0 mod source code

CODE
static int get_capacity(char *fileName, u32 *sector_size, u32 *sector_count)
{
DISK_GEOMETRY dg;
PARTITION_INFORMATION pi;

DWORD bytes;
HANDLE *handle = CreateFile(fileName, GENERIC_READ | GENERIC_WRITE, 0, NULL, OPEN_EXISTING, FILE_FLAG_NO_BUFFERING, NULL);

if (handle == INVALID_HANDLE_VALUE)
{
wbfs_error("could not open drive");
return 0;
}

if (DeviceIoControl(handle, IOCTL_DISK_GET_DRIVE_GEOMETRY, NULL, 0, &dg, sizeof(DISK_GEOMETRY), &bytes, NULL) == FALSE)
{
CloseHandle(handle);
wbfs_error("could not get drive geometry");
return 0;
}

*sector_size = dg.BytesPerSector;

if (DeviceIoControl(handle, IOCTL_DISK_GET_PARTITION_INFO, NULL, 0, ?, sizeof(PARTITION_INFORMATION), &bytes, NULL) == FALSE)
{
CloseHandle(handle);
wbfs_error("could not get partition info");
return 0;
}

*sector_count = (u32)(pi.PartitionLength.QuadPart / dg.BytesPerSector);

CloseHandle(handle);
return 1;
}

DeviceIoControl @ msdn

[edit]
Oops forgot to add the useful bits from the wbfs_open_partition function

p->n_wii_sec = (p->n_hd_sec/p->wii_sec_sz)*(p->hd_sec_sz);
p->n_wbfs_sec = p->n_wii_sec >> (p->wbfs_sec_sz_s - p->wii_sec_sz_s);

[/edit]

 

[Skizzo]

Any chance you could explain what that means nubecoder? The only reason I picked up on that info in the other code was simply because it was commented. Was I correct in my findings though?

 

[nubecoder]

ok from a second look at the function I missed some important code, so here's a stripped down version of the wbfs_open_partition function with some notations..

Spoiler

-snipped-
p->wii_sec_sz = 0x8000;// ** a wii disc's sector size is 32k bytes
p->wii_sec_sz_s = size_to_shift(0x8000);// ** wii sector size_s (power of 2)
p->n_wii_sec = (num_hd_sector / 0x8000) * hd_sector_size; // ** wii sector count is hd sector count divided by 32k multiplied by hd sector size
-snipped-
if (reset)// ** if formatting
{
unsigned char sz_s;// ** create variable to be used below
-snipped-
// choose minimum wblk_sz that fits this partition size
for(sz_s = 6; sz_s < 11; sz_s++)// ** start with 6 and stop at 11
{
// ensure that wbfs_sec_sz is big enough to address every blocks using 16 bits
if(p->n_wii_sec < ((1U wii_sec_sz_s;// ** wbfs sector size_s (power of 2) is value found above plus wii sector size_s (power of 2)
}
-snipped-
p->n_wii_sec = (p->n_hd_sec/p->wii_sec_sz)*(p->hd_sec_sz);// ** wii sector count is hd sector count divided by wii sector size multiplied by hd sector size
p->wbfs_sec_sz = 1 wbfs_sec_sz_s;// ** wbfs sector size is 1 shifted left by wbfs sector size_s (power of 2);
p->n_wbfs_sec = p->n_wii_sec >> (p->wbfs_sec_sz_s - p->wii_sec_sz_s);// ** wbfs sector count is wii sector count shifted right by wbfs sector size_s (power of 2) minus wii sector size_s (power of 2)
-snipped-

That's the best I can do at the moment. Comments added by me are noted with **.

I dislike binary shifts because they confuse me, and this power of 2 stuff adds to my confusion (cuz I'm a nube).

But to put it simply, as I understand it, the wbfs sector size is created by making sure the total wbfs sector count for the partition will fit into 16 bits.

Without going into any math I would assume what you posted above about partition size would make sense.

Hope this helps a little, maybe someone can explain it better?

=]

[edit]
Now that I think about it.. wasn't there some info a while back about wbfs limiting partitions to 512 GB?
Maybe that was earlier on and the source code that I have adjusts the sector size so it will fit into 16 bits.
[/edit]

 

[Skizzo]

Thanks for that nubecoder. Now I feel comfortable proceeding with my drive without fearing that I'll learn right after I finish transferring all the games that I could have gotten the smaller sector size.

 

[mousex]

We've got the exact same enclosures/drives, if that makes any difference to the tools.

If you own the same games and have the same drives I guess dd is the best tool (provided that you WBFS partition is nearly full).
It will copy your whole WBFS partition with full speed to the other HDD.

 

[Skizzo]

Have you ever done that mousex? I assume it will do a 1:1 copy, including any fragmentation too, although that shouldn't be a problem with a newly copied to drive that hasn't had any games deleted from it. I'd like to give it a try but I've never used dd before. I wonder what 'speed' the WBFS managers are copying at compared to dd's 'full speed'? Have you ever compared them?

 

[moparisthebest]

We've got the exact same enclosures/drives, if that makes any difference to the tools.

If you own the same games and have the same drives I guess dd is the best tool (provided that you WBFS partition is nearly full).
It will copy your whole WBFS partition with full speed to the other HDD.

Took the words straight out of my mouth. dd will copy the entire hard drive if you want it to bit for bit at the fastest speed both will spin. If everything is the same, it should be perfect and faster than any WBFS->WBFS tool because they involve some processing, where dd doesn't have to do any.

 

[Wiimm]

We've got the exact same enclosures/drives, if that makes any difference to the tools.

If you own the same games and have the same drives I guess dd is the best tool (provided that you WBFS partition is nearly full).
It will copy your whole WBFS partition with full speed to the other HDD.

Took the words straight out of my mouth. dd will copy the entire hard drive if you want it to bit for bit at the fastest speed both will spin. If everything is the same, it should be perfect and faster than any WBFS->WBFS tool because they involve some processing, where dd doesn't have to do any.

That's not right! If the WBFS is not full dd needs much time to copy all unused disc space too.

 

[moparisthebest]

That is true, never thought about that, but I would say dd would still be faster for the 150 games he says he has.