Hacking wbfs2fat.py

[DimitriPilot3]

Thank you for the tips. They kinda motivated me to look into this "corruption" in more detail, hoping to find the problem with the knowledge/experience I have...
And it looks like you figured out about the ":" colon issue already... "Illegal" characters can be evil as I already witnessed once

what vers of windows are you useing and on what fs (ntfs/fat) the backup was on?

- My OS is Windows 7 Ultimate 32-bit (aka x86), in French. My basic computer specs are in my signature.
- As I said (unless I edited the post after you've read it), the backup was saved in the wbfs2fat/windows folder, which itself sits on NTFS drive "C:" (E: is my system drive, C: is another drive; both drives are NTFS).

I would like you to give me a backup of the first part of the drive (~1-2% I guess, I would need to calculate that) e.g. using "dd"
I mainly would like it to analyse what went wrong
its possible I find a way to fix your fs, but that would take time and I can't garantee anything
If you are willing to do that I can look into the details

I have managed to dump some raw data using EditHexa, just enough (less than 60 MiB after un-RAR-ing) to contain (what I think are) the "FAT directory entries" at the end of the dump. Is that enough? (The whole HDD contains a total of 973198272 sectors each being 0x200 bytes long. The total storage capacity is 465 GB (GiB?) according to windows.)

EditHexa reports each "wbfs file chunk" as "deleted" (the first character of each game ID is garbage), though they appear to have a valid pointer to the wbfs game data (for example, the first chunk of "åSBP01" aka "RSBP01" does point to the beginning of the Brawl ISO, at sector #71073792). I wonder, is that caused by wbfs2fat? Or was that actually done by Chkdsk? Either way, undeletion should be possible, isn't it?
As for the "WBFS" folder (entry #16), EditHexa says that it begins at sector #973197824. Oddly enough, the sectors before this one are readable (containing 0s), but sectors #973197824 through #973198263 are unreachable for EditHexa (checksum error, appearing after some seconds of processing -- what does this error mean?). And inside sector #973198272 there is some more data (let me know if you need it), including ASCII text "RMCP01~WBF ".

I am not very familiar with how FAT32 works, but I believe the clues above should be a good start to find what is wrong in the FAT structure.

in your place I would try other filesystem checkers (in readonly mode first)
I would use fsck.vfat but there are other ones for windows besides chkdsk
...
you could try wwt to fix the filesystem
afaik it has a readonly mode too, called testing iirc

I haven't tried other verification tools yet...

what makes this even more strange is that if the fat created was corrupt
wbfs2fat would have been unable to rename/move the files
causing them to still be in the root of the drive instead of the wbfs folder
(and would have printed errors to the log)

From what I remember, nothing looked strange in the wbfs2fat log. If there was anything interesting in it, it might have been drown by the verbosity (logging about each WBFS file)...

was there some other program accessing the drive at the same time?
like a wbfs manager?

To me, it wouldn't have been very smart if I had left "WBFS Manager 3.0" open during the conversion... I'd be surprised if that were the case... So I'd say "no".

I am terribly sorry

I have the feeling it should end quite well, especially if it helps you with bug-fixing (improving the reliability)

EDIT: So the backup data was stored as an Alternate Data Stream thanks to the colon?! Now I understand why the "wbfs.G" file is created but not written to; the relative destination is in fact ".1329486159.back".
Managed to find the name of the backup after a lot of trial & error (batch-opening notepad instances for various timestamps until one file seems to be found) : wbfs.G:.1329486159.back

 

[PsyBlade]

I have managed to dump some raw data using EditHexa, just enough (less than 60 MiB after un-RAR-ing) to contain (what I think are) the "FAT directory entries" at the end of the dump. Is that enough? (The whole HDD contains a total of 973198272 sectors each being 0x200 bytes long. The total storage capacity is 465 GB (GiB?) according to windows.)

EditHexa reports each "wbfs file chunk" as "deleted" (the first character of each game ID is garbage), though they appear to have a valid pointer to the wbfs game data (for example, the first chunk of "åSBP01" aka "RSBP01" does point to the beginning of the Brawl ISO, at sector #71073792). I wonder, is that caused by wbfs2fat? Or was that actually done by Chkdsk? Either way, undeletion should be possible, isn't it?
As for the "WBFS" folder (entry #16), EditHexa says that it begins at sector #973197824. Oddly enough, the sectors before this one are readable (containing 0s), but sectors #973197824 through #973198263 are unreachable for EditHexa (checksum error, appearing after some seconds of processing -- what does this error mean?). And inside sector #973198272 there is some more data (let me know if you need it), including ASCII text "RMCP01~WBF ".

I am not very familiar with how FAT32 works, but I believe the clues above should be a good start to find what is wrong in the FAT structure.

I think you already found the problem.
I believe that the sectors in the end are defective, while this cant happen because of software there are a lot of potential causes (age, production error, bad luck, shock, vibration, ...)
Since there are multiple sectors in a row my first guess would be shock.
To confirm this theory look for errors in the windows system log or run a test that looks for defective sectors
(mostly called surface scan or bad block test - consider using "badblocks" if you can use linux/unix)

Yes undeleting should work. You should try it as fast as possible before the deleted files are overwritten.

There is nothing in wbfs2fat that should delete files.
I believe that chkdsk did that because it could not read the directory the files where in (and you answered that you did not want to keep them).
Maybe that notice at the start was only about automatic fixes without asking for confirmation.
As there is no way of knowing what else was already changed by it I would advise against restoring the backup.

Assuming that the sectors are defect:
You need to "fix" them.
Drives have a contingent of spare sectors that can be used for automatic remapping.
You need to read the whole drive (the bad block scan does that) to "find" the defect ones
and then try to write at least the devective ones (it should do that too)
(If the drive fails to read a sector it internally marks it as bad, the next write then causes a remap.)
I only ever used "badblocks -vns" for that - ymmv.

If after this there are still bad sectors left (because the spare ones are used up)
you should really throw the drive away.
even if old filesystems such as fat still have provisions in place to handle (unremaped) bad sectors
its bound to get worse - fast.

 

[MassiveRican]

OMG!! I'm really crying... why the hell didn't I find this any sooner... hours of my life lost and I only have myself to blame because you made this wonderful program.. damn it I wish I would have found this months ago!!

Thanks for this man, a real time (therefore) life saver

 

[Skater4599]

I think I'm missing something simple here, I'm in a windows machine and I can't get the GUI to start at all, I double click the .exe file and the command prompt pulls up flashes something I can't read cause its so fast then nothing happens. Any hints would be appreciated. Thanks in advance!

 

[PsyBlade]

try to start it from a command prompt to see the error
maybe even use the commandline options to run it without gui
see the "--help" option for details

 

[Skater4599]

Running it from a command line this is the output I get.

Traceback (most recent call last):
File "", line 11, in 
File "F:\src\wbfs2fat.py", line 592, in main
File "f:\pyinstaller\build\pyi.win32\wbfs2fat\outPYZ1.pyz/gui
startgui
File "f:\pyinstaller\build\pyi.win32\wbfs2fat\outPYZ1.pyz/gui
17, in __init__
File "F:\src\wbfs2fat.py", line 48, in scanwbfs
File "F:\src\wbfs2fat.py", line 92, in __init__
struct.error: unpack_from requires a buffer of at least 8 bytes

EDIT: Fixed... strange thing to happen for what fixed it, I simply removed the DVD I had in my drive GUI now opens and command line options function as well.

 

[PsyBlade]

try http://www.unix-ag.u...4-beta1.tar.bz2

edit: saw your edit
It kind of explains it
wbfs2fat tries reading all drives to find the wbfs ones
seems it did not like what it read from the dvd

 

[Skater4599]

Ah thanks for the help anyway. This program worked super fast!

Curious if it formats it to FAT32 with 32kb clusters or 64kb cluster size?

 

[PsyBlade]

the cluster size is 32k (or less on smaller devices)

 

[mabeard]

okay newbee here. I have downloaded the programe and found thewindows application but it will not open what am i doing wrong

 

[KingMuk]

I know this is gonna sound real stupid but after I go from WBFS to FAT32 my HD is renamed to wbfs2fat...so should I rename that to "fat32"???

 

[PsyBlade]

I know this is gonna sound real stupid but after I go from WBFS to FAT32 my HD is renamed to wbfs2fat...so should I rename that to "fat32"???

you can rename it to whatever you want
personally I use "Wii-Games"

 

[PsyBlade]

okay newbee here. I have downloaded the programe and found thewindows application but it will not open what am i doing wrong

run it from a commandline to see if there is an error message

press win+r and enter "cmd" to get a command line
use "cd" to got to the dir where you extracted it to
then run it with "wbfs2fat.exe"

 

[Mantis41]

Hi, I just wanted to say thanks for the util.I have been putting off moving away from WBFS for ages. It was under a minute and finished and worked great.

 

[Loop]

Wow! This app sounds awesome & exactly what I need.
I'll definitely give it a try.
Thanks PsyBlade!

 

[asktoby]

I just ran this to convert my WBFS drive to FAT32 and ended up with a directory called /wbfs containing further directories for all my games. Each of those contained files like RLBPWR.wbfs.
When I connect the drive to my Wii now, it says I have zero games on the disk! Do I need to do a further step?

 

[smf]

I just ran this to convert my WBFS drive to FAT32 and ended up with a directory called /wbfs containing further directories for all my games. Each of those contained files like RLBPWR.wbfs.
When I connect the drive to my Wii now, it says I have zero games on the disk! Do I need to do a further step?

Sounds like you might need to configure the loader you are using, which one is it?

 

[PsyBlade]

I just ran this to convert my WBFS drive to FAT32 and ended up with a directory called /wbfs containing further directories for all my games. Each of those contained files like RLBPWR.wbfs.

So far everthing seems right.
If you cant get it to work with your loader try if WBM (WiiBackupManager) sees the games.
If it does that means everything worked as intended and you might need to use a better loader.

 

[PsyBlade]

In light of the recent GC usb loading support (and a possible influx of users) I think I should post two concerns I have.

Hard drive started to come with differently sized sectors.
Its a hardware thing that can't be changed by the user.
(A handfull of drives have a jumper that enables emulation of 512B sectors.)

Because all my drives only support the traditional 512B sectors wbfs2fat is untested with the new 4KiB sector drives (or any other size not 512B).
But it should still work perfectly. Even if not your games should not be in danger. Simply use wbfs2fat to restore the automatic backup.
In either case please report your findings.

Additionally SOME tools have recently added support for LYING about the sector size.
That means the information in the WBFS says that the drive has a different sector size from what it really has.
This does only work with tools that support it.
wbfs2fat does not.
I curently have no other way of obtaining that info and do need the real value to create the fat32.
No need to try the resulting FS will simply not work.
If you do try it by accident use the restore function to fix it and get back to WBFS.
If a reasonable number of people want me to support this I will try to do this in the future.
But I can't make any promisses about success (unless someone shows my how to obtain the real sector size).


TL;DR: 4KiB support untested - sector size lying unsupported - restore autobackup should work regardless

 

[PsyBlade]

just uploaded 0.8.5
it now prefers the clustersizes as Dios Mios likes them
default/maximum is still 32KiB
but now it only goes to smaller values if it absolutely has to