Hacking wbfs2fat.py

[PsyBlade]

wbfs2fat.py - WBFS to FAT32 conversion
version 0.8.6 - 2012-07-27
© 2012 by PsyBlade - all rights reserved
http://gbatemp.net/t291320-wbfs2fat-py

Features:

• converts a WBFS partition into a Fat32 partition - on the fly/in place
• keeps Games as 4GiB splited ID6.wbfs files in wbfs folder
• blazing fast - over 10GiB/s! (50GiB Test in 4.27s)
• needs very little space on other drives
  only ~0.0125% of the partition size for Backup and RAM
• Windows + Linux (with some adjustments everything that runs python)
• both GUI and CLI available
• designed with security in mind
  only modifies disk after all other computations
  creates backup of modified data

Requirements:

Windows(.exe): nothing!​

Linux(.py): python (+PyQt4 for GUI)​

Windows(.py): python + pywin32 + WMI for python (+PyQt4 for GUI)​

Windows(Cygwin .py): Cygwin + its python package​

(The versions marked "update" need to be copied over the next earlier full version)​

​

Usage GUI:

run wbfs2fat without options (e.g. double click)​

Select partition from list​

Click "convert to Fat32"​

​

Usage CLI:

"wbfs2fat --help" to see all options​

"wbfs2fat -m scan" to list of all wbfs partitions​

"wbfs2fat -m convert " to convert​

​

FAQ

Spoiler

• yes Error 6 can be fixed - the msg even says how - see the error section below for details
• no it does NOT delete the games - that's the whole fucking point
• no it's not Linux only - look at the edit dates of the posts
• the first post is updated - the rest is not
• uploading it to every pirate hoster is pointless - you can get it from me free of charge
• I don't like it either and therefore forbid hosting it unless I give explicit permission
• yes I know you are gonna ignore this
• if you expect an answer from me you should post here - not in some other forum where I'm not registered
• looking at the (implied) Q's of this FAQ make makes one wanna bang ones head against a wall
• I still think it should be named "Tips for the mentally challenged" but apparently no one reads that

Exit Codes and Error descriptions

Spoiler

0: No Error
Everything went as expected

1: General Error
Something totally unexpected happened
Post the output and ask

2: Argument Error
Check the command line options for typos and the like

3: Not a WBFS partition (missing magic)
Partition is probably not in WBFS format.
If it really should be a WBFS at least the identifier in the header is missing.

4: Unknown WBFS Version
So far I only encountered versions 0 and 1.
If other versions appear wbfs2fat needs to be updated first.

5: additional free Blocks needed
Conversion needs more free space.
See chapter "Space" it the tutorial for details.

6: WBFS is corrupted: Blocks used multiple Times
Two or more games claim ownership of the same WBFS data, but only one can be right about it.
That meant that at least parts of some games are corrupted.
As possibly only a tiny part of a game was overwritten this may have gone unnoticed for years.

This is caused by a known bug in the original version of the wbfs C library.
If you delete a game using a manager, loader or whatever uses an affected library,
parts of another game are marked as free too. If you add a game (even with a good lib)
these parts are overwritten and the original game is corrupted.
Unfortunately it is impossible to recover the data, delete and re-rip is the only way to fix it.

wiimms "wwt" can figure out which game the block really belongs to and delete the corrupted ones.
You need to download it and run it from a command-line.
For details on how to do this see wit verify

If you can or want not do this there is another way.
Run wbfs2fat with the "ignore-error-6" option. This will generate one copy of the affected block for
each game that claims it. This will NOT fix nor identify the corrupted games.
But it will allow you to go ahead while keeping your games in the same (bad) shape they are in now.

7: permission denied
wbfs2fat was unable to read any data from the partition.
Most likely you simply lack the rights required to do so. Try running wbfs2fat as admin/root.
And try closing all WBFS mangers because they sometimes lock the drives.
Empty CD drives and the like can show this too.

Licence

Spoiler

wbfs2fat is copyrighted software, use it as such with the exceptions mentioned below

You may to download and use it free of charge.

You are not allowed to host/sell/rent or in any other way provide it to others.
(simply give them the 2. Link from below instead)
In case this link stops working you are allowed to host unmodified versions till it works again.

You may make changes to it for yourself.
If you want to make them available to others you can only do so through me.
(For now at least, given a serious contributor asks it I will consider a copyleft licence e.g. GPL.
But I see no reason to read, understand and choose them right now)

No warranty is given at all. Not even implied ones. If you want to be sure it does not eat you cat
you will have to read the python files yourself.[/list]

Links:
Download wbfs2fat
Homepage/Discussion Thread
Detailed Tutorial/Howto
GUI demo video

If you have questions/suggestions/comments just ask.
Especially patches are welcome.

 

[kylster]

could you post a windows copy? i would like to try it but alas im not linux friendly LoL

 

[PsyBlade]

no sorry no windows version from me
I'm not opposed to include patches for windows
but I dont know how to do it

I'm invoke mkfs.vfat, fsck.vfat, mount, umount & sudo and dont know a windows alternative for any of them. Plus I dont use windows

 

[kylster]

Well that sucks

tried running ubuntu from cd but all i get is a install program type screen. I cannot actually do anything on linux using the cd boot

 

[PsyBlade]

at the moment you will need knowledge of linux and python to make use of this
this release is intended for review not for actual use

but a live cd/usb should do fine running it - no need to install anything

 

[ManFranceGermany]

can't u run this very easily with a python lib for windows?

well, if you can't here a list with Live CDs (Damn Small Linux and NimbleX are nice): http://www.livecdlist.com/

 

[PsyBlade]

the linux dependency is only really basic stuff, any distri should be able to do it
any other decent OS should too, but I dont use anything else

I dont use any fancy python libs too afaik

I think I need only the the following 4 lines ported for Windows support:
mkfs.vfat -I -f1 -F32 -R32 -s -l
fsck.vfat -a
mount umask=000
umount

python code or program calls will both do

 

[PsyBlade]

Is there any interest in this?
While I do code for fun, Im not going to do the more boring stuff like usage guide, code documentation, cleanup, UI and such if no one will ever bother to take a look at it.

 

[Rydian]

I'm sorry if you feel discouraged. I asked a few people if they'd know anything about porting it, but the people I asked aren't familiar enough with the concepts to do it easily, or have bigger projects they're working on.

 

[XFlak]

I wouldn't worry about porting this to Windows, Windows users already have access to WiiBackupManager for Windows, which can do exactly what this app does and more. This app might be faster than WBM when transferring\converting games from a WBFS partition, I'm not sure, I've never tested the speed of that function of WBM b4.

My point is, I'm sure people who run linux would find this app useful cuz they can't use WBM (which only works on Windows), but for this app to be truly useful it should support converting games to\from cISO, ISO, WBFS partitions and WBFS files (and perhaps discex extracted format for s/uneek+di). From what I gather, in it's current form this app will only be useful for people running linux and would like to switch from a WBFS formatted HDD to FAT32... but these days most people don't use WBFS partitions anymore so u probably won't see a lot of demand for this app in it's current form.

Don't be discouraged, you're working on this and sharing it out of the kindness of your heart, as long as it doesn't corrupt ppls data (and I'm not saying it does) then you can do no wrong.

 

[PsyBlade]

Yes the scope of this program is quite limited.
It converts a WBFS partition into a FAT32 partition while keeping the games and using very little space (some megs ram +

 

[DarkMatterCore]

You did. WBM doesn't have that function, AFAIK (I believe XFlak misundertood your description and thought this program is for converting Wii game image files to the WBFS format).

In any case, good luck trying to port this to Windows.

 

[PsyBlade]

just upped a windows version
see first post

I reimplemented all required functionality in python
but you will need cygwin because I dont know how to open partitions using the windows api
I would be glad if someone fixed this

plus its much faster now

 

[oggzee]

So this thing creates the FAT filesystem in place and links the files to the data without moving any data?
If I understood this correctly then you've done an amazing work, good job!

Or even if it works differently it's still amazing

What is backed up? Only the first couple of sectors or everything?

 

[PsyBlade]

So this thing creates the FAT filesystem in place and links the files to the data without moving any data?
If I understood this correctly then you've done an amazing work, good job!

Or even if it works differently it's still amazing

What is backed up? Only the first couple of sectors or everything?

Yes, essentially it keeps the data in place and creates fitting metadata
For this I need some additional space in the beginning of the drive so I have to move some blocks around
The original state of this start area is backed up (6M on a 50G drive)
The backup is not needed, its only for disaster recovery and to allow a quick reset for testing
Skipping it is not implemented but can done easily if space is really tight

 

[XFlak]

So if I'm understanding correctly now... this will allow users with a wbfs partition to switch to using a FAT32 partition (and wbfs files) without needing to first backup the games on the wbfs partition to another drive, reformat the wbfs partition as FAT32 and copy the games back? Instead this app can take a wbfs partition and directly convert it to FAT32 while retaining all the games\data that was on the wbfs partition to begin with... right?

If I understand this correctly... that's pretty sweet and has never been done before.

 

[Cyan]

I find this tool interesting, but I'm a little afraid of using it and loosing my partition, or corrupting the other partitions. (I have a dual FAT32 + WBFS).
And, my wbfs partition is working fine, I like the "don't fix it if it's not broken".

I'll wait until some users report their result.
But thank you for all your work on making this tool, it was asked a lot of time.
It's just too bad it's a little late in the scene, many users would have use it when FAT32 became possible.

There's certainly a lot of users like me with their WBFS partitions wishing to convert it.

 

[PsyBlade]

@XFlak: yes you do understand correctly

@Cyan:
Loosing another partition is near impossible; I only open a partition, not the entire drive.
The operating system then enforces it boundaries, even if I try to access outside of it.
You will just end up with two fat32 partitions, but other tools might join them.
I remember "Partition Magic" from over 10 years ago.
But loosing the partition is a danger yes. but Im pretty confident the backup feature will save you.

Yes the tool is a bit late to the party, but I switched to fat32 almost immediately and have no need


I and potential users would be grateful for any test results.
A test does not need to be on live data, it can be done on partitions created only for the test.
If you already have Python or Cygwin installed it will take little time.
A test on 4KiB sector drives would be really great, since I don’t know how to emulate or have them.

 

[Gabelvampir]

(Full disclore: I'm a friend of the author)

I just tried the program and it worked for me. But since I use FAT32 for my Wii USB HDD for some time now I used a wbfs on a 2GB USB stick to test it with only 3 games on it.
So it is not really a realistic case, but it worked and should work in most real cases as far as I understand it. I also tried the recovery via dd and that worked too.
I mostly ignored his instructions a I know my way around a Unix system, although it was my first time using CygWin.
A little bit of advice: 1) If you have Windows 7 or Vista and have the UAC activated run CygWin a an administrator (right click->run as adminstrator). It is not really necessary but some CygWin device handling stuff is a bit weird if you don't do that and the dd recovery does not work.
2) Give program the optional Windows mount point for your device. Saves a bit of file renaming, but will probably want to use some manage to give the file human readable names and/or directories. But if you let the program rename the files a bit they at least work in a USB loader, they don't if you do nothing as they are not in the right dir and do not have any supported name scheme.

Stuff my test case did not account for:
1) Devices full to the brim with images.
2) Splitted images (for games over 4GB).
3) Devices with 4 kB sectors.
4) probably some other stuff I don't think of right now.

 

[nunavat2010]

I'll try this tonight, I use ubuntu 11.04, have dual layer games, will report result. Just one question, you can't go back to wbfs format just like the XP tool to convert Fat to NTFS?

One more thing. Is there an easy way to add *.iso to fat partition, if they are bigger than the file size limit? Right now my main USB disk is ntfs (not /ext1/ etc) formatted. So when I have several iso files in it I use

wwt add *.iso --part /dev/sdb2

where /dev/sdb2 is my wbfs partition. What single line command would I use in ubuntu after converting it to fat? Sorry, If I am deviating from the main point.