Hacking wwt+wit: Wiimms WBFS+ISO Tools

[W hat]

I love your updates to your already awesome program Wiimm. I understand it will take time. How much time? Can I expect a release by the end of 2010?

 

[Wiimm]

I love your updates to your already awesome program Wiimm. I understand it will take time. How much time? Can I expect a release by the end of 2010?

I speak about some weeks/month. A big problems are some disquieter like business, FIFA WM and hollydays; all of them are stealing my time.

 

[giantpune]

at least 2 of us homebrew devs have a birthday coming up next month. and i was just saying how this is what i wanted for my present. i think he was, too. well, it was between this and a pony, and i think this will be better than a pony.

so, i look forward to it

 

[W hat]

Thanks for your answer Wiimm. I knew you'd be faster than 6 months, but how much faster I was not sure.

Edit: Either Wiimm or Giantpune knows the answer to this one I'm sure.

What exactly is in an update partition? My understanding is it includes the newest system update (which also includes the IOSs to run the new system menu). It also has any new IOSs / IOS updates that the developer thinks you need to play the game. Games like Mario Kart and Wii Fit have channels as part of the update partition, which is why you can't install the channels from a USB Loader. I know games like Your Shape have a reason for new IOSs / IOS updates, but do most games contain a new IOS / IOS updates? Gecko OS only installs updates from the disc that are needed from the game, and only works with a retail game or modchip.

Did I get all that right?

 

[Wiimm]

WIT: Wiimms ISO Tools v1.01c - 2010-07-01
A command line WBFS tool set for various os.

WIT (Wiimms ISO Tools) is a set of command line tools to manipulate Wii ISO images and WBFS containers. The two main tools are called wit (Wiimms ISO Tool) and wwt (Wiimms WBFS Tool, the WBFS manager).

Feature overview:

• WIT (Wiimms ISO Toolset) is a set of command line tools.
• wit (Wiimms ISO tool) is the main Wii ISO manipulation tool.
• wwt (Wiimms WBFS tool) is a complete WBFS manager including FS checking and repairing, time stamps and multi WBFS support.
• wdf-cat and wdf-dump are WDF and CISO support tools.
• WIT supports 5 Wii disc file types (ISO, CISO, WDF, WBFS and FST) and can read from pipes (indirect RAR/ZIP support).
• wit can verify and compare (diff) Wii ISO images.
• wit can extract and compose Wii ISO images.
• wit and wwt support many diagnostic commands.
• Advanced shell support.

Visit http://wit.wiimm.de/features.html for more details.

Download of the current version:

• wit-v1.01c-r1313-i386.tar.gz : linux/i386 version
• wit-v1.01c-r1313-x86_64.tar.gz : linux/x86_64 version
• wit-v1.01c-r1313-mac.tar.gz : Mac OS universal binaries (ppc, i386 and x86-64)
• wit-v1.01c-r1313-cygwin.zip : cygwin (win32) version
• Other releases are available from http://wit.wiimm.de/download.html.

There are also some GUI available:

• WiiJManager Multiplatform, a WBFS manager GUI
  A java GUI that acts like a WBFS manager. Converting of ISO images is also possible.
  .
• qtwitgui, manipulate and convert single ISO images.
  This QT based GUI focus on manipulating single ISO images.

Change log:

wit v1.01c r1313 - 2010-07-01

- Bug fix for "wit COPY": If converting an ISO to a plain iso file (*.iso) in
ÂÂ scrubbing mode the destination file was truncated to the needed size. Now
ÂÂ it is extended to 4 699 979 776 bytes again.

Missing docu:

- wit commands ILIST + EXTRACT
- Option --show

The source is published under the licence of GPL2. Checkout the sources directly from the SVN repository or browse the source.
Visit http://wit.wiimm.de/ for more information.

 

[giantpune]

@w hat
the game developers most likely dont know a single thing about the update partition. they do not design it or have anything to do with adding it to the game. this is all added by nintendo. think of it like this...

a game developer purchases the SDK from nintendo to make a wii game. nintendo sells them "SDK v58" or whatever they are calling it. every game made with this SDK will use IOS58. once this game developer has made a game and sent it off to nintendo to get turned into a DVD, nintendo wants to make sure that when the end user gets it, they will have the needed firmware ( IOS ) to play this game or else it wont work.

they COULD simply put IOS58 on this partition and it would work. but, as is common practice for many companies, they dont want you to update only part of the firmware. they take this opportunity to update all the firmwares used by other games ( other IOSes ), the system menu, the boring channels, etc.

so, the update partition contains wads for all the stuff you would get via a network update as well as a .inf file detailing its contents.



you could actually use this update process for other things if you want. it can be used to trigger the wad installer from the system menu and use it to install other wads ( assuming there is a signing bug present in the proper IOS ). IE, you can make a update partition containing all of cIOScrap, slap it on a disc, run the system menu with a cIOS and it will do it all the work for you.


the channel installers are not on the update partition. they are a completely different partition. i dont have any games with channel installer partitions, so i cant say for sure. USB loaders dont support switching partitions. switching a partition effectively makes a game into a multi-dol game which dont work.

 

[Wiimm]

The general disc layout is to start the update partition at 0x50000 and the data partition at 0xf800000. The channel partition follows the data partition. All three partition types uses the same data structure with ticket+tmd+cert+h3+data.

The Mario Kart DVD has all three partitions types.

 

[giantpune]

i found a little cosmetic bug in the output from "DUMP ... --show=map" it treats the ID as a char string. but update partitions start with a null byte in the ID "00555045 = .UPE". this makes the output line

ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ:ÂÂÂÂÂÂ50000 ..ÂÂÂÂÂÂ502c0 :ÂÂÂÂÂÂ 2c0 : P.0.0, UPDATE 1, header+ticket, id=

i say file this one in the ultra-low priority bugs

 

[Wiimm]

i found a little cosmetic bug in the output from "DUMP ... --show=map" it treats the ID as a char string. but update partitions start with a null byte in the ID "00555045 = .UPE". this makes the output line

ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ:ÂÂÂÂÂÂ50000 ..ÂÂÂÂÂÂ502c0 :ÂÂÂÂÂÂ 2c0 : P.0.0, UPDATE 1, header+ticket, id=

i say file this one in the ultra-low priority bugs

Already fixed a week ago, but only in my private branch with the new wiidisc lib.
And please bother me again.

BTW: Rewriting the wiidisc lib *was* easier than I had thought. The lib is ready and at the moment I will change the the other code to use the new interface.


EDIT

And here is a mem map of the newest "wit DUMP". Holes within partitions are also recognized.
CODE# ./wit dump pool/wdf/wpl.wdfÂÂ--show dmap

Dump of file pool/wdf/wpl.wdf

ISO Memory Map:

ÂÂÂÂÂÂ unused :ÂÂ off(beg) ..ÂÂ off(end) :ÂÂÂÂÂÂsize : info
------------------------------------------------------------------------------------------
ÂÂÂÂÂÂÂÂÂÂÂÂÂÂ:ÂÂÂÂÂÂÂÂÂÂ0 ..ÂÂÂÂÂÂÂÂ100 :ÂÂÂÂÂÂ 100 : Header, magic=5d-1c-9e-a3, id=RHAP01
ÂÂÂÂÂÂÂÂ3ff00 :ÂÂÂÂÂÂ40000 ..ÂÂÂÂÂÂ40020 :ÂÂÂÂÂÂÂÂ20 : Partition address table
ÂÂÂÂÂÂÂÂÂÂÂÂÂÂ:ÂÂÂÂÂÂ40020 ..ÂÂÂÂÂÂ40030 :ÂÂÂÂÂÂÂÂ10 : Partition table #0 with 2 partitions
ÂÂÂÂÂÂÂÂ dfd0 :ÂÂÂÂÂÂ4e000 ..ÂÂÂÂÂÂ4e020 :ÂÂÂÂÂÂÂÂ20 : Region settings, region=0
ÂÂÂÂÂÂÂÂ 1fdc :ÂÂÂÂÂÂ4fffc ..ÂÂÂÂÂÂ50000 :ÂÂÂÂÂÂÂÂ 4 : Magic2: c3-f8-1a-8e
ÂÂÂÂÂÂÂÂÂÂÂÂÂÂ:ÂÂÂÂÂÂ50000 ..ÂÂÂÂÂÂ502c0 :ÂÂÂÂÂÂ 2c0 : P.0.0 UPDATE: ticket, id=.UPP
ÂÂÂÂÂÂÂÂÂÂÂÂÂÂ:ÂÂÂÂÂÂ502c0 ..ÂÂÂÂÂÂ504c8 :ÂÂÂÂÂÂ 208 : P.0.0 UPDATE: tmd, ios=4, id=.UPP
ÂÂÂÂÂÂÂÂÂÂ 18 :ÂÂÂÂÂÂ504e0 ..ÂÂÂÂÂÂ50ee0 :ÂÂÂÂÂÂ a00 : P.0.0 UPDATE: cert
ÂÂÂÂÂÂÂÂ 7120 :ÂÂÂÂÂÂ58000 ..ÂÂÂÂÂÂ70000 :ÂÂÂÂ 18000 : P.0.0 UPDATE: h3
ÂÂÂÂÂÂÂÂÂÂÂÂÂÂ:ÂÂÂÂÂÂ70000 ..ÂÂÂÂ4870000 :ÂÂ 4800000 : P.0.0 UPDATE: data+fst, N(fst)=18
ÂÂÂÂÂÂaf90000 :ÂÂÂÂf800000 ..ÂÂÂÂf8002c0 :ÂÂÂÂÂÂ 2c0 : P.0.1 DATA: ticket, id=RHAP
ÂÂÂÂÂÂÂÂÂÂÂÂÂÂ:ÂÂÂÂf8002c0 ..ÂÂÂÂf8004c8 :ÂÂÂÂÂÂ 208 : P.0.1 DATA: tmd, ios=21, id=RHAP
ÂÂÂÂÂÂÂÂÂÂ 18 :ÂÂÂÂf8004e0 ..ÂÂÂÂf800ee0 :ÂÂÂÂÂÂ a00 : P.0.1 DATA: cert
ÂÂÂÂÂÂÂÂ 7120 :ÂÂÂÂf808000 ..ÂÂÂÂf820000 :ÂÂÂÂ 18000 : P.0.1 DATA: h3
ÂÂÂÂÂÂÂÂÂÂÂÂÂÂ:ÂÂÂÂf820000 ..ÂÂÂÂfbf8000 :ÂÂÂÂ3d8000 : P.0.1 DATA: data+fst, N(fst)=259
ÂÂÂÂ101ab8000 :ÂÂ1116b0000 ..ÂÂ1173c0000 :ÂÂ 5d10000 : P.0.1 DATA: data+fst
ÂÂÂÂÂÂ e80000 :ÂÂ118240000 ..ÂÂ118240000 :ÂÂÂÂÂÂÂÂ 0 : -- End of file/disc --

 

[W hat]

My plans: I will rewrite 'wiidisc.c' (part of libwbfs) to improve handling of the disc
QUOTE(Wiimm @ Jul 2 2010, 05:13 AM) BTW: Rewriting the wiidisc lib *was* easier than I had thought. The lib is ready and at the moment I will change the the other code to use the new interface.

A quick Google search shows these are the same thing. Good job Wiimm!

 

[soulfoot]

Edit: Was able to find it by running the command prompt as administrator, however i was only able to "fix" 83 out of 200 games.



I am currently reading through this thread to find the answer but I cant seem to make wwt see my WBFS drive.

I am using your program to hopefully restore a trashed drive.

WBFS manager can see the drive and the games. My wii can See the drive but not the games. One of the Gui in the OP can see the drive but not the games.

but the command prompt wwt list -a returns no drive found

does it matter where i install this?

 

[Wiimm]

@soulfoot
what does "wwt -a dump" say?
And what "wwt analyze"?

 

[soulfoot]

Dump:

WBFS Header:
WBFS-Magic: 57 42 46 53 = 'WBFS'
WBFS Version: 0x1 = 1
HD Sectors: 0x71868570 = 1904641392
HD Sector size: 0x200 = 512 = 2^9
WBFS sector size: 0x1000000 = 16777216 = 2^24


Disc table :
0..49: all empty
50..99: all empty
100..149 : ................................. ##############################
150..199 : ##############################################
200..249 : ###


(an explanation: I used hex editor to try and "turn on" these sections by typing 01 in them. I have about 89 out of 203 games and they are the last 89 i installed which is in agreement with what the Disc Table is showing (the first 114 games being empty). Another interesting note is thats about how many games i cloned over from my previous usb HD)


and

Analyze /dev/sde1
-------------------------
---------

Name Magic vrs num size num size num size aditional info
header ok 1 71868570 200 e30d 1000000 1f4 600 wbfs header scanning
inode-tim: ok 1 71868570 200 e30d 1000000 1f4 600 2010-07-04 20:37:21 n=500
discs: - 0 71868570 200 e30d 1000000 1f4 600 47 disc header found
calc 512 - 1 71868570 200 e30d 1000000 1f4 600 calculation of init fuction
calc 1024 - 1 38c342b8 400 e30d 1000000 3f4 800 " but sector size =1024
calc 2048 - 1 1c61a15c 800 e30d 1000000 7f4 800 " but sector size = 2048
calc 4096 - 1 e30d0ae 1000 e30d 1000000 ff4 1000 " but sector size = 4096


not sure if any of that will help, again, the games that "seem" to be missing are mostly the ones that were cloned over from another hard drive using WBFS, not sure if that is the issue or why my HD only remembers what "seems" to be the latest 80 games or so that were installed.


anyways my current plan after waiting a week or so to see what resources i can exhaust getting them back is to add all of these games to a 3rd WBFS partition then to add them to my backup. Then totally reformat this HD and then take a look at all the different formatting options before deciding which way to go and moving on. Your utitlity is awesome and i would say i'm 90-100% recovered thanks to it and my backups and i thank you for that, but if you have any other insight i can hex edit the HD further to see what we can come up with.

 

[Pazznoz]

Hi Wiimm,

First of all thanks for your awesome tool, currently i am writing some forward iso creator program that uses it for the iso building.

As there are no rar compressing libraries available i am sort of stuck with the creation of 256 meg iso's. I don't want to instruct the user to save the files in a compressed NTFS folder and zipping the files makes the program incompatible with WBFSManager. CISO is also not supported by WBFSManager, so that's not an option too.



The process takes shape in the following order :

extract iso (8 meg forwarder iso i got somewhere) using wit.exe
replace opening.bnr
create main.dol forwarder from user input
build iso (which results in the 256 meg file) using wit.exe


Now my question is : How to create a smaller output file?


Any suggestion will do.

Pazznoz

 

[giantpune]

supply with your program a folder with the base files. this will probably be like 5MB.

then when the user creates a forwarder ISO, you can simply copy the base files, replace the dol & bnr, then use wit or wwt to directly add the game to their HDD, or save a game in any format you want. if you save a .wbfs or .ISO, it will ( usually ) still take up about the same amount of space on the PC. on filesystems that support it, ISOs are created as sparse files.

 

[Pazznoz]

Thanks for the reply.

While waiting for it i stumbled upon the file attribute 'compress' so the problem was solved half already.
Isn't there any way to let wit create smaller isos?


Haven't even thought about adding the iso to the drive yet, will look into that when i have some spare time. Don't want to ****up my drive testing so i'll have to find a solution for that (spare usb thingy or something like that).


Thanks, pazznoz

 

[Wiimm]

wwt knows the option --trunc to truncate PLAIN ISO images to the needed size. Perhaps I implement it in wit. I plan also a --trim option to produce small images; later.

There is a way to set the location of the partition. Get the beta test version (see wit.wiimm.de), go into the partition dir (dir of tmd.bin) and create the file "setup.txt":

part-offset = 0

This will overwrite the default position for data partitions (0xf800000). It is only a hint: "Set the beginning of this partition is not before #value#". wit will never overlap data. So "part-offset = 0x50000" does the same.

 

[Wiimm]

WIT: Wiimms ISO Tools v1.10a - 2010-07-15
A command line WBFS tool set for various os.

WIT (Wiimms ISO Tools) is a set of command line tools to manipulate Wii ISO images and WBFS containers. The two main tools are called wit (Wiimms ISO Tool) and wwt (Wiimms WBFS Tool, the WBFS manager).

Feature overview:

• WIT (Wiimms ISO Toolset) is a set of command line tools.
• wit (Wiimms ISO tool) is the main Wii ISO manipulation tool.
• wwt (Wiimms WBFS tool) is a complete WBFS manager including FS checking and repairing, time stamps and multi WBFS support.
• wdf-cat and wdf-dump are WDF and CISO support tools.
• WIT supports 5 Wii disc file types (ISO, CISO, WDF, WBFS and FST) and can read from pipes (indirect RAR/ZIP support).
• wit can verify and compare (diff) Wii ISO images.
• wit can extract and compose Wii ISO images.
• wit and wwt support many diagnostic commands.
• Advanced shell support.

Visit http://wit.wiimm.de/features.html for more details.

Download of the current version:

• wit-v1.10a-r1386-i386.tar.gz : linux/i386 version
• wit-v1.10a-r1386-x86_64.tar.gz : linux/x86_64 version
• wit-v1.10a-r1386-mac.tar.gz : Mac OS universal binaries (ppc, i386 and x86-64)
• wit-v1.10a-r1386-cygwin.zip : cygwin (win32) version
• Other releases are available from http://wit.wiimm.de/download.html.

There are also some GUI available:

• WiiJManager Multiplatform, a WBFS manager GUI
  A java GUI that acts like a WBFS manager. Converting of ISO images is also possible.
  .
• QtWitGui, manipulate and convert single ISO images.
  This QT based GUI focuses on manipulating single ISO images.

Highlights of this release:

• I have rewritten the wiidisc library (part of libwbfs) from scratch. It has a complete new interface and an improved functionality. It uses dynamic data structures and collect data as needed. This has considerable advantages for the whole WIT project: All information about a Wii disc is now in one hand. Things like usage maps and partition data are loaded/built once and only if needed.
  
  This new library facilitate the planed patching functionality.
  .

Change log:

Aggregation of the previous beta versions:

- New option: --width wd: Define the width (number of columns) for help and
ÂÂ other messages and disables the automatic detection of the terminal width.
- I have rewritten the wiidisc lib (part of libwbfs) from scratch. It has a
ÂÂ complete new interface and an improved functionality. It uses dynamic data
ÂÂ structures and collect data as needed. This has considerable advantages
ÂÂ for the whole WIT project.
- ID renaming: A point ('.') means: leave id character unchanged.
- Options --ios, --id and --name modify only disc data and the DATA partition,
ÂÂ but no longer other any other partitions.
- There is an complete new implementation of option --psel: Combinations like
ÂÂ '--psel DATA,UPDATE' or '--psel -CHANNEL' are supported now.
- wwt EXTRACT: Complete new and improved calculation of output filename.
- wwt EXTRACT: Support '%X' and '%Y' in output filenames.
- Option --files: Macro calls are now prefixed by a colon (':'). The previous
ÂÂ prefix '=' is obsolete, but still allowed.
- Bug fix: File size calculation error for when composing large discs.

wit v1.10a r1386 - 2010-07-15

- wittdb.com: Support of new languages: ru, zhcn, zhtw
- wwt ANALYZE: If option --long is set then calculated values are printed too
ÂÂ if other values are available. If option --long is set twice calculated
ÂÂ values are always printed.
- wit+wwt: Option --trunc: Truncate created PLAIN ISO image to needed size.

Missing docu:

- wit commands ILIST + EXTRACT
- Option --show => see built in help

The source is published under the license of GPL2. Checkout the sources directly from the SVN repository or browse the source. Visit http://wit.wiimm.de/ for more information.

 

[msaraiva]

Hi, i'm having a hard time with wit stdin support:

unrar p -inul Wii_Party_JAP_WII-TMD.rar *.iso | wit -v -P COPY --psel=data --wbfs --dest /media/Wii\ Games/ -

wit: Wiimms ISO Tool v1.10a r1386 x86_64 - Dirk Clemens - 2010-07-15

* wit SCRUB 1/1 ISO:- (stdin) -> WBFS:/media/Wii Games/SUPJ01.wbfs
!! wit: ERROR #13 [INVALID WII DISC] in wd_load_part() @ src/libwbfs/wiidisc.c#1080
!! Position of partition '1 [UPDATE]' exceeds file size: - (stdin)
!! wit: ERROR #13 [INVALID WII DISC] in wd_load_part() @ src/libwbfs/wiidisc.c#1080
!! Position of partition '0 [DATA]' exceeds file size: - (stdin)
256 MiB copied in 0:09, 26.2 MiB/sec
!! wit: ERROR #26 [WBFS ERROR] in CopyToWBFS() @ src/lib-sf.c#2324
!! Error while creating disc [] @/media/Wii Games/.SUPJ01.wbfs.Rn0l1F
wit: Command 'COPY' returns with status #26 [WBFS ERROR]


Is there anything wrong? Verify works just 'fine':


unrar p -inul Wii_Party_JAP_WII-TMD.rar *.iso | wit -v -P VERIFY --psel=data -
wit: Wiimms ISO Tool v1.10a r1386 x86_64 - Dirk Clemens - 2010-07-15

!! wit: ERROR #13 [INVALID WII DISC] in wd_load_part() @ src/libwbfs/wiidisc.c#1092
!! Invalid header in partition '1 [UPDATE]': - (stdin)
+OK .1 DATA SUPJ01 - (stdin)

 

[Wiimm]

@msaraiva
"exceeds file size" should not happen if reading pipes because the file size is unknown. I put this on my bug list (delayed until the next feature is running)