Hacking wwt+wit: Wiimms WBFS+ISO Tools

[PsyBlade]

sounds like alignment to me somehow

 

[Wiimm]

sounds like alignment to me somehow

... in combination with the disc order. All brstm files are stored at the beginning (small offset) with an alignment of 0x20. And the freeze comes with an bad sound.

EDIT:
And some *.dat files are aligned to 32KiB. For the next test I try --align=0x20 and --align=32K
(good to have a nice tool that support such things )

 

[Wiimm]

So, you mean that when we repack the game, all files are sorted in a new order, and this cause the repacked game broken? Is there any other reason?

In theory, if we didn't replace any file from the extracted game, just keep all the extracted files original, then the repacked new game must be absolutely equal with the original game.

I think you could record the file order or some other information not very clear from original game into a log file during the unpack process, and read the log file to rebuild game when repacking it, so that it can be "absolutely equal" with the original game.


"Super Paper Mario" is a very very strange game, upacking/repacking will cause it broken, and sometimes I replace some file(like *.bfn font file, *.txt message file) with WiiScrubber tools but not wit.exe, it will also be broken on Wii, but it works well on Dolphin... oh my god, I didn't know what happened " border="0" alt="frown.gif" />

I have changed the sorting algorithm and now the files are in identical order as the original. But the game still freeze at the same point. There must be something special with this game.

I have solved this problem. Beginning with the next release, my tools create for each extracted partition a file named "align-files.txt" (see below). It contains a little alignment analysis for each group of files (group= all files of a dir with the same extension). If the new option --align-files is set on composing, this file is used for aligning.

I have tested it with Super Paper Mario and it works.

Spoiler: File "align-files.txt" of Super Paper Mario

0x4 = files/*.bin
0x4 = files/*.bnr
0x20 = files/*.tpl
0x8 = files/a/*.bin
0x8000 = files/eff/*.dat
0x20 = files/eff/*.tpl
0x8000 = files/eff/fr/*.dat
0x20 = files/eff/fr/*.tpl
0x8000 = files/eff/ge/*.dat
0x20 = files/eff/ge/*.tpl
0x8000 = files/eff/it/*.dat
0x20 = files/eff/it/*.tpl
0x8000 = files/eff/jp/*.dat
0x20 = files/eff/jp/*.tpl
0x8000 = files/eff/nl/*.dat
0x20 = files/eff/nl/*.tpl
0x8000 = files/eff/sp/*.dat
0x20 = files/eff/sp/*.tpl
0x8000 = files/eff/uk/*.dat
0x20 = files/eff/uk/*.tpl
0x8000 = files/eff/us/*.dat
0x20 = files/eff/us/*.tpl
0x20 = files/font/*.bfn
0x4 = files/hbm/*.bin
0x4 = files/lyt/*.fr
0x20 = files/lyt/*.ge
0x4 = files/lyt/*.it
0x8 = files/lyt/*.jp
0x8 = files/lyt/*.sp
0x20 = files/lyt/*.uk
0x8 = files/lyt/*.us
0x4 = files/map/*.bin
0x4 = files/msg/FR/*.txt
0x4 = files/msg/GE/*.txt
0x4 = files/msg/IT/*.txt
0x4 = files/msg/JP/*.txt
0x4 = files/msg/SP/*.txt
0x4 = files/msg/UK/*.txt
0x4 = files/msg/US/*.txt
0x4 = files/rel/*.bin
0x8000 = files/setup/*.dat
0x40 = files/sound/*.brsar
0x20 = files/sound/*.brstm
0x10000 = files/sound/*.dat
0x20 = files/tizu/*.tpl
0x8000 = files/win/*.dat
0x20 = files/win/*.tpl
0x20 = files/win/fr/*.tpl
0x20 = files/win/ge/*.tpl
0x20 = files/win/it/*.tpl
0x20 = files/win/jp/*.tpl
0x20 = files/win/sp/*.tpl
0x20 = files/win/uk/*.tpl
0x20 = files/win/us/*.tpl

 

[W hat]

I have solved this problem. Beginning with the next release, my tools create for each extracted partition a file named "align-files.txt" (see below). It contains a little alignment analysis for each group of files (group= all files of a dir with the same extension). If the new option --align-files is set on composing, this file is used for aligning.

I have tested it with Super Paper Mario and it works.

Provided this doesn't create new bugs, shouldn't this be enabled by default? Otherwise, there's the possibility of creating dumps that aren't perfect, or even worse, don't function.

I recall the Metroid Prime GameCube games having file alignment issues as well, for example.

 

[Wiimm]

That are my plans:
In a first phase we have an opt in. But for each extracted fst, the info is collected. Later I'll enable this kind of aligning by default and negate the option. Some images will grow a little bit, because some files may randomly aligned. On the other hand, the algorithm to find the align values is very simple and may not working for all images.

btw:
This kind of aligning is totally rubbish.

A Wii sector of 32 KiB contains only 31 KiB data, the first 1 KiB are hash values.

This means, that a virtual block with offset 0x8000 and size 0x8000 starts real at 0x8800 and ends at 0x10c00 with an hole in the middle. If ever, a aligning to max 0x400 make sense on a physical kind of view. I think Nin used some old programs which loads data by a block number.

 

[XFlak]

It amazes how much knowledge you have on all this stuff. You don't get enough credit\recognition, props to you man, you're one of the amazing unsung heroes of the wii scene

 

[Wiimm]

WIT: Wiimms ISO Tools v2.02a - 2011-11-25
A command line ISO+WBFS tool set for various os.

WIT (Wiimms ISO Tools) is a set of command line tools to manipulate Wii ISO images and WBFS containers. The two main tools are called wit (Wiimms ISO Tool) and wwt (Wiimms WBFS Tool, the WBFS manager).

Feature overview:

• WIT (Wiimms ISO Toolset) is a set of command line tools.
• wit (Wiimms ISO tool) is the main Wii and GameCube ISO manipulation tool.
• wwt (Wiimms WBFS tool) is a complete WBFS manager including FS checking and repairing, time stamps and multi WBFS support.
• wdf (Wiimms WDF Tool) is a little tool to manage WDF, CISO and WIA files.
• wfuse (Wiimms FUSE Tool) mounts Wii and GameCube images and WBFS files and partitions using FUSE.
• WIT supports 6 Wii disc file types (ISO, CISO, WDF, WIA, WBFS and FST) and can read from pipes (indirect RAR/ZIP/7Z support).
• wit can verify and compare (diff) Wii and GameCube ISO images.
• wit can extract and compose Wii ISO images.
• wit can patch Wii and GameCube ISO images while copying.
• wit can mix partitions from different sources into a new Wii ISO images.
• wit and wwt support many diagnostic commands.
• Advanced shell support.

Visit http://wit.wiimm.de/features.html for more details.

Download of the current version:

• wit-v2.02a-r3240-i386.tar.gz : Linux/i386 version
• wit-v2.02a-r3240-x86_64.tar.gz : Linux/x86_64 version
• wit-v2.02a-r3240-mac.tar.gz : Mac OS universal binaries (ppc, i386 and x86-64)
• wit-v2.02a-r3240-cygwin.zip : Cygwin (Windows) version
• Other releases are available at http://wit.wiimm.de/download.html.

There are also some GUI available:

• QtWitGui, manipulate and convert single ISO images.
  This QT based GUI focuses on manipulating single ISO images. It can do much more.
  
• Wii Backup Fusion, a WBFS and ISO manager.
  This QT based GUI is a complete WBFS manager and ISO tool.
  
• Wii Covers Download & WBFS Manager for Windows, a WBFS and ISO manager.
  This is a very young Java based WBFS manager. (GBA temp discussion)

Change log:

wit v2.02a r3240 - 2011-11-25

- The default output filename for WBFS files changed from "ID6.wbfs" ("%+")
to "TITLE [ID6].wbfs" ("%X").
- The file order for composed files is now Nintendo like.
- If extracting an image, a new files called 'align-files.txt' is created. It
stores information about file alligning. If creating a new image and option
--align-files is set, this file is read to align files, that have a previous
aligning of >=0x8000 (Wii sector size). This fixes the bug for "Super Paper
Mario" and perhaps for some other games.
- wit FILES: If the file list is sorted by offset, than an 'unused' column is
added before the 'offset' column.
- wfuse for Mac is back again. It was lost because non installed FUSE package.


Known bugs:

- There are new problems while reading from non seekable sources (e.g. pipes).
The reason is the complete new and more intelligent wiidisc lib which makes
more lookups ==> The best is to use version 1.01c for pipes.

The source is published under the license of GPL2. Checkout the sources directly from the SVN repository or browse the source. Visit http://wit.wiimm.de/ for more information.

 

[Burton]

Thanks..I'll be playing for a while with this!

 

[ice1006]

Hi, wiimm, I'm very glad to see you updated this tool!

It seems that you sort the files in "Nintendo like" order in this version while repacking, but there are still some unknow bugs in this version.

Serveral months ago I reported a bug to you: when we extract the Super Paper Mario(R8PJ01.ISO, JAP region) to a folder, then we DO Nothing, DO NOT REPLACE ANY FILE, DO NOT MODIFY ANY FILE, just repack this folder to a new ISO immediatly, the result is that the Old ISO works well on CFG USB Loader, but the New ISO will crash before enter game.

I compared the two files(Old ISO and New ISO) with Beyond Compare in hex mode, and found they are not 100% same. But in fact we DID NOT replace any file and sort the files in the original "Nintendo like" order this time, so it seems that the Old ISO and New ISO should be 100% same, but why they not? Can you let them 100% same? because it is the ony way to make sure there are no errors while extracting(unpacking) and repacking.

 

[ice1006]

I have solved this problem. Beginning with the next release, my tools create for each extracted partition a file named "align-files.txt" (see below). It contains a little alignment analysis for each group of files (group= all files of a dir with the same extension). If the new option --align-files is set on composing, this file is used for aligning.

I have tested it with Super Paper Mario and it works.

Provided this doesn't create new bugs, shouldn't this be enabled by default? Otherwise, there's the possibility of creating dumps that aren't perfect, or even worse, don't function.

I recall the Metroid Prime GameCube games having file alignment issues as well, for example.

Oh, I got your message but It seems that it did not work with R8PJ01(Paper Mario, JAP) I tested it just now, old iso works well, new iso will crash...

I think there is a best way: while unpacking, extract the files and all the other extra data and store thier orders in a log; while repacking, just read the log and compose files and extra data to a new iso —— it will make sure the two ISO 100% same without any errors.

 

[Wiimm]

Have you extracted the image again (a new align file is created) and than on composing used the option --align-files ?
For the PAL version it works, but you must redo both steps.

 

[ice1006]

sorry, i did not see the "--align-files" option, i will try it later and report the result

 

[ice1006]

Great!!!!!! The problem is solved with "--align-files" option, the repacked iso works well!!!!!! And even I rebuild the message files and replace them, it works well, too!!!!!

Thanks great wiimm, you have solved one big problem, as I know there are not any tools could solve this bug except wit tools 2.02a version, I have tried many many Wii ISO unpacking/repacking/replaceing tools, all the other wii tools(include WiiScrubber 1.4 and PartitionBuilder 1.2) could not repack a normal iso, you are the first and the only one!!!!

Thanks very much for your hard work!!!!!!

 

[Wiimm]

I will make more tests with --align-files. If it works without disadvantages, I'll enable this option by default.


EDIT:
Idea: There is no need to store the alignment in an extra file. fst.bin contains all infos and also the sorting order.

 

[kylster]

Hi, I have a question about WIA format has this been tested thoroughly where converting from WIA< ISO or WBFS will not corrupt the game? I would like to store a backup of my games using this format as I've went from having 35GB of games to only 14.5GB of games which is really impressive. The only other reliable way of compressing my games for storage that I seemed to find effective was scrubbing the ISO and then compressing with winrar/7zip but then I found so much info about compressing with them that I'm sort of lost by the details or do not have the proper time to analyze the file sizes cause of current situations at hand.

I want to know if scrubbing and ISO and compressing it is obsolete now cause of WIA or is WIA still in testing phase?
What do you recommend? and if possible what are the best setting to compress and ISO file after scrubbing using both winRAR and 7zip; I see you have had much experience regarding testing with such software.

Thanks for your time and consideration

EDIT: what's the harm in using a program like the filesplitter vs splitting via winRAR or 7zip? here is a link to the filesplitter http://www.dekabyte.com/filesplitter/

 

[Wiimm]

Read this first, it will answer most questions about WIA, scrubbing and rar/zip: http://wit.wiimm.de/info/wia.html

Some notes:
If you don't enable raw mode (with --raw or --psel=raw) all operations of all my tools scrub while copying/converting.

I have made test series with WIA:
- Convert ANY:IMAGE1 to WIA:IMAGE2
- Convert back to ANY:IMAGE3
- wit diff IMAGE1 IMAGE2
No one failed. But I don't use WIA by myself, because I have only some discs and there is no need for me. For me it was an academic project.

If unsure, do the same 3 steps above and remove only the IMAGE1 only if the diff is ok. Backup my tools too and you are sure, that you can convert the WIA files into normal images at any time you want.

 

[kylster]

Thank you for the info

 

[Patryc]

I have been using WDF to archive gamecube images and seems to work very well, thank you Wimm.
Would it do any harm to use WIA on gamecube images?

 

[Wiimm]

I have been using WDF to archive gamecube images and seems to work very well, thank you Wimm.
Would it do any harm to use WIA on gamecube images?

That WDF works is clear, because it only find out zeroed areas and manage this holes.

I have no experience with GameCube-WIA. WIA is an attack against hash values and encryption, GC discs are always decrypted and haven't any hash values. WIA might work, but a simple rar/zip compression too.

If you do the 4 steps above than you find out the compression rate of WIA and if it works fine and correct.

 

[Patryc]

Thanks for your reply! Using GUI WiiBaFu-1.1 (Wii BackUp Fusion) by the way.