Hacking wwt+wit: Wiimms WBFS+ISO Tools

[ice1006]

How can I exract the file system of a game disc from a WBFS partition? I notice that there is no --fst option for wwt, only wit has this option.


Oh... I got it~ great!

wit extract /dev/sdb1/SVMJ01 c:\sss

 

[Wiimm]

wwt extract is using the old libwbfs code (I have changed it yesterday, but without FST support). Therefor patching options also also disabled.

Use "wit extract /dev/wbfs/id6 --dest dir" or "wit extract --auto --include id6 --dest dir" instead.

 

[ice1006]

Any plan to support WiiWare and VC etc. on Wii platform?

 

[myron0815]

@myron0815
"Könige" looks like an UTF-8 problem.

yes, for sure.
but in your supplied title-de.txt, it is correctly stated with ö.
(checked in UltraEdit - file is UTF8/unix)

so where's the problem?!
maybe any needed windows/cygwin locale env setting missing?!

Not useing WIT that often now... but just updated to latest 1.22a, and it seems this issue is back.
Now WIT replaces all the german umlauts with an " · "

 

[Wiimm]

The WIT tools have 2 options: --utf-8 and --no-utf-8: Enabling and disabling UTF-8 support.

UTF-8 is enables by default for all systems but Cygwin because Cygwin had problems with UTF-8. Theses problems are vanished away. From now on Cygwin has enabled UTF-8 by default.

A workaround for old versions is
a) using the option --utf-8 or
b) setup environment variables:

WIT_OPT=--utf-8
WWT_OPT=--utf-8

 

[ice1006]

Error occured:

D :\wit>wit llll "d:\www.wia"
!! wit: ERROR #39 [FATAL ERROR] in XReadF() @ src/lib-file.c#2049
!! ASSERTION FAILED: 124 10000

"www.wia" is a GC game file.

Wii game in wia format has no this problem.


Windows XP 32bit

 

[Wiimm]

GameCube support is only experimental and will be the next large step but in the moment I process other little things of my todo list. I have put this bug into my todo list.

 

[ice1006]

"Convert" = "Copy" + "Delete", is that right ?

I use the "convert" option to convert my game files from one format to another format, but I found the the dest path or filename of converted file is a little strange:

1、sometimes the dest path is source path, and the dest filename is source filename.(I hope convert option works is this way) for example:
C :\wit\wit convert D :\A.wbfs --wdf
D :\A.wbfs Convert=>To D :\A.wdf

2、sometimes the dest path is current path, and the dest filename ID + Name for example:
C :\wit\wit convert D :\B.wdf --wbfs
D :\B.wdf Convert=>To D :\wit\Rainbow Pop [RB2J2K].wdf


The following is a snapshot on my system, notice the source and dest path and filename:
D :\wit>wit cv "d :\Rainbow Pop [RB2J2K].wdf" --wbfs
***** wit: Wiimms ISO Tool v1.22a r2012 cygwin - Dirk Clemens - 2010-11-22 *****
* COPY+SCRUB 1/1 WDF:/cygdrive/d/Rainbow Pop [RB2J2K].wdf -> WBFS:/cygdrive/d/RB2J2K.wbfs

D :\wit>wit cv "d :\RB2J2K.wbfs" --wdf
***** wit: Wiimms ISO Tool v1.22a r2012 cygwin - Dirk Clemens - 2010-11-22 *****
* COPY+SCRUB 1/1 WBFS:/cygdrive/d/RB2J2K.wbfs -> WDF:Rainbow Pop [RB2J2K].wdf


Because the dest path/filename is not always same, and it seems that "convert" option could not accept a dest path parameter such as "d:\%6I.%E", so I think if I could repalce the "convert" with "copy" + "delete",because the "copy" option could accept the dest path parameter

 

[Wiimm]

CONVERT was planned as in place conversion. In the first releases the name was SCRUB to allow in place scrubbing. Then I decided to copy+replace the file because a) file type conversions and b) better sparse file generation. Later I renamed SCRUB to the more convenient name CONVERT.

I will check your suggestions in the next days.

 

[ice1006]

Another question:
============================================================
D :\wit>wit copy "d:\super mario collection [RMGJ01].wdf" "d:\SVMJ01" --fst -vv

wit: Wiimms ISO Tool v1.22a r2012 cygwin - Dirk Clemens - 2010-11-22

wit: EXTRACT WDF:/cygdrive/d/super mario collection [RMGJ01].wdf -> /cygdrive/d/SVMJ01/
- will extract 48 files of 1 partition, 11 MiB total
12 MiB copied in 0:01, 11.5 MiB/sec
============================================================

When the command finished, I found there is a folder named "cygdrive" with none file in it added in D :\ (it doesn't exists before), perhaps it is a bug, check it please.

 

[Wiimm]

@ice1006:
I have tested it with cygwin/WinXP and an other game and it worked as expected.

 

[Wiimm]

WIT: Wiimms ISO Tools v1.23a - 2010-12-10
A command line ISO+WBFS tool set for various os.

WIT (Wiimms ISO Tools) is a set of command line tools to manipulate Wii ISO images and WBFS containers. The two main tools are called wit (Wiimms ISO Tool) and wwt (Wiimms WBFS Tool, the WBFS manager).

Feature overview:

• WIT (Wiimms ISO Toolset) is a set of command line tools.
• wit (Wiimms ISO tool) is the main Wii and GameCube ISO manipulation tool.
• wwt (Wiimms WBFS tool) is a complete WBFS manager including FS checking and repairing, time stamps and multi WBFS support.
• wdf-cat and wdf-dump are WDF and CISO support tools.
• WIT supports 6 Wii disc file types (ISO, CISO, WDF, WIA, WBFS and FST) and can read from pipes (indirect RAR/ZIP/7Z support).
• wit can verify and compare (diff) Wii and GameCube ISO images.
• wit can extract and compose Wii ISO images.
• wit can patch Wii and GameCube ISO images while copying.
• wit can mix partitions from different sources into a new Wii ISO images.
• wit and wwt support many diagnostic commands.
• Advanced shell support.

Visit http://wit.wiimm.de/features.html for more details.

Download of the current version:

• wit-v1.23a-r2092-i386.tar.gz : Linux/i386 version
• wit-v1.23a-r2092-x86_64.tar.gz : Linux/x86_64 version
• wit-v1.23a-r2092-mac.tar.gz : Mac OS universal binaries (ppc, i386 and x86-64)
• wit-v1.23a-r2092-cygwin.zip : Cygwin (Windows) version
• Other releases are available from http://wit.wiimm.de/download.html.

There are also some GUI available:

• QtWitGui, manipulate and convert single ISO images.
  This QT based GUI focuses on manipulating single ISO images.
  .
• Wii Backup Fusion, a WBFS and ISO manager.
  This is a very young QT based GUI and still in development.
  .
• WiiJManager Multiplatform, a WBFS manager GUI
  A java GUI that acts like a WBFS manager. Converting of ISO images is also possible. (The project is dead)

Highlights of this release:

• Preallocation to avoid fragmentation
  I have implemented disc space preallocation. Under Windows it can avoid fragmentation. Therefor it is only enabled for Cygwin by default. It can be disabled with --prealloc=0 or enabled with --prealloc. See http://wit.wiimm.de/opt/prealloc for more details.
  The needed preallocation function is not available for Mac. So Mac will ignore --prealloc.
  .
• ID wildcards for some wwt commands
  The wwt commands EXTRACT + REMOVE + TOUCH + VERIFY will support now ID wildcards to select different discs. See http://wit.wiimm.de/info/disc-id.html for more details.
  .
• wwt EXTRACT rewritten
  "wwt EXTRACT" is now able to patch while extracting and to create WIA files like "wit COPY". Some new supported options are --wia --fst --pmode --psel --raw --files and --sneek. See http://wit.wiimm.de/wwt/cmd-extract.html for more details.
  .
• Environment Variables
  The WIT tools support environment variables to change the defaults. This is a very old feature but seems forgotten. See http://wit.wiimm.de/info/environ.html for details.
  .

Change log:

wit v1.23a r2092 - 2010-12-10

- Preallocation of disc space implemented. It is only enabled for Cygwin by
ÂÂ default. It can be disabled with --prealloc=0 or enabled with --prealloc.
ÂÂ The needed preallocation function is not available for mac.
- wwt EXTRACT + REMOVE + TOUCH + VERIFY:
ÂÂ These commands accept now ID6 pattern like '...P' or '+1'. Each '.' is a
ÂÂ wildcard that matches any single character. A '+' matches any number of
ÂÂ characters. For details see http://wit.wiimm.de/info/disc-id.html
- wwt EXTRACT is now able to patch while extracting and to create WIA files.
ÂÂ It support also the options --fst --pmode --psel --raw --files and --sneek.
- Cygwin will support UTF-8 by default.
- Bug fix: Error while scanning LC_CTYPE for language detection.
- Support of environment variable 'WWT_LANG' removed. Use 'WIT_LANG' instead.
- "wit EDIT" supports now the options --psel and --raw.
- Option --pmode accepts now the keywords AUTO, NONE, POINT, ID, NAME, INDEX
ÂÂ and COMBI. Mode 'AUTO' switches to 'COMBI' if there are multiple partitions
ÂÂ with the same partition type.


Known bugs:

- The GameCube support is in development and EXPERIMENTAL!
- wit COPY --raw: Copying an image in raw mode to a WBFS file is buggy.
- There are new problems while reading from non seekable sources (e.g. pipes).
ÂÂ The reason is the complete new and more intelligent wiidisc lib which makes
ÂÂ more lookups ==> The best is to use version 1.01c for pipes.
- No WIA/bzip2 support for MAC/x86_64. (bug in the Mac bzip2 library)


Missing docu:

- wit commands FILES + EXTRACT => see built in help
- Tool wdf => see built in help

The source is published under the license of GPL2. Checkout the sources directly from the SVN repository or browse the source. Visit http://wit.wiimm.de/ for more information.

 

[Wiimm]

WIT: Wiimms ISO Tools v1.23b - 2010-12-12
A command line ISO+WBFS tool set for various os.

WIT (Wiimms ISO Tools) is a set of command line tools to manipulate Wii ISO images and WBFS containers. The two main tools are called wit (Wiimms ISO Tool) and wwt (Wiimms WBFS Tool, the WBFS manager).

Download of the current version:

• wit-v1.23b-r2096-i386.tar.gz : Linux/i386 version
• wit-v1.23b-r2096-x86_64.tar.gz : Linux/x86_64 version
• wit-v1.23b-r2096-mac.tar.gz : Mac OS universal binaries (ppc, i386 and x86-64)
• wit-v1.23b-r2096-cygwin.zip : Cygwin (Windows) version
• Other releases are available from http://wit.wiimm.de/download.html.

There are also some GUI available:

• QtWitGui, manipulate and convert single ISO images.
  This QT based GUI focuses on manipulating single ISO images.
  .
• Wii Backup Fusion, a WBFS and ISO manager.
  This is a very young QT based GUI and still in development.
  .
• WiiJManager Multiplatform, a WBFS manager GUI
  A java GUI that acts like a WBFS manager. Converting of ISO images is also possible. (The project is dead)

Highlights of this release:

• There are only 2 little changes that are needed/wanted by other projects.
  .

Change log:

wit v1.23b r2096 - 2010-12-12

- The progress counter is updated more frequently while creating a WIA file.
- Bug fix for options --one-job and --job-limit.


Known bugs:

- The GameCube support is in development and EXPERIMENTAL!
- wit COPY --raw: Copying an image in raw mode to a WBFS file is buggy.
- There are new problems while reading from non seekable sources (e.g. pipes).
ÂÂ The reason is the complete new and more intelligent wiidisc lib which makes
ÂÂ more lookups ==> The best is to use version 1.01c for pipes.
- No WIA/bzip2 support for MAC/x86_64. (bug in the Mac bzip2 library)

The source is published under the license of GPL2. Checkout the sources directly from the SVN repository or browse the source. Visit http://wit.wiimm.de/ for more information.

 

[Hielkenator]

I have a problem..

I can read and extract .iso files from my 1TB .wbfs drive.

But i cannot add any games with any backup manager.
After a few seconds it just stops.
I've checked with wb fusion 1.23. I had various errors.

After i repaired the drive everything was fine. I could add .iso files again.
Then i tried to repair the free space errors.

Now my drive has no errors ( wb fusion tells me ) but I can't add iso's any more ( again ).

Can somebody help me?

About 5% transferring and it all stops....

 

[Wiimm]

try "wwt check --auto" to check your drive.

 

[airline38]

@Wiimm , I have a GC game ISO (size:1.35GB) and I have no problem to use a tool:GC-Tool
to open/check this ISO.(see below)

However, wit shows this error message below. It seems wit can't recognize this ISO.
(It's no problem for me to run this game on my Wii after burning this ISO to a disc.)

Just wonder if this is a problem of "wit" ? (suppose v1.23 should be able to handle GC ISO)
or ISO file problem ?

D:\Wii\WiiGF>wit verify luigi.iso
*****ÂÂwit: Wiimms ISO Tool v1.23b r2096 cygwin - Dirk Clemens - 2010-12-12ÂÂ***
**
!! wit: ERROR #27 [WRONG FILE TYPE] in SourceIteratorHelper() @ src/lib-sf.c#416
5
!!ÂÂÂÂÂÂWii ISO image expected: luigi.iso
! wit: No valid source file found.

 

[deo]

Hey Wiimm,

I've just updated my sandbox and found this error during make:

$ make
***ÂÂobject gen-ui.oÂÂÂÂÂÂÂÂÂÂ
***ÂÂobject LzFind.oÂÂÂÂÂÂÂÂÂÂ
***ÂÂobject Lzma2Dec.oÂÂÂÂÂÂÂÂ
***ÂÂobject Lzma2Enc.oÂÂÂÂÂÂÂÂ
***ÂÂobject LzmaDec.oÂÂÂÂÂÂÂÂ 
***ÂÂobject LzmaEnc.oÂÂÂÂÂÂÂÂ 
***ÂÂobject cert.oÂÂÂÂÂÂÂÂÂÂÂÂ
***ÂÂobject dclib-utf8.oÂÂÂÂÂÂ
***ÂÂobject debug.oÂÂÂÂÂÂÂÂÂÂ 
***ÂÂobject file-formats.oÂÂÂÂ
***ÂÂobject iso-interface.oÂÂ 
***ÂÂobject lib-bzip2.oÂÂÂÂÂÂ 
***ÂÂobject lib-ciso.oÂÂÂÂÂÂÂÂ
***ÂÂobject lib-file.oÂÂÂÂÂÂÂÂ
src/lib-file.c: In function ‘PreallocHelper’:
src/lib-file.c:1778: warning: implicit declaration of function ‘posix_fallocate’
***ÂÂobject lib-lzma.oÂÂÂÂÂÂÂÂ
***ÂÂobject lib-sf.oÂÂÂÂÂÂÂÂÂÂ
***ÂÂobject lib-std.oÂÂÂÂÂÂÂÂ 
***ÂÂobject lib-wdf.oÂÂÂÂÂÂÂÂ 
***ÂÂobject lib-wia.oÂÂÂÂÂÂÂÂ 
***ÂÂobject libwbfs.oÂÂÂÂÂÂÂÂ 
***ÂÂobject match-pattern.oÂÂ 
***ÂÂobject patch.oÂÂÂÂÂÂÂÂÂÂ 
***ÂÂobject rijndael.oÂÂÂÂÂÂÂÂ
***ÂÂobject sha1_one.oÂÂÂÂÂÂÂÂ
***ÂÂobject sha1dgst.oÂÂÂÂÂÂÂÂ
***ÂÂÂÂ asm ssl-asm.oÂÂÂÂÂÂÂÂ 
***ÂÂobject titles.oÂÂÂÂÂÂÂÂÂÂ
***ÂÂobject tools.oÂÂÂÂÂÂÂÂÂÂ 
***ÂÂobject ui.oÂÂÂÂÂÂÂÂÂÂÂÂÂÂ
***ÂÂobject wbfs-interface.oÂÂ
***ÂÂobject wiidisc.oÂÂÂÂÂÂÂÂ 
***ÂÂhelper gen-uiÂÂÂÂÂÂÂÂÂÂÂÂ
Undefined symbols:
ÂÂ"_posix_fallocate", referenced from:
ÂÂÂÂÂÂ_PreallocHelper in lib-file.o
ld: symbol(s) not found
collect2: ld returned 1 exit status
make: *** [gen-ui] Error 1

I'm running OSX 10.6.x here, and after googling for a while I found this reference.

My C skills are... well, forget about it, but please consider incorporate something along these lines:

CODEIndex: src/lib-file.c
===================================================================
--- src/lib-file.cÂÂÂÂ(revision 2100)
+++ src/lib-file.cÂÂÂÂ(working copy)
@@ -45,6 +45,10 @@
ÂÂ #include
#endif

+#if defined(HAVE_FALLOCATE)
+ÂÂ#include
+#endif
+
#include "lib-std.h"
#include "lib-sf.h"
#include "wbfs-interface.h"
@@ -1769,15 +1773,37 @@
ÂÂÂÂÂÂÂÂÂÂÂÂfound = *ptr;
ÂÂÂÂÂÂÂÂÂÂÂÂfound_size = found->size;
ÂÂÂÂÂÂÂÂÂÂÂÂ}
+
ÂÂÂÂÂÂÂÂif (!found)
ÂÂÂÂÂÂÂÂÂÂÂÂbreak;

-ÂÂÂÂÂÂÂÂPRINT("CALL posix_fallocate(%d,%9llx,%9llx [%s])\n",
+ÂÂÂÂÂÂÂÂPRINT("CALL specific allocate method with (%d,%9llx,%9llx [%s])\n",
ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂf->fd, (u64)found->off, (u64)found->size,
ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂwd_print_size_1024(0,0,found->size,true) );
-ÂÂÂÂÂÂÂÂposix_fallocate(f->fd,found->off,found->size);
+
+ # ifdef HAVE_FALLOCATE
+
+ÂÂÂÂÂÂÂÂfallocate(f->fd, FALLOC_FL_KEEP_SIZE, found->off, found->size);
+
+ # elif defined(__APPLE__)
+
+ÂÂÂÂÂÂÂÂfstore_t fst;
+ÂÂÂÂÂÂÂÂfst.fst_flags = F_ALLOCATECONTIG;
+ÂÂÂÂÂÂÂÂfst.fst_posmode = F_PEOFPOSMODE;
+ÂÂÂÂÂÂÂÂfst.fst_offset = found->off;
+ÂÂÂÂÂÂÂÂfst.fst_length = found->size;
+ÂÂÂÂÂÂÂÂfst.fst_bytesalloc = 0;
+ÂÂÂÂÂÂÂÂfcntl(f->fd, F_PREALLOCATE, &fst);
+
+ # else
+
+ÂÂÂÂÂÂÂÂposix_fallocate(f->fd, found->off, found->size);
+
+ # endif
+
ÂÂÂÂÂÂÂÂfound->size = 0;
-ÂÂÂÂÂÂÂÂPRINT("posix_fallocate() done\n");
+ÂÂÂÂÂÂÂÂPRINT("allocate method done\n");
+
ÂÂÂÂÂÂÂÂ}
ÂÂÂÂ}
ÂÂÂÂ }

Thanks! :-)

--deo

 

[Wiimm]

@deo:
-DNO_PREALLOC=1 doest the same as your patch.

 

[deo]

Wiimm,

actually, no.

The patch adds support for preallocation on Mac OS X, using the fcntl system call. It also gives preference to the Linux syscall that
uses filesystem-level preallocation support, if available. From what I can tell, it really does a difference when enabled on Mac OS X.

When this is in place, you can remove the NO_PREALLOC flag from the makefiles-local/Makefile.local.mac file.

BTW, all these preallocation calls (not only the mac one) should probably be done in a safer way, don't you think? :-)

Even if the patch isn't accepted, I would expect that by running 'make' it should auto-detect the correct flags and don't issue any error/warning.

As always, keep the great work!

Take care,

-- deo

 

[Wiimm]

Ok, thanx.
I have read your post very fast because of no time.