Hacking wwt+wit: Wiimms WBFS+ISO Tools

[drh]

1 MiB (Mebibyte and not "Men in Black") is 1 mega byte based on 1024 (1 MiB=1024*1024 bytes)

Oh, I'm confused now, how does that affect the calculation?

 

[Wiimm]

1 MiB (Mebibyte and not "Men in Black") is 1 mega byte based on 1024 (1 MiB=1024*1024 bytes)

Oh, I'm confused now, how does that affect the calculation?

Your calculation is bit based on 22 mega bits per second, but my rate is 22 MiB/s (Mebibytes) ~ 185 mega bits per second.

CODE

1 KiB = 1024 bytes
1 MiB = 1024*1024 bytes
1 GiB = 1024*1024*1024 bytes
1 TiB = 1024*1024*1024*1024 bytes

==> 1TiB / 22 MiB/s / 3600 == 13.24 hours
==> 1TiB / 25 MiB/s / 3600 == 11.65 hours

 

[drh]

Hmm, I am not able to transfer that quickly, not sure why though, I am using same drive and a quad core iMac. Maybe its the NAS thats causing a bottleneck, I know thats not that fast.

 

[zon7]

Right now the Add parameter only works with isos and wdf files. Would it be possible to implement the posibility to add also wbfs files?
As the wwt tool is able to extract to wbfs, i suppose it wouldn't be difficult to make the opposite. Right now the only way to do it is to convert the wbfs to iso and then transfer it to the wbfs partition. Correct me if I'm wrong.

Thanks a lot

 

[Wiimm]

Right now the Add parameter only works with isos and wdf files. Would it be possible to implement the posibility to add also wbfs files?

This is done months ago, since "wwt v0.12a r364 - 2009-11-05". The add command accept wbfs files as whole and with a selector like WBFS_FILE/ID6 or WBFS_FILE/INDEX or WBFS_FILE/#SLOT.

The wit command can also use WBFS files as source.
Example: wit cmp WBFS/mcp01 "mario kart wii.wdf"
This will compare the contents of both sources.

Extract of the docu:
CODE///////////////////////////////////////////////////////////////////////////////
///////////////ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂProcessing ISO filesÂÂÂÂÂÂÂÂÂÂÂÂ ///////////////
///////////////////////////////////////////////////////////////////////////////

Commands like "wwt ADD" or "wit COPY" uses ISO files as input. They accept
plain ISO files and ISO files in WDF and WBFS containers. In detail:

/path/to/PLAIN_ISO:
ÂÂÂÂA PLAIN ISO file is a 1:1 copy of a Wii disc, may be scrubbed. It is not
ÂÂÂÂcompressed and not part of any container. The standard extension is ".iso".

/path/to/WDF_ISO:
ÂÂÂÂThis is an ISO packed in a WDF container. WDF container may be used for
ÂÂÂÂany files and not only for ISO images. The standard extension is ".wdf".

/path/to/WBFS_FILE
ÂÂÂÂAn WBFS file/device is also accepted like a list of ISO files. The standard
ÂÂÂÂextension for WBFS files is ".wbfs".

/path/to/WBFS_FILE/SELECTOR
ÂÂÂÂThis is a special construct. The ISO image of the WBFS_FILE selected
ÂÂÂÂby SELECTOR is used as source. The selector may be one of:

ÂÂÂÂ - 'ABCDEF' : ID6 of a disc (exact 6 characters)

ÂÂÂÂ - indexÂÂÂÂ: The zero based index of the disc. The range goes from zero
ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂup to the number of discs in the WBFS minus 1.
ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ(decimal number, but not 6 digits)

ÂÂÂÂ - '#' slot : The decimal slot number within the WBFS. The range goes from
ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂzero up to the maximal number of possible discs in the WBFS
ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂminus 1.

ÂÂÂÂExamples:
ÂÂÂÂÂÂ.../a.wbfs/rmcp01 : use "Mario Kart" from 'a.wbfs' as source.
ÂÂÂÂÂÂ.../a.wbfs/5ÂÂÂÂÂÂ: use the image with index #5 from 'a.wbfs' as source.
ÂÂÂÂÂÂ.../a.wbfs/#5ÂÂÂÂ : use the image at slot #5 from 'a.wbfs' as source.

 

[Wiimm]

DELETE ME

 

[zon7]

Well, this is the output I get with release 0.33

D:\usb\WingIsland [RWIP18]>wwt find
/dev/sdb1

D:\usb\WingIsland [RWIP18]>wwt ft R83PA4.wbfs

file
type filename
-------------------------------------------------------------------------------
WBFS R83PA4.wbfs


D:\usb\WingIsland [RWIP18]>wwt.exe -p /dev/sdb1 add R83PA4.wbfs
***** wwt: Wiimms WBFS Tool v0.33a r659 cygwin - Dirk Clemens - 2010-03-01 ***
**
WBFSv0 #1 opened: /dev/sdb1
* Summary of WBFS Check: 3 errors found:
3 free WBFS sectors marked as used!
Note: 3 errors are based on a bug in libwbfs v0.

!! wwt: ERROR #16 [WRONG FILE TYPE] in exec_add() @ wwt.c#1955
!! Not a ISO image: R83PA4.wbfs

D:\usb\WingIsland [RWIP18]>wit copy R83PA4.wbfs
***** wit: Wiimms ISO Tool v0.33a r659 cygwin - Dirk Clemens - 2010-03-01 ****
*

As you can see, I want to dump the wbfs file to the wbfs partition. With iso and wdf it works as I told you before, but with wbfs is not working

 

[Wiimm]

Yes, it *was* a bug in the virtual file layer!
I have used a wrong file size (iso image size within wbfs instead of block device size) to control reading behind end of file and fill with zeros.

 

[zon7]

So it needs a fix?

 

[Wiimm]

I meant: I fixed it today.

Because changing many thinks I will proof the tools before the next release.

 

[Wiimm]

wwt: Wiimms WBFS Tool v0.34a - 2010-03-11
A command line WBFS tool set for various os.

I have updated the tool. The first post with more details is updated too.

Download this EXPERIMENTAL release:

• wwt-v0.34a-r691-i386.tar.gz : linux/i386 version
• wwt-v0.34a-r691-cygwin.zip : cygwin (win32) version

Highlights/bug fixes of this release:

• The WBFS inode has a space of 128 bytes for additinal information. I used this to store timestamps (itime, mtime, ctime, atime) and more. The base idea is described here.
  
  Some details for timestamps:
  .
  • The 'itime' is the insertion time.
    The insertion time is set only after adding a game to the WBFS.
    
  • The 'mtime' is the last modification time.
    While adding a disc it is copied from the source. Changing title or ID in the ISO header will update mtime. When extracting a ISO the mtime of the destination file is set to this value.
    
  • The 'ctime' is the last status change time.
    The status change time is set after adding a game or changing the ID or the title.
    
  • The 'atime' is the last access time.
    It is set when ctime is set or when extracing is finished.
  .
• The wit+wwt command LIST print one timestamp controlled by the options --itime, --mtime, --ctime, --atime and --time=modelist.
  .
• The sort option --sort accepts the new keywords 'DATE' and 'TIME' for sorting by date and 'ascending', 'descending' or 'reverse' to controll the main sort direction.
  .
• With the new wwt command TOUCH any of the timestamps can be set by the user.
  .

Last changes:

wwt v0.33a r659 - 2010-03-01

- Support of the new extended WBFS inodes with the data records
ÂÂ 'itime', 'mtime', 'ctime', 'atime' and 'load_count'.
ÂÂÂÂ- ADD updates itime, mtime, ctime and atime. mtime is taken from source.
ÂÂÂÂ- EXTRACT updates atime. mtime is copied to destination file.
ÂÂÂÂ- RENAME+SETTILE @INODE updates ctime and atime.
ÂÂÂÂ- RENAME+SETTILE @ISO updates mtime, ctime and atime.
ÂÂÂÂ- LIST may show one of the time values (-ll, --*time, --time=mode)
ÂÂÂÂ- DUMP show all times if disc dump is enabled (--long).
- wit+wwt LIST: The new options --itime, --mtime, --ctime, --atime 
ÂÂ and --time=modelist will force output of a time stamp. The meaning
ÂÂ of --long is changed.
- Option --sort= supports new modes 'DATE', 'ASCENDING' and 'DESCENDING'.
- New wwt command: LIST-LL or LLL: Abbreviation of 'LIST -ll'.
- New wit command: LIST-LLL or LLLL: Abbreviation of 'LIST -lll'.
- Bug fix: wit LIST: The WBFS file size was listed instead the ISO size.

wwt v0.34a r691 - 2010-03-11

- New timestamp: 'dtime' is set when a disc is deleted.
- wwt FORMAT: If formatting a block device empty inodes with empty but valid
ÂÂ extended inode infos are written and the dtime is set.
- wwt PHANTOM: Setup extended inode info (time stamps + ...) too.
- wwt CHECK/REPAIR: new mode: --repair=inode: Setup all missing inode infos.
- New command: wwt TOUCH: Set timestamps of inode info (created if not exist).
ÂÂ Supported time options: --itime --mtime --ctime --atime --set-time=time
- All tools can write to a block or character device if --overwrite is set.
- wwt FIND --quiet: report only via return status if a WBFS is found.
- The split size for WBFS and ISO is now rounded down to a multiple of 32 KiB.
ÂÂ Any split sizes are allowed for WDF.
- wwt bug fix: If extracting to WBFS the extension '.wbfs' was not added.
- New wwt DUMP option: --inode: show status of all inodes (deleted discs too).
- libcrypto is now statically linked, so wwt runs on older linux systems.
- wwt+wit FILETYPE: Support of option --no-header
- Test script test-ADD+EXTRACT.sh runs again.
- Bug fix: Sometimes reading an ISO from a WBFS runs into "Not a ISO image".

Sources comes with GPL2. Checkout the sources directly from the SVN repository or browse the source.

 

[zon7]

Hi again. It continues working the same way. Here you have a extraction and then the failed attempt to add it again

D:\usb\WingIsland [RWIP18]>wwt -p /dev/sdb1 EXTRACT R83PA4 -B -z -P -v

wwt: Wiimms WBFS Tool v0.34a r691 cygwin - Dirk Clemens - 2010-03-11

One WBFS partition found.

WBFSv0 #1 opened: /dev/sdb1
* Summary of WBFS Check: 3 errors found:
3 free WBFS sectors marked as used!
Note: 3 errors are based on a bug in libwbfs v0.

- EXTRACT R83PA4 -> WBFS:R83PA4.wbfs
* WBFS #1: 1 disc extracted.

D:\usb\WingIsland [RWIP18]>wwt -p /dev/sdb1 remove R83PA4
***** wwt: Wiimms WBFS Tool v0.34a r691 cygwin - Dirk Clemens - 2010-03-11 ***
**
WBFSv0 #1 opened: /dev/sdb1
* Summary of WBFS Check: 3 errors found:
3 free WBFS sectors marked as used!
Note: 3 errors are based on a bug in libwbfs v0.

* WBFS #1: 1 disc removed.

D:\usb\WingIsland [RWIP18]>wwt -p /dev/sdb1 ADD R83PA4.wbfs
***** wwt: Wiimms WBFS Tool v0.34a r691 cygwin - Dirk Clemens - 2010-03-11 ***
**
WBFSv0 #1 opened: /dev/sdb1
* Summary of WBFS Check: 3 errors found:
3 free WBFS sectors marked as used!
Note: 3 errors are based on a bug in libwbfs v0.

!! wwt: ERROR #16 [WRONG FILE TYPE] in exec_add() @ wwt.c#2009
!! Not a ISO image: R83PA4.wbfs

 

[Wiimm]

This is another bug since changing to a more intelligent add-interface (first counting and than doing). There is no error if the WBFS contains more than 1 disc.

Try a work around until the next release:

your code: wwt -p /dev/sdb1 ADD R83PA4.wbfs
ÂÂÂÂÂÂtry: wwt -p /dev/sdb1 ADD R83PA4.wbfs/0

The "/0" means: add the ISO with index #0 (zero based index).

 

[drh]

1 MiB (Mebibyte and not "Men in Black") is 1 mega byte based on 1024 (1 MiB=1024*1024 bytes)

Oh, I'm confused now, how does that affect the calculation?

Your calculation is bit based on 22 mega bits per second, but my rate is 22 MiB/s (Mebibytes) ~ 185 mega bits per second.

CODE

1 KiB = 1024 bytes
1 MiB = 1024*1024 bytes
1 GiB = 1024*1024*1024 bytes
1 TiB = 1024*1024*1024*1024 bytes

==> 1TiB / 22 MiB/s / 3600 == 13.24 hours
==> 1TiB / 25 MiB/s / 3600 == 11.65 hours

It took 49 Hours 23 mins to do 465 discs:

Total: 465/500 discs, 951104 MiB ~ 929 GiB used, 16 MiB ~ 0 GiB free.

 

[Wiimm]

wwt: Wiimms WBFS Tool v0.34b - 2010-03-12
A command line WBFS tool set for various os.

I have updated the tool. The first post with more details is updated too.

Download this EXPERIMENTAL release:

• wwt-v0.34b-r701-i386.tar.gz : linux/i386 version
• wwt-v0.34b-r701-cygwin.zip : cygwin (win32) version

Last changes:

wwt v0.34a r691 - 2010-03-11

- New timestamp: 'dtime' is set when a disc is deleted.
- wwt FORMAT: If formatting a block device empty inodes with empty but valid
ÂÂ extended inode infos are written and the dtime is set.
- wwt PHANTOM: Setup extended inode info (time stamps + ...) too.
- wwt CHECK/REPAIR: new mode: --repair=inode: Setup all missing inode infos.
- New command: wwt TOUCH: Set timestamps of inode info (created if not exist).
ÂÂ Supported time options: --itime --mtime --ctime --atime --set-time=time
- All tools can write to a block or character device if --overwrite is set.
- wwt FIND --quiet: report only via return status if a WBFS is found.
- The split size for WBFS and ISO is now rounded down to a multiple of 32 KiB.
ÂÂ Any split sizes are allowed for WDF.
- wwt bug fix: If extracting to WBFS the extension '.wbfs' was not added.
- New wwt DUMP option: --inode: show status of all inodes (deleted discs too).
- libcrypto is now statically linked, so wwt runs on older linux systems.
- wwt+wit FILETYPE: Support of option --no-header
- Test script test-ADD+EXTRACT.sh runs again.
- Bug fix: Sometimes reading an ISO from a WBFS runs into "Not a ISO image".

wwt v0.34b r701 - 2010-03-12

- Bug fix: Reading from a WBFS with exact one discs without a disc selector
ÂÂ is possible again.
- Bug fix: Progress information is also printed if extracting to wbfs files.

Sources comes with GPL2. Checkout the sources directly from the SVN repository or browse the source.

 

[Skizzo]

BTW, is there any type of timestamp within WBFS where we could tell when the games were added? Just wondering if I could see how long it was up and running before the reboot.

I think about an extension in LIBWBFS: In each inode there is a copy of the first 256 bytes of the ISO. But only the first 128 bytes are used to mange WBFS. My idea: use the other 128 for some statistics like a timestamp and action counter. This makes repairing, undelete and recovering much easier. It's only a idea ...

Hello again Wiimm...long time. My main computer's power supply took a dump on me and I haven't bothered getting it fixed yet because I've been busy with my new Xbox 360.

Anyways, it's good to see you are still improving your already fantastic tools, and I think the new timestamping features are a great addition. Glad to see your idea reach fruition. Now's it time to get that power supply fixed so I can give it a go.

Thanks again for all your hard work, help and dedication.

 

[Wiimm]

wwt: Wiimms WBFS Tool v0.35a - 2010-03-14
A command line WBFS tool set for various os.

I have updated the tool. The first post with more details is updated too.

Download of the current version:

• wwt-v0.35a-r713-i386.tar.gz : linux/i386 version
• wwt-v0.35a-r713-cygwin.zip : cygwin (win32) version

Download of the previous stable version:

• wwt-v0.33a-r659-i386.tar.gz : linux/i386 version
• wwt-v0.33a-r659-cygwin.zip : cygwin (win32) version

Last changes:

wwt v0.34b r701 - 2010-03-12

- Bug fix: Reading from a WBFS with exact one discs without a disc selector
ÂÂ is possible again.
- Bug fix: Progress information is also printed if extracting to wbfs files.

wwt v0.35a r713 - 2010-03-14

- Bug fix: The "wwt TOUCH" command didn't accept most of the allowed options.
- wwt ADD: New option --newer: Copy only newer files if mtime of source and
ÂÂ destination are available and non zero.
- RENAME+SETTITLE: Timestamps are only set if ID or title are really changing.
- Doumentation has been improved a little bit.

Sources comes with GPL2. Checkout the sources directly from the SVN repository or browse the source.

 

[zon7]

Wiimms:
I've finally had time this weekend and I have finished the "GUI" for Wiimms.
It's posted in this forum. Check it and let me know if you want something
http://gbatemp.net/index.php?showtopic=214002

 

[iranzo]

Latest addition to Makefile:
(revert diff):
--- Makefile (revisión: 717)
+++ Makefile (copia de trabajo)
@@ -106,7 +106,7 @@
LIBS = -lcrypt
else
LIBS =
-LDFLAGS += /usr/lib/libcrypto.a
+#LDFLAGS += /usr/lib/libcrypto.a
endif

LDFLAGS := $(strip $(LDFLAGS))


-----------------

Breaks compilation on x64_86 (Fedora 13).

Removing that line, seems to allow linking to continue and get useful binaries


btw, ¿are you interested in getting a spec file for creating rpm's directly from your SVN code?

 

[Wiimm]

wwt: Wiimms WBFS Tool v0.36a - 2010-03-19
A command line WBFS tool set for various os.

I have updated the tool. The first post with more details is updated too.

Download of the current version:

• wwt-v0.36a-r728-i386.tar.gz : linux/i386 version
• wwt-v0.36a-r728-cygwin.zip : cygwin (win32) version

Download of the previous stable version:

• wwt-v0.35a-r713-i386.tar.gz : linux/i386 version
• wwt-v0.35a-r713-cygwin.zip : cygwin (win32) version

Last changes:

wwt v0.35a r713 - 2010-03-14

- Bug fix: The "wwt TOUCH" command didn't accept most of the allowed options.
- wwt ADD: New option --newer: Copy only newer files if mtime of source and
ÂÂ destination are available and non zero.
- RENAME+SETTITLE: Timestamps are only set if ID or title are really changing.
- Doumentation has been improved a little bit.

wwt v0.36a r728 - 2010-03-19

- Option --sort=key: new keys: 'ITIME', 'MTIME', 'CTIME' and 'ATIME'.
- wwt+wit LIST: Support of multiple time columns.
- wwt+wit LIST+ERROR: new option --sections: print machine readable output
ÂÂ devided into sections "[...]". Each section has lines like "name=value".
- Makefile: Changed some lib settings (explicit naming of libcrypto removed).

Sources comes with GPL2. Checkout the sources directly from the SVN repository or browse the source.


@iranzo:
libcrypto problem should be solved.