Hacking wwt+wit: Wiimms WBFS+ISO Tools

jivaros

Member
Newcomer
Joined
Sep 29, 2009
Messages
8
Trophies
0
XP
12
Country
France
I'm am very glad to see a linux tool intensively developped !

Like i said before, a check function should be great, but i suggest another cool and unique feature :
importing an iso/wdf from a .rar (or .zip) file, without uncompressing the archive in a temporary directory ...

It seems with unrar-nonfree, it is possible to extract a file to stdin, but not with the unrar-free tool.

If you give me some pointers, (and if you wish) i could try to submit a patch for the rar2wbfs function !
 

Wiimm

Developer
OP
Member
Joined
Aug 11, 2009
Messages
2,286
Trophies
0
Location
Germany
Website
wiimmfi.de
XP
1,442
Country
Germany
jivaros said:
I'm am very glad to see a linux tool intensively developped !

Like i said before, a check function should be great, but i suggest another cool and unique feature :
importing an iso/wdf from a .rar (or .zip) file, without uncompressing the archive in a temporary directory ...

It seems with unrar-nonfree, it is possible to extract a file to stdin, but not with the unrar-free tool.

If you give me some pointers, (and if you wish) i could try to submit a patch for the rar2wbfs function !

This is a hard job because the WBFS framework (written by others) needs random access and uses the seek() functions to address to the file. And thats mean that the image must be decompressed completely before the access.

On the other hand most (but not the header) of the seek access are at increased offsets: A seek simulation is possible when caching only a few (about 8 MiB) of data. In the next month i will go deeper into the WBFS framework.
 

Wiimm

Developer
OP
Member
Joined
Aug 11, 2009
Messages
2,286
Trophies
0
Location
Germany
Website
wiimmfi.de
XP
1,442
Country
Germany
wwt: Wiimms WBFS Tool v0.03a - 2009-10-08
A command line WBFS tool for linux.

I have updated the tool. The first post with more details is updated too.

I have tested this tool very well and the software do what I want but I can't give any warranty. I have made different import and export (ISO and WDF) tests with about 20 games and the resulting ISO images are absolute identical as the source files.

wwt is now Beta and I'm using it as replacement for the tool 'wbfs'.


Download: http://wiimm.de/x/./wwt-v0.03a-r200.tar.gz (Documentation is included)
More Infos about WDF: http://svn.wiimm.de/open/wii/trunk/wiimms-...WDF.txt?view=co

Last changes:
Code:
wwt V0.03a r200 - 2009-10-08

- Enhanced error management again.
- Misspelling in --help: The shortcut for 'DUMP' is 'D' and not 'DP'.
- Make: Support of environment variable 'XFLAGS' -> 'XFLAGS=-march=x86-64 make'
- File handling: Support of open() and fopen() as alternatives for testing.
- Script test-wdf-tools.sh: Positive test is now done with >1 million files.
- New option for 'wdf-dump': --chunk: Print chunk header table.
- WDF specification is complete.

@moparisthebest and others:
Just create a file named 'Makefile.local' and make you local modifications here.
 

Wiimm

Developer
OP
Member
Joined
Aug 11, 2009
Messages
2,286
Trophies
0
Location
Germany
Website
wiimmfi.de
XP
1,442
Country
Germany
Feature Request:

Code:
ÂÂREQUEST:
ÂÂÂÂI suggest another cool and unique feature:
ÂÂÂÂImporting an iso/wdf from a .rar (or .zip) file, without uncompressing the
ÂÂÂÂarchive in a temporary directory ...

ÂÂANSWER:
ÂÂÂÂThis is a hard job because the WBFS framework (written by others) needs
ÂÂÂÂrandom access and uses the seek() functions to address to the file. And
ÂÂÂÂthat means that the image must be decompressed completely before access. 

ÂÂÂÂOn the other hand most (but not the header) of the seek access are at
ÂÂÂÂincreased offsets: A seek simulation is possible. In the next month i will
ÂÂÂÂgo deeper into the WBFS framework.

ÂÂANALYSIS:
ÂÂÂÂI have quickly tested 3 iso images:
ÂÂÂÂIt seems to be possible when 2 file ranges are cached:
ÂÂÂÂ-ÂÂÂÂÂÂ 0x0.. 0x200000 (len=0x200000 = 2 MiB)
ÂÂÂÂ- 0xf800000..0xfe00000 (len=0x600000 = 6 MiB)
ÂÂÂÂAll other file access is in sequential order.

ÂÂIMPLEMENTATION:
ÂÂÂÂDone but experimental. Use option --cache to enable it.
Is there anybody who wants to test this feature with many game discs?
 

patrick9999

Active Member
Newcomer
Joined
Apr 8, 2009
Messages
32
Trophies
0
XP
31
Country
France
Hi,

Thanks you, i love and use it every day
tongue.gif
 

Wiimm

Developer
OP
Member
Joined
Aug 11, 2009
Messages
2,286
Trophies
0
Location
Germany
Website
wiimmfi.de
XP
1,442
Country
Germany
wwt: Wiimms WBFS Tool v0.04a - 2009-10-11
A command line WBFS tool for linux.

I have updated the tool. The first post with more details is updated too.

I have tested this tool very well and the software do what I want but I can't give any warranty. I have made different import and export (ISO and WDF) tests with about 20 games and the resulting ISO images are absolute identical as the source files.

wwt is now Beta and I'm using it as replacement for the tool 'wbfs'.


Download: http://wiimm.de/x/./wwt-v0.04a-r212.tar.gz (Documentation is included)
More Infos about WDF: http://svn.wiimm.de/open/wii/trunk/wiimms-...WDF.txt?view=co

Last changes:
CODEwwt V0.04a r212 - 2009-10-11

- ADD: ID6 is now also recognized when source is a WDF.
- New command: FILETYPE: Print a status line for each given file.
- New script: test-ADD+EXTRACT.sh: ADD + EXTRACT test with various options.
- *EXPERIMENTAL*:
ÂÂÂÂ - Option --cache enables caching of well known file parts while reading
ÂÂÂÂÂÂ an ISO image. Perhaps cached areas must be expanded.
ÂÂÂÂ - Adding a game from a pipe if option --cache is set. Commands like
ÂÂÂÂÂÂ "wwt ... ADD --cache
 

thza

New Member
Newbie
Joined
Sep 12, 2009
Messages
2
Trophies
0
XP
2
Country
United States
I really like this tool, but please follow the Linux Filesystem Hierarchy Standard (FHS). The titles.txt files should be stored under $PREFIX/share/wwt/titles.txt, not under $PREFIX/lib/titles.txt.
 

Wiimm

Developer
OP
Member
Joined
Aug 11, 2009
Messages
2,286
Trophies
0
Location
Germany
Website
wiimmfi.de
XP
1,442
Country
Germany
thza said:
I really like this tool, but please follow the Linux Filesystem Hierarchy Standard (FHS). The titles.txt files should be stored under $PREFIX/share/wwt/titles.txt, not under $PREFIX/lib/titles.txt.
done.
(But the install script must be improved -- later)
 

gizmomelb

Well-Known Member
Member
Joined
Jul 13, 2006
Messages
930
Trophies
0
XP
1,647
Country
Australia
Hi Wiimm, this looks like a very interesting tool.

I just wanted to clarify if your scrub ISO option produces files the same as the cISO format please (ie: as ripped by any USB loader) or is there another option which would produce the same output?

Thank you.
 

Wiimm

Developer
OP
Member
Joined
Aug 11, 2009
Messages
2,286
Trophies
0
Location
Germany
Website
wiimmfi.de
XP
1,442
Country
Germany
gizmomelb said:
Hi Wiimm, this looks like a very interesting tool.

I just wanted to clarify if your scrub ISO option produces files the same as the cISO format please (ie: as ripped by any USB loader) or is there another option which would produce the same output?

Thank you.
SCRUB is planned but not subject of interest in the moment. And answers about cISO are hard because I don't know what cISO is (missing technical info).
 

Bauldrick

Well-Known Member
Member
Joined
Jan 22, 2009
Messages
103
Trophies
0
XP
100
Country
How can I obtain the source, I wish to compile on mips architecture. I try svn but get:

svn: Repository moved permanently to '/open/wii/trunk/wiimms-wbfs-tool/'; please relocate

Many thanks
 

Wiimm

Developer
OP
Member
Joined
Aug 11, 2009
Messages
2,286
Trophies
0
Location
Germany
Website
wiimmfi.de
XP
1,442
Country
Germany
Bauldrick said:
How can I obtain the source, I wish to compile on mips architecture. I try svn but get:

svn: Repository moved permanently to '/open/wii/trunk/wiimms-wbfs-tool/'; please relocate

Many thanks
This is only the web view to the open parts off the svn. The svn itself is closed (account is needed).
Please use the tarball (link at the bottom in every directory view).
 

Bauldrick

Well-Known Member
Member
Joined
Jan 22, 2009
Messages
103
Trophies
0
XP
100
Country
Hi Wiimm,

Thanks, I haven't tested it on my box yet, have only just got it to compile. The only way I could compile was to comment out
Code:
CFLAGSÂÂÂÂÂÂÂÂ += -mtune=generic

else I would get
Code:
cc1: error: unrecognized command line option "-mtune=generic"

This is on a mips cpu

Well, it appears to work - can find wbfs hdd and list games on it, great stuff...

-edit-

This is awesome work, many thanks for this! iso2wdf (moreover wdf) is great
smile.gif
 

Wiimm

Developer
OP
Member
Joined
Aug 11, 2009
Messages
2,286
Trophies
0
Location
Germany
Website
wiimmfi.de
XP
1,442
Country
Germany
wwt: Wiimms WBFS Tool v0.05a - 2009-10-14
A command line WBFS tool for linux.

I have updated the tool. The first post with more details is updated too.

I have tested this tool very well and the software do what I want but I can't give any warranty. I have made different import and export (ISO and WDF) tests with about 20 games and the resulting ISO images are absolute identical as the source files.

wwt is now Beta and I'm using it as replacement for the tool 'wbfs'.


Download: http://wiimm.de/x/./wwt-v0.05a-r247.tar.gz (Documentation is included)
More Infos about WDF: http://svn.wiimm.de/open/wii/trunk/wiimms-...WDF.txt?view=co

Last changes:
Code:
wwt V0.05a r247 - 2009-10-14

- Complete new interface to the WBFS subsystem:
ÂÂÂÂ- unused source files removed.
ÂÂÂÂ- improved error messages of libwbfs
- WBFS file handling rewritten. It supports now file (using open() function)
ÂÂ and stream (using fopen() function) methods. Use --io to select method.
- New option: --ignore: Ignore non existing files/discs without warnings.
ÂÂ Can be used with: ADD, EXTRACT, REMOVE (replaces --force) and FILETYPE.
- The lib path moved from '/usr/local/lib/' to '/usr/local/share/wwt/'.
- New option --sector-size: Set logical hd sector size larger than 512
ÂÂ while formatting WBFS => The max disc count of a WBSF will be raised.
- Title database implemented.
- New command: TITLES: Print the internal title database to stdout.
- ADD: ISO image is acepted from stdin if a file is named '-'. For pipes
ÂÂ the experimental option --cache must be set.

The --cache option is object to test. If adding a file from pipe fails then please use the binary wwt from 'debug/' folder and add the file again without piping and without --cache. The debug version create a file called '_trace.tmp' which much info. I need this file for analysis.

-----

EDIT:
WARNING: The last version of WWT v0.05a fails if formatting a block device (a real partition).


reason: playing with sector size ...
frown.gif
 

Wiimm

Developer
OP
Member
Joined
Aug 11, 2009
Messages
2,286
Trophies
0
Location
Germany
Website
wiimmfi.de
XP
1,442
Country
Germany
Bauldrick said:
Hi Wiimm,

Thanks, I haven't tested it on my box yet, have only just got it to compile. The only way I could compile was to comment out
Code:
CFLAGSÂÂÂÂÂÂÂÂ += -mtune=generic

else I would get
Code:
cc1: error: unrecognized command line option "-mtune=generic"

This is on a mips cpu

Well, it appears to work - can find wbfs hdd and list games on it, great stuff...The option "-mtune=generic" has now moved to Makefile.local (a local makefile for each users.)

There is a script called scripts/test-ADD+EXTRACT.sh :
Just call it with a list of ISO or WDF files: it will add and extract the image with various options and compare the results. It needs much time ....

QUOTE(Bauldrick @ Oct 14 2009, 02:28 PM) This is awesome work, many thanks for this! iso2wdf (moreover wdf) is great
smile.gif
be carefull: iso2wdf does not scrubbing the image.
 
General chit-chat
Help Users
  • AncientBoi @ AncientBoi:
    And to shoot it well. :)
    Gift
  • Psionic Roshambo @ Psionic Roshambo:
    Indeed! Lol
    +1
    Gift
  • Dark_Phoras @ Dark_Phoras:
    Do you guys know what's Mr. Freeze's favorite date program? Netflix and chill
    Gift
  • Gift
  • Gift
  • FAST6191 @ FAST6191:
    Isn't Mr Freeze's backstory about him losing his wife?
    Gift
  • Flame @ Flame:
    @FAST6191 his called @Dark_Phoras for a reason
    Gift
  • Dark_Phoras @ Dark_Phoras:
    Mr. Freeze's wife is frozen while he looks for a cure to her illness
    Gift
  • Dark_Phoras @ Dark_Phoras:
    But, in the meantime, he knows how to throw a cool party
    Gift
  • Psionic Roshambo @ Psionic Roshambo:
    He's a chill dude lol
    Gift
  • Flame @ Flame:
    is that true @Dark_Phoras ? thats so cool
    Gift
  • Flame @ Flame:
    a little ice cold too at the same time
    Gift
  • Psionic Roshambo @ Psionic Roshambo:
    Dude is frosty
    Gift
  • gudenau @ gudenau:
    Does anyone happen to know if the gecko code handler will always be in the same location? It would be useful if you could call into it's own subroutines in ASM codes.
    Gift
  • FAST6191 @ FAST6191:
    Are there any codes that modify the handler to do fun extras like there are for DS codes?
    Gift
  • gudenau @ gudenau:
    I don't see why you would need that, Gecko just allows you to do ASM directly instead of needing to make codes to add new code types.
    Gift
  • gudenau @ gudenau:
    There is nothing stopping you from doing that though.
    Gift
  • FAST6191 @ FAST6191:
    I was thinking more if there were (some of the DS stuff allowing I think it was different boolean masks than stock) then it would confirm that.
    Gift
  • gudenau @ gudenau:
    Some of the later DS stuff was pretty impressive. I do want to figure out how some of the lower level stuff worked one of these days.
    Gift
  • gudenau @ gudenau:
    Is this chat still on IRC?
    Gift
  • FAST6191 @ FAST6191:
    I don't know if there is a second room connected to it
    but I doubt it
    Gift
  • FAST6191 @ FAST6191:
    There is still an IRC server/channel though
    Gift
  • KenniesNewName @ KenniesNewName:
    Neat steam deck dock shipped finally
    Gift
  • mthrnite @ mthrnite:
    psi gets a dick dock and you get a deck dock, poetic!
    Gift
  • KenniesNewName @ KenniesNewName:
    Ye but mine comes with a promise of less viruses
    Gift
    KenniesNewName @ KenniesNewName: Ye but mine comes with a promise of less viruses