Hacking wwt+wit: Wiimms WBFS+ISO Tools

iranzo

Member
Newcomer
Joined
Mar 17, 2010
Messages
5
Trophies
0
Location
Valencia
XP
27
Country
United States
QUOTE said:
@iranzo:
libcrypto problem should be solved.


Fixed and working like a charm, regarding to the other question, ¿are you interested in a spec file to automatically download source code and create an rpm from your code for easier installation?
 

Wiimm

Developer
OP
Member
Joined
Aug 11, 2009
Messages
2,292
Trophies
1
Location
Germany
Website
wiimmfi.de
XP
1,519
Country
Germany
wwt: Wiimms WBFS Tool v0.37a - 2010-03-26
A command line WBFS tool set for various os.

I have updated the tool. The first post with more details is updated too.

Download this EXPERIMENTAL release:Download of the previous stable version:
There is also a GUI available:
Highlights/bug fixes of this release:
  • The new wwt command ANALYSE checks a file or a partition for WBFS usage. It does the following steps:
    • Check the WBFS header (first 12 bytes)
    • Search valid INODE-INFO data (an extension of wwt with timestamps and a copy of the WBFS header).
    • Search disc headers (look for valid ID and MAGIC).
    • Calculate WBFS geometry like the init command with different sector sizes.
    The result of all steps are printed as table. This information can be used to find good values for the options --hss= and --wss= when using the INIT command togehter with --recover.
    .
  • The wwt command INIT --recover has been improved:
    If --hss (hd sector size) or --wss (wbfs sector size) is not set then an internal call to ANALYSE (output suppressed) is made to determine the values. The data of the first virtual row is used to override the default values of --hss and --wss.
    This should complete WBFS recovering.
    .
  • For the ANALYSE command I have written a complete new function to calculate the WBFS geometry (outsourcing of the calculation). If formatting a drive this new function is called to compare the results with the old calculation. If there are differences the program stops with an assertion failure. Please tell me such errors!
    .
Last changes:
Code:
wwt v0.36a r728 - 2010-03-19

- Option --sort=key: new keys: 'ITIME', 'MTIME', 'CTIME' and 'ATIME'.
- wwt+wit LIST: Support of multiple time columns.
- wwt+wit LIST+ERROR: new option --sections: print machine readable output
ÂÂ devided into sections "[...]". Each section has lines like "name=value".
- Makefile: Changed some lib settings (explicit naming of libcrypto removed).

wwt v0.37a r753 - 2010-03-26

- wwt: new command: ANALYSE: Analyse files and partitions for WBFS usage.
ÂÂ It try to find existing WBFS structures and make calculations for new WBFS.
ÂÂÂÂ- Check WBBS header
ÂÂÂÂ- Search INODE-INFO data (includes copy of WBFS header)
ÂÂÂÂ- Search discs for valid ID6 and MAGIC.
ÂÂÂÂ- Make example calculations for different sector sizes.
ÂÂ All results are printed as table with the main geometry data.
- wwt INIT:
ÂÂÂÂ- New option: --wss (Wbfs Sector Size): Defines the WBFS sector size.
ÂÂÂÂ- The option --sector-size (defines the HD sector size) was replaced by
ÂÂÂÂÂÂthe more handy --hss (Hd Sector Size). The old option is still allowed.
ÂÂÂÂ- If option --recover is set the command ANALYSE is called to determine
ÂÂÂÂÂÂthe values of --hss and --wss if not set before.
ÂÂ This points should complete WBFS recovering.
- For the ANALYSE command I have written a complete new function to calculate
ÂÂ the WBFS geometry (outsourcing of the calculation). If formatting a drive
ÂÂ this new function is called to compare the results with the old calculation.
ÂÂ If there are differences the program stops with an assertion failure.
ÂÂ ** Please tell me such errors! **

Sources comes with GPL2. Checkout the sources directly from the SVN repository or browse the source.
 

Wiimm

Developer
OP
Member
Joined
Aug 11, 2009
Messages
2,292
Trophies
1
Location
Germany
Website
wiimmfi.de
XP
1,519
Country
Germany
mousex said:
Possible Feature Idea (if not already implemented): Extracting of all files from the game partition (especially as SNEEK requires them
tongue.gif
).

This is my private 2do list:
- sparse files on NTFS
- extract and compose ISOs
- diff and patch to transport changes for self made ISOs (e.g deliver "Mario Kart Custom" as small patch).
 

sylver78

Well-Known Member
Member
Joined
Oct 16, 2006
Messages
110
Trophies
1
Website
Visit site
XP
269
Country
France
Hello Wiimm !
Thanks for your great work
smile.gif

The latest version of wwt (compile by myself from svn) is not working well on my Macbook Pro under MacOS 10.6.2
frown.gif

It fails because of an assert here :
Code:
wwt -p /dev/disk2s1 ADD /Users/Sylver78/rs2.wdf -P
*****ÂÂwwt: Wiimms WBFS Tool v0.37a r762 mac - Dirk Clemens - 2010-03-28ÂÂ*****
!! wwt: ERROR #28 [FATAL ERROR] in wbfs_open_partition_param() @ libwbfs/libwbfs.c#394
!!ÂÂÂÂÂÂASSERTION FAILED !!!

commenting the line 394 ( ASSERT(!memcmp(head,&xh,sizeof(xh))); ) seems to solve the problem ...

To find the differences between head and xh, I added this code before the assert :
CODEÂÂÂÂÂÂÂÂ{
ÂÂÂÂÂÂÂÂÂÂÂÂu8 *data_1;
ÂÂÂÂÂÂÂÂÂÂÂÂu8 *data_2;
ÂÂÂÂÂÂÂÂÂÂÂÂu8 i;
ÂÂÂÂÂÂÂÂÂÂÂÂdata_2 = (u8 *)head;
ÂÂÂÂÂÂÂÂÂÂÂÂdata_1 = (u8 *)&xh;
ÂÂÂÂÂÂÂÂÂÂÂÂfor (i=0; i
 

Wiimm

Developer
OP
Member
Joined
Aug 11, 2009
Messages
2,292
Trophies
1
Location
Germany
Website
wiimmfi.de
XP
1,519
Country
Germany
wwt: Wiimms WBFS Tool v0.38b - 2010-03-28
A command line WBFS tool set for various os.

I have updated the tool. The first post with more details is updated too.

Download this EXPERIMENTAL release:Download of the previous stable version:
There is also a GUI available:
Highlights/bug fixes of this release:
  • Support of CISO-files (Compressed ISO)
    • CISO manage like WDF the holes, classic data compression is not supported. Because of the large block sizes CISO files are a little be larger than WDF files. But the data of CISO files are aligned to 32 KiB boundaries and may be used by USB loaderstoplay discs.
    • As input files the CISO format is automatically detected (analyse content and ignore filename) like ISO, WDF and WBFS before.
    • Writing to file can be controlled with the option --ciso. If the destination filename ends with the extension ".ciso" or ".wbi" the output file format is set to CISO too.
    • The tool wdf-dump dumps the data structure of WDF files and if CISO files.
    • The tool wdf-cat dumps the data of WDF files and if CISO files.
    .
Last changes:
Code:
wwt v0.37a r753 - 2010-03-26

- wwt: new command: ANALYZE: Analyze files and partitions for WBFS usage.
ÂÂ It try to find existing WBFS structures and make calculations for new WBFS.
ÂÂÂÂ- Check WBBS header
ÂÂÂÂ- Search INODE-INFO data (includes copy of WBFS header)
ÂÂÂÂ- Search discs for valid ID6 and MAGIC.
ÂÂÂÂ- Make example calculations for different sector sizes.
ÂÂ All results are printed as table with the main geometry data.
- wwt INIT:
ÂÂÂÂ- New option: --wss (Wbfs Sector Size): Defines the WBFS sector size.
ÂÂÂÂ- The option --sector-size (defines the HD sector size) was replaced by
ÂÂÂÂÂÂthe more handy --hss (Hd Sector Size). The old option is still allowed.
ÂÂÂÂ- If option --recover is set the command ANALYZE is called to determine
ÂÂÂÂÂÂthe values of --hss and --wss if not set before.
ÂÂ This points should complete WBFS recovering.
- For the ANALYZE command I have written a complete new function to calculate
ÂÂ the WBFS geometry (outsourcing of the calculation). If formatting a drive
ÂÂ this new function is called to compare the results with the old calculation.
ÂÂ If there are differences the program stops with an assertion failure.
ÂÂ ** Please tell me such errors! **

wwt v0.38a r769 - 2010-03-28

- Some dtime improvements.
- wwt+wit: Option -C is not longer the short version of option --trunc.
- CISO support (EXPERIMENTAL):
ÂÂÂÂ- wdf-dump will also dump the structure and memory map of CISO files.
ÂÂÂÂ- wdf-cat will copy the extended ISO to standard output.
ÂÂÂÂ- On all places where a plain ISO is allowed as source file a CISO is
ÂÂÂÂÂÂallowed too.
ÂÂÂÂ- To force CISO-output use option --ciso or file type ".ciso" or ".wbi".

wwt v0.38b r771 - 2010-03-28

- Bug fix for wwt INIT: Wrong assertion while comparing old and new
ÂÂ calculation for WBFS geometry.

Sources comes with GPL2. Checkout the sources directly from the SVN repository or browse the source.
 

Wiimm

Developer
OP
Member
Joined
Aug 11, 2009
Messages
2,292
Trophies
1
Location
Germany
Website
wiimmfi.de
XP
1,519
Country
Germany
@sylver78
to late for the realease of today
frown.gif


Only the first 10 bytes (WBFS_INODE_INFO_HEAD_CMP_SIZE) should be compared, not the whole data structure.


EDIT: Bug fix release v38b is out!
 

sylver78

Well-Known Member
Member
Joined
Oct 16, 2006
Messages
110
Trophies
1
Website
Visit site
XP
269
Country
France
Wiimm said:
try --io=3 to change file io from open() to fopen().
(1 for WBFS IO and 2 for ISO IO => 1+2=3 for both)
Actually it's even worse with --io=2 or 3 (around 4 hours)
frown.gif

I'm using an 16GB usb flash key, don't know if it's an important information.
 

sylver78

Well-Known Member
Member
Joined
Oct 16, 2006
Messages
110
Trophies
1
Website
Visit site
XP
269
Country
France
Wiimm said:
@sylver78:
flash is slow.
Flash is slow, but not that slow
wink.gif

I checked in the WBFS for MacOSX thread and I found that the speed problem was also a problem in previous versions of this tool !
One first change speed up the things a little bit :
http://gbatemp.net/index.php?showtopic=144...t&p=1886370

But the major speed up trick is to use /dev/rdiskXsX instead of /dev/diskXsX ( http://gbatemp.net/index.php?showtopic=144...t&p=1887020 ) ! the /dev/rdisk device is a character device (/dev/disk is a block device), and because of that, wwt is refusing to use it (there are several checks in the code about the device beeing a file or a block device ...). I guess that there are change to be done to be able to access disk content through a char device ...I'll try to dig deeper tomorrow
wink.gif
 

Wiimm

Developer
OP
Member
Joined
Aug 11, 2009
Messages
2,292
Trophies
1
Location
Germany
Website
wiimmfi.de
XP
1,519
Country
Germany
@sylver78
I have committed a version that allow char devices. My tests are poor because i have no good char device for testing.

If all works fine try this patch:
Code:
--- wbfs-interface.c
+++ wbfs-interface.c
@@ -145,7 +145,7 @@
ÂÂÂÂ else
ÂÂÂÂ {
ÂÂÂÂÂÂ#if __APPLE__
-ÂÂÂÂÂÂ static char dev_prefix[] = "disk";
+ÂÂÂÂÂÂ static char dev_prefix[] = "rdisk";
ÂÂÂÂÂÂ#else
ÂÂÂÂÂÂÂÂstatic char dev_prefix[] = "sd";
ÂÂÂÂÂÂ#endif
@@ -160,7 +160,7 @@
ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂif (!dent)
ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂbreak;
ÂÂÂÂÂÂÂÂÂÂÂÂ #ifdef _DIRENT_HAVE_D_TYPE
-ÂÂÂÂÂÂÂÂÂÂÂÂÂÂ if ( dent->d_type == DT_BLK
+ÂÂÂÂÂÂÂÂÂÂÂÂÂÂ if ( ( dent->d_type == DT_BLK || dent->d_type == DT_CHR )
ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ&& !memcmp(dent->d_name,dev_prefix,len_prefix))
ÂÂÂÂÂÂÂÂÂÂÂÂ #else
ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂif (!memcmp(dent->d_name,dev_prefix,len_prefix))
It searches /dev/rdisk* instead of /dev/disk* when using the --auto option.
 

sylver78

Well-Known Member
Member
Joined
Oct 16, 2006
Messages
110
Trophies
1
Website
Visit site
XP
269
Country
France
@Wiimm
When trying from latest svn, I'm now able to format & add using my /dev/rdisk (while it was complaining about not beeing a block device before), but the speed are still slow
blink.gif
.
When adding the patch in wbfs-interface.c, it's complaining about not finding any device with WBFS partitions, but if I'm adding the --io=3 option, guess what ? It's working perfectly and fast (6 minutes instead of 1 hour, running @ 9MB/s which is the max my flask key can handle).

Great work Wiimm !
bow.gif


So, should "--io=3" be the default on MacOS X ?
 

Wiimm

Developer
OP
Member
Joined
Aug 11, 2009
Messages
2,292
Trophies
1
Location
Germany
Website
wiimmfi.de
XP
1,519
Country
Germany
@sylver78

And what is when using --io=1 (or --io=2)? It is possible that the fopen() methods only improve the speed of character devices (fopen() uses user level buffering). And if this is true, I will force fopen() for char devices.
 

iranzo

Member
Newcomer
Joined
Mar 17, 2010
Messages
5
Trophies
0
Location
Valencia
XP
27
Country
United States
Wiimm said:
iranzo said:
¿are you interested in a spec file to automatically download source code and create an rpm from your code for easier installation?
Yes

Have a look at 'http://fpaste.org/r8xT/' it will last there for 1 day.

After you got curl, svn and gcc installed (as well as rpmbuild in order to have the command), you can issue:

rpmbuild -bb wwt.spec to get the source downloaded and the binary created.

Regards
Pablo
 

sylver78

Well-Known Member
Member
Joined
Oct 16, 2006
Messages
110
Trophies
1
Website
Visit site
XP
269
Country
France
Wiimm said:
@sylver78

And what is when using --io=1 (or --io=2)? It is possible that the fopen() methods only improve the speed of character devices (fopen() uses user level buffering). And if this is true, I will force fopen() for char devices.
It was working well with --io=1 if I remember well ... I'll confirm this later today.
 

Wiimm

Developer
OP
Member
Joined
Aug 11, 2009
Messages
2,292
Trophies
1
Location
Germany
Website
wiimmfi.de
XP
1,519
Country
Germany
@sylver
The next step is the --auto feature. Please connect a WBFS drive and try
Code:
wwt find -l /dev/disk*
wwt find -l /dev/rdisk*
wwt find -l /dev/disk*ÂÂ--io=3
wwt find -l /dev/rdisk* --io=3
If "/dev/rdiskX" is the char dev of the block dev "/dev/diskX" all four output lists should be equal (if ignoring filenames).


@iranzo
I have downloaded the code and will test it later. A fast code view offers that the title files in /usr/local/share/wwt not included. Thanx
 

sylver78

Well-Known Member
Member
Joined
Oct 16, 2006
Messages
110
Trophies
1
Website
Visit site
XP
269
Country
France
Wiimm said:
@sylver
The next step is the --auto feature. Please connect a WBFS drive and try
Code:
wwt find -l /dev/disk*
wwt find -l /dev/rdisk*
wwt find -l /dev/disk*ÂÂ--io=3
wwt find -l /dev/rdisk* --io=3
If "/dev/rdiskX" is the char dev of the block dev "/dev/diskX" all four output lists should be equal (if ignoring filenames).


@iranzo
I have downloaded the code and will test it later. A fast code view offers that the title files in /usr/local/share/wwt not included. Thanx

all of these 3 commands
Code:
wwt find -l /dev/disk*
wwt find -l /dev/disk*ÂÂ--io=3
wwt find -l /dev/rdisk* --io=3
give me this result :
Code:
typeÂÂwbfs d.usageÂÂÂÂsizeÂÂfile (sizes in MiB)
-----------------------------------------------
-ÂÂÂÂÂÂ--ÂÂÂÂÂÂÂÂ0ÂÂÂÂÂÂ 0ÂÂ/dev/disk0
-ÂÂÂÂÂÂ--ÂÂÂÂÂÂÂÂ0ÂÂÂÂÂÂ 0ÂÂ/dev/disk0s1
-ÂÂÂÂÂÂ--ÂÂÂÂÂÂÂÂ0ÂÂÂÂÂÂ 0ÂÂ/dev/disk0s2
BLOCKÂÂ--ÂÂÂÂÂÂÂÂ0ÂÂ 19508ÂÂ/dev/disk1
BLOCKÂÂ--ÂÂÂÂÂÂÂÂ0ÂÂÂÂÂÂ 0ÂÂ/dev/disk1s1
-ÂÂÂÂÂÂ--ÂÂÂÂÂÂÂÂ0ÂÂÂÂÂÂ 0ÂÂ/dev/disk1s2
BLOCKÂÂ--ÂÂÂÂÂÂÂÂ0ÂÂ 15320ÂÂ/dev/disk2
BLOCK WBFSÂÂÂÂÂÂ 0ÂÂ 15320ÂÂ/dev/disk2s1
or
Code:
typeÂÂwbfs d.usageÂÂÂÂsizeÂÂfile (sizes in MiB)
-----------------------------------------------
-ÂÂÂÂÂÂ--ÂÂÂÂÂÂÂÂ0ÂÂÂÂÂÂ 0ÂÂ/dev/rdisk0
-ÂÂÂÂÂÂ--ÂÂÂÂÂÂÂÂ0ÂÂÂÂÂÂ 0ÂÂ/dev/rdisk0s1
-ÂÂÂÂÂÂ--ÂÂÂÂÂÂÂÂ0ÂÂÂÂÂÂ 0ÂÂ/dev/rdisk0s2
CHARÂÂ --ÂÂÂÂÂÂÂÂ0ÂÂ 19508ÂÂ/dev/rdisk1
CHARÂÂ --ÂÂÂÂÂÂÂÂ0ÂÂÂÂÂÂ 0ÂÂ/dev/rdisk1s1
CHARÂÂ --ÂÂÂÂÂÂÂÂ0ÂÂ 19508ÂÂ/dev/rdisk1s2
CHARÂÂ --ÂÂÂÂÂÂÂÂ0ÂÂ 15320ÂÂ/dev/rdisk2
CHARÂÂWBFSÂÂÂÂÂÂ 0ÂÂ 15320ÂÂ/dev/rdisk2s1
this command
Code:
wwt find -l /dev/rdisk*
give me this (which is wrong)
CODEtypeÂÂwbfs d.usageÂÂÂÂsizeÂÂfile (sizes in MiB)
-----------------------------------------------
-ÂÂÂÂÂÂ--ÂÂÂÂÂÂÂÂ0ÂÂÂÂÂÂ 0ÂÂ/dev/rdisk0
-ÂÂÂÂÂÂ--ÂÂÂÂÂÂÂÂ0ÂÂÂÂÂÂ 0ÂÂ/dev/rdisk0s1
-ÂÂÂÂÂÂ--ÂÂÂÂÂÂÂÂ0ÂÂÂÂÂÂ 0ÂÂ/dev/rdisk0s2
CHARÂÂ --ÂÂÂÂÂÂÂÂ0ÂÂ 19508ÂÂ/dev/rdisk1
CHARÂÂ --ÂÂÂÂÂÂÂÂ0ÂÂÂÂÂÂ 0ÂÂ/dev/rdisk1s1
CHARÂÂ --ÂÂÂÂÂÂÂÂ0ÂÂ 19508ÂÂ/dev/rdisk1s2
CHARÂÂ --ÂÂÂÂÂÂÂÂ0ÂÂ 15320ÂÂ/dev/rdisk2
CHARÂÂ --ÂÂÂÂÂÂÂÂ0ÂÂ 15320ÂÂ/dev/rdisk2s1

BTW, --io=1 and --io=3 is working but --io=2 is not working.

Sylver78
 

Site & Scene News

Popular threads in this forum

General chit-chat
Help Users
    B @ btjunior: i love sigma meals, skibidi slicers!!