Hacking WIT: Wiimms ISO Tools / beta test

Wiimm

Developer
OP
Member
Joined
Aug 11, 2009
Messages
2,292
Trophies
1
Location
Germany
Website
wiimmfi.de
XP
1,519
Country
Germany
Wiimm said:
that means it is a ntfs problem!?
For more tests I have switched to linux. And there we have similar problems.

Some examples: copy ssbb data partition, image made with "make debug test all" (new private version of wit, i will commit it later)

CODE# time ./wit cp -ovv --psel data ssbb.wdf pool/a.wbfs --sparse
...
ÂÂÂÂ 6902 MiB copied in 6:20, 18.1 MiB/secÂÂÂÂÂÂÂÂÂÂÂÂ
...
realÂÂÂÂ6m20.381s
userÂÂÂÂ0m8.693s
sysÂÂÂÂ0m29.563s
pool/a.wbfs -> /dev/sdb3 , 6908/6910 MiB, frag=7824

CODE# time ./wit cp -ovv --psel data ssbb.wdf pool/a.wbfs --pa-limit=100M
...
ÂÂ 0.930ÂÂCALL posix_fallocate(5,ÂÂ 400000,ÂÂ6300000 [ÂÂ99 MiB])
...
ÂÂ88.142ÂÂCALL posix_fallocate(5,ÂÂÂÂÂÂÂÂ0,ÂÂÂÂ 2800 [ÂÂ10 KiB])
ÂÂ88.143ÂÂ...
...
ÂÂÂÂ 6902 MiB copied in 7:38, 15.0 MiB/secÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ

realÂÂÂÂ7m38.642s
userÂÂÂÂ0m9.964s
sysÂÂÂÂ0m38.083s
pool/a.wbfs -> /dev/sdb3 , 6908/6910 MiB, frag=7769

CODE# time ./wit cp -ovv --psel data ssbb.wdf pool/a.wbfs --pa-limit=1G
...
ÂÂ 0.913ÂÂCALL posix_fallocate(5,ÂÂ 400000, 3a500000 [ 933 MiB])
ÂÂ 8.513ÂÂCALL posix_fallocate(5, 3a900000, 3a500000 [ 933 MiB])
ÂÂ26.785ÂÂCALL posix_fallocate(5, 74e00000, 3a500000 [ 933 MiB])
ÂÂ38.891ÂÂCALL posix_fallocate(5, af300000, 3a4c8000 [ 933 MiB])
ÂÂ50.390ÂÂCALL posix_fallocate(5, eabf0000, 31500000 [ 789 MiB])
ÂÂ60.408ÂÂCALL posix_fallocate(5,11c0f0000, 31500000 [ 789 MiB])
ÂÂ68.615ÂÂCALL posix_fallocate(5,14d5f0000, 31500000 [ 789 MiB])
ÂÂ78.573ÂÂCALL posix_fallocate(5,17eaf0000, 31260000 [ 786 MiB])
ÂÂ88.381ÂÂCALL posix_fallocate(5, e9958000,ÂÂ1088000 [ÂÂ17 MiB])
ÂÂ88.381ÂÂCALL posix_fallocate(5,ÂÂ 240000,ÂÂÂÂ10000 [ÂÂ64 KiB])
ÂÂ88.381ÂÂCALL posix_fallocate(5,ÂÂ 200000,ÂÂÂÂ 8000 [ÂÂ32 KiB])
ÂÂ88.381ÂÂCALL posix_fallocate(5,ÂÂÂÂÂÂÂÂ0,ÂÂÂÂ 2800 [ÂÂ10 KiB])
ÂÂ88.381ÂÂ...
....
ÂÂÂÂ 6902 MiB copied in 7:39, 15.0 MiB/secÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ

realÂÂÂÂ7m39.483s
userÂÂÂÂ0m10.130s
sysÂÂÂÂ0m38.411s
pool/a.wbfs -> /dev/sdb3 , 6908/6910 MiB, frag=7769

CODE# time ./wit cp -ovv --psel data ssbb.wdf pool/a.wbfs --pa-limit=10G
...
ÂÂ 0.717ÂÂCALL posix_fallocate(5,ÂÂ 400000, e93c8000 [3732 MiB])
ÂÂ50.431ÂÂCALL posix_fallocate(5, eabf0000, c5160000 [3153 MiB])
ÂÂ88.850ÂÂCALL posix_fallocate(5, e9958000,ÂÂ1088000 [ÂÂ17 MiB])
ÂÂ88.850ÂÂCALL posix_fallocate(5,ÂÂ 240000,ÂÂÂÂ10000 [ÂÂ64 KiB])
ÂÂ88.850ÂÂCALL posix_fallocate(5,ÂÂ 200000,ÂÂÂÂ 8000 [ÂÂ32 KiB])
ÂÂ88.850ÂÂCALL posix_fallocate(5,ÂÂÂÂÂÂÂÂ0,ÂÂÂÂ 2800 [ÂÂ10 KiB])
ÂÂ88.850ÂÂ...
...
ÂÂÂÂ 6902 MiB copied in 7:47, 14.8 MiB/secÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ

realÂÂÂÂ7m47.684s
userÂÂÂÂ0m10.179s
sysÂÂÂÂ0m38.757s
pool/a.wbfs -> /dev/sdb3 , 6908/6910 MiB, frag=7816


SUMMARY for Linux + ext3:

The preallocation consumes 88 seconds. The copy only time is in all 4 cases about 6:20. The number of fragments is always about 7800. This means that preallocation has no advantage on linux+ext3. I think, preallocation allocate disc space and fill it with zeros. Btw: fragmentation is not a ext2/ext3 problem because it is designed to handle with fragments.


Last not least the command I used:
CODEmake test debug all

for i in --sparse --pa-limit=100M --pa-limit=1G --pa-limit=10G; \
do \
ÂÂÂÂecho "##########ÂÂ$iÂÂ#########"; \
ÂÂÂÂrm -f pool/a.wbfs* pool/.a.wbfs*; \
ÂÂÂÂsync; \
ÂÂÂÂsleep 1; \
ÂÂÂÂtime ./wit cp -ovv --psel data srcdir/ssbb.wdf pool/a.wbfs $i; \
ÂÂÂÂ./print-disc-blocks.sh pool/a.wbfs \
done 2>&1 | tee a.log

# 'pool/' is my temp and working dir (it is ignored by svn)
 

oggzee

Well-Known Member
Member
Joined
Apr 11, 2009
Messages
2,333
Trophies
0
XP
188
Country
Slovenia
So, creating files under windows now works properly and doesn't create fragmented files anymore?
Very happy to hear that!
Good job wiimm!

and sorry if i was too annoying by constantly pointing this out, but it had to be fixed!
wink.gif
 

Wiimm

Developer
OP
Member
Joined
Aug 11, 2009
Messages
2,292
Trophies
1
Location
Germany
Website
wiimmfi.de
XP
1,519
Country
Germany
Here are some more statstics:

Code:
# linux, 64 bit, copy ssbb.wdf to ext3
6902 MiB copied in 6:20, 18.1 MiB/secÂÂno prealloc
6902 MiB copied in 7:38, 15.0 MiB/secÂÂprealloc in chunks a 100 MiB
6902 MiB copied in 7:39, 15.0 MiB/secÂÂprealloc in chunks a 1 GiB
6902 MiB copied in 7:47, 14.8 MiB/secÂÂcomplete prealloc

Code:
# linux, 64 bit, copy tra.wdf to fat32
3168 MiB copied in 1:28, 35.8 MiB/secÂÂno prealloc
3168 MiB copied in 3:02, 17.4 MiB/secÂÂprealloc in chunks a 100 MiB
3168 MiB copied in 3:02, 17.3 MiB/secÂÂprealloc in chunks a 1 GiB
3168 MiB copied in 3:02, 17.3 MiB/secÂÂcomplete prealloc

Code:
# linux, 64 bit, copy tra.wdf to ntfs
3168 MiB copied in 0:40, 78.0 MiB/secÂÂno prealloc
3168 MiB copied in 1:15, 41.9 MiB/secÂÂprealloc in chunks a 100 MiB
3168 MiB copied in 1:15, 41.8 MiB/secÂÂprealloc in chunks a 1 GiB
3168 MiB copied in 1:16, 41.5 MiB/secÂÂcomplete prealloc

Code:
win XP, 32 bit, copy tra.wdf to fat32
3168 MiB copied in 2:22, 22.2 MiB/secÂÂno prealloc
3168 MiB copied in 2:41, 19.6 MiB/secÂÂprealloc in chunks a 100 MiB
3168 MiB copied in 2:17, 23.0 MiB/secÂÂprealloc in chunks a 1 GiB
3168 MiB copied in 2:10, 24.3 MiB/secÂÂcomplete prealloc

Code:
# win XP, 32 bit, copy tra.wdf to ntfs
3168 MiB copied in 5:54,ÂÂ8.9 MiB/secÂÂno prealloc
3168 MiB copied in 3:36, 14.7 MiB/secÂÂprealloc in chunks a 100 MiB
3168 MiB copied in 4:17, 12.3 MiB/secÂÂprealloc in chunks a 1 GiB
3168 MiB copied in 4:12, 12.5 MiB/secÂÂcomplete prealloc

Conclusion:
  • Preallocation seems only good if windows writes to NTFS.
  • On linux preallocation slows down the copy operation without any advantage.
  • If preallocating don't split the preallocation chunks.
Does anyone have tools that shows the number of FAT fragments?
In general I'm searching for non interactive tools analyzing the fragmentation for each os+filesystem combi.
 

giantpune

Well-Known Member
Member
Joined
Apr 10, 2009
Messages
2,860
Trophies
0
XP
213
Country
United States
maybe pyfragtools is what you are looking for?

http://ubuntuforums.org/showthread.php?t=169551

check a single file
Code:
$ sudo '/home/jenna/pyfragtools-0.1/defrag' --analyze '/media/WiiFat500/wbfs/Guitar Hero 5_[SXEE52].wbf1' 
Building list of files to analyze... done!
ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ 
Analyze finished. 100.0 % fragmentation (1 files), 8.0 average frags/MB
Fragmented files:
7.96ÂÂÂÂ/media/WiiFat500/wbfs/Guitar Hero 5_[SXEE52].wbf1
========= COMPLETE ===========
Remaining Fragmented Files:
7.96ÂÂÂÂ/media/WiiFat500/wbfs/Guitar Hero 5_[SXEE52].wbf1
Frags/MB Before:ÂÂÂÂ 7.96
Frags/MB After: ÂÂÂÂ 7.96
Improvement:ÂÂÂÂÂÂÂÂ 0.0 %
===============================


check a directory
Code:
jenna@pc:~/pyfragtools-0.1$ sudo '/home/jenna/pyfragtools-0.1/defrag' --analyze '/media/WiiFat500/wbfs/' 
Building list of files to analyze... done!
ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ 
Analyze finished. 10.6 % fragmentation (11 files), 2.2 average frags/MB
Fragmented files:
7.96ÂÂÂÂ/media/WiiFat500/wbfs/Guitar Hero 5_[SXEE52].wbf1
7.73ÂÂÂÂ/media/WiiFat500/wbfs/New Super Mario Bros. Wii_[SMNE01].wbfs
2.12ÂÂÂÂ/media/WiiFat500/wbfs/Pikmin 2_[R92P01].wbfs
1.89ÂÂÂÂ/media/WiiFat500/wbfs/Guitar Hero - Warriors of Rock_[SXIP52].wbfs
1.88ÂÂÂÂ/media/WiiFat500/wbfs/Guitar Hero 5_[SXEE52].wbfs
1.49ÂÂÂÂ/media/WiiFat500/wbfs/Guitar Hero Aerosmith Custom - Mini Concerts Edition 2_[CGVEM2].wbfs
1.28ÂÂÂÂ/media/WiiFat500/wbfs/Guitar Hero Aerosmith Custom - Mini Concerts Edition_[CGVEMC].wbfs
0.00ÂÂÂÂ/media/WiiFat500/wbfs/Rock Band Green Day Rock_[R36E69].wbfs
0.00ÂÂÂÂ/media/WiiFat500/wbfs/RFPP01.wbfs
0.00ÂÂÂÂ/media/WiiFat500/wbfs/.____deleteme.wbfs.olOsd1.tmp
...
========= COMPLETE ===========
Remaining Fragmented Files:
0.00ÂÂÂÂ/media/WiiFat500/wbfs/James Bond - Golden Eye_[SJBE52].wbfs
0.00ÂÂÂÂ/media/WiiFat500/wbfs/.____deleteme.wbfs.olOsd1.tmp
0.00ÂÂÂÂ/media/WiiFat500/wbfs/RFPP01.wbfs
0.00ÂÂÂÂ/media/WiiFat500/wbfs/Rock Band Green Day Rock_[R36E69].wbfs
1.28ÂÂÂÂ/media/WiiFat500/wbfs/Guitar Hero Aerosmith Custom - Mini Concerts Edition_[CGVEMC].wbfs
1.49ÂÂÂÂ/media/WiiFat500/wbfs/Guitar Hero Aerosmith Custom - Mini Concerts Edition 2_[CGVEM2].wbfs
1.88ÂÂÂÂ/media/WiiFat500/wbfs/Guitar Hero 5_[SXEE52].wbfs
1.89ÂÂÂÂ/media/WiiFat500/wbfs/Guitar Hero - Warriors of Rock_[SXIP52].wbfs
2.12ÂÂÂÂ/media/WiiFat500/wbfs/Pikmin 2_[R92P01].wbfs
7.73ÂÂÂÂ/media/WiiFat500/wbfs/New Super Mario Bros. Wii_[SMNE01].wbfs
7.96ÂÂÂÂ/media/WiiFat500/wbfs/Guitar Hero 5_[SXEE52].wbf1
Frags/MB Before:ÂÂÂÂ 24.35
Frags/MB After: ÂÂÂÂ 24.35
Improvement:ÂÂÂÂÂÂÂÂ 0.0 %
===============================
 

Wiimm

Developer
OP
Member
Joined
Aug 11, 2009
Messages
2,292
Trophies
1
Location
Germany
Website
wiimmfi.de
XP
1,519
Country
Germany
I have updated the beta branch. New features:
  • If preallocating the largest block is preallocated first.
  • The new option --pa-limit=size sets a limit for the allocation blocks. See "wit help copy" for details.
@giantpune:
thanx, but only if the tool can show the number of fragments without defragmenting
wink.gif
 

giantpune

Well-Known Member
Member
Joined
Apr 10, 2009
Messages
2,860
Trophies
0
XP
213
Country
United States
its not defragmenting anything. using --analyze it is only reading. thats why the numbers are the same before and after.

theres also the perl script here http://forums.gentoo.org/viewtopic-p-31114...t-.html#3111409. it doesnt print out as much info as youd probably like, but you can always add more print() in there for more details

heres a quick bit that prints the filename and the number of fragments..
CODE#!/usr/bin/perl -w

#this script search for frag on a fs
use strict;

#number of files
my $files = 0;
#number of fragment
my $fragments = 0;
#number of fragmented files
my $fragfiles = 0;

#search fs for all file
open (FILES, "find " . $ARGV[0] . " -xdev -type f |");

while (defined (my $file = )) {
ÂÂÂÂÂÂÂÂ#quote some chars in filename
ÂÂÂÂÂÂÂÂ$file =~ s/!/\\!/g;
ÂÂÂÂÂÂÂÂ$file =~ s/#/\\#/g;
ÂÂÂÂÂÂÂÂ$file =~ s/&/\\&/g;
ÂÂÂÂÂÂÂÂ$file =~ s/>/\\>/g;
ÂÂÂÂÂÂÂÂ$file =~ s/ 1) {
ÂÂÂÂÂÂÂÂÂÂÂÂprint ( $file . "\t" . $fragment ."\n" );
ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ$fragfiles++;
ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ}
ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ$files++;
ÂÂÂÂÂÂÂÂ} else {
ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂprint ("$res : not understand for $file.\n");
ÂÂÂÂÂÂÂÂ}
ÂÂÂÂÂÂÂÂclose (FRAG);
}
close (FILES);

print ( $fragfiles / $files * 100 . "% non contiguous files, " . $fragments / $files . " average fragments.\n");
 

giantpune

Well-Known Member
Member
Joined
Apr 10, 2009
Messages
2,860
Trophies
0
XP
213
Country
United States
this is the output from the perl script. it seems to find the exact same files as the one that allows 200 fragments/GB
Code:
$ sudo '/home/jenna/Desktop/perlFragChecker.pl' '/media/WiiFat500/wbfs' 
/media/WiiFat500/wbfs/Rock Band Green Day Rock_[R36E69].wbfsÂÂÂÂ16
/media/WiiFat500/wbfs/RFPP01.wbfsÂÂÂÂ2
/media/WiiFat500/wbfs/Guitar Hero 5_[SXEE52].wbf1ÂÂÂÂ48
/media/WiiFat500/wbfs/Guitar Hero - Warriors of Rock_[SXIP52].wbfsÂÂÂÂ7732
/media/WiiFat500/wbfs/Guitar Hero 5_[SXEE52].wbfsÂÂÂÂ7685
/media/WiiFat500/wbfs/Pikmin 2_[R92P01].wbfsÂÂÂÂ4176
/media/WiiFat500/wbfs/New Super Mario Bros. Wii_[SMNE01].wbfsÂÂÂÂ4175
/media/WiiFat500/wbfs/James Bond - Golden Eye_[SJBE52].wbfsÂÂÂÂ2
/media/WiiFat500/wbfs/Guitar Hero Aerosmith Custom - Mini Concerts Edition_[CGVEMC].wbfsÂÂÂÂ4608
/media/WiiFat500/wbfs/Guitar Hero Aerosmith Custom - Mini Concerts Edition 2_[CGVEM2].wbfsÂÂÂÂ4608
/media/WiiFat500/wbfs/.____deleteme.wbfs.olOsd1.tmpÂÂÂÂ3
10.5769230769231% non contiguous files, 318.730769230769 average fragments.

does this mean that those 200 frags are really negligible?
 

oggzee

Well-Known Member
Member
Joined
Apr 11, 2009
Messages
2,333
Trophies
0
XP
188
Country
Slovenia
I haven't read in detail the last couple of pages, but I guess you're having problem with preallocation on linux?
In wbfs_file, preallocation is #ifdefed only for windows, because that's the only os that needs such measures to prevent fragmentation, and on windows preallocating has no performance impact, only positive effects.
 

Wiimm

Developer
OP
Member
Joined
Aug 11, 2009
Messages
2,292
Trophies
1
Location
Germany
Website
wiimmfi.de
XP
1,519
Country
Germany
Here is my decision, discuss it if you like:[*] The test options --sparse --defrag and --pa-limit=size will be removed.[*] I implement a new option --prealloc with an optional parameter.[*] --prealloc=0 disables preallocation. This is the default for NON-CYGWIN.[*] --prealloc=1 enables an smart preallocation ignoring unused blocks.[*] --prealloc=2 or simple --prealloc preallocates the complete CISO or WBFS file. This is the default for CYGWIN because --prealloc=1 doesn't work like thought if the sparse holes are small.[*] Modes '1' and '2' are only different for WBFS and CISO files. ISO files will always use '1', and WDF and WIA files always '2' if nit disabled.[*] Later I will implement the preallocation for non image files (e.g. extracted files).
 

oggzee

Well-Known Member
Member
Joined
Apr 11, 2009
Messages
2,333
Trophies
0
XP
188
Country
Slovenia
Wiimm said:
--prealloc=2 or simple --prealloc preallocates the complete CISO or WBFS file. This is the default for CYGWIN because --prealloc=1 doesn't work like thought if the sparse holes are small.
sounds good to me.
 

Wiimm

Developer
OP
Member
Joined
Aug 11, 2009
Messages
2,292
Trophies
1
Location
Germany
Website
wiimmfi.de
XP
1,519
Country
Germany
Ok, I have committed v1.23,beta4 (nice digit order) inti the beta branch.

New option: --prealloc
Code:
--prealloc [=mode]
ÂÂÂÂÂÂ This option enables or disables the disc space preallocation. If
ÂÂÂÂÂÂ enabled the tools try to allocate disc space for the new files before
ÂÂÂÂÂÂ writing the data. This reduces the fragmentation but also disables the
ÂÂÂÂÂÂ sparse effect for prealocated areas.
ÂÂÂÂÂÂÂÂ The optional parameter decides the preallocation modus: OFF (or 0),
ÂÂÂÂÂÂ SMART (or 1), ALL (or 2). If no parameter is set, ALL is used.
ÂÂÂÂÂÂÂÂ Mode 'OFF' disables the preallocation. This is the default for all
ÂÂÂÂÂÂ non Cygwin releases because preallocation has only advantages on
ÂÂÂÂÂÂ Windows systems.
ÂÂÂÂÂÂÂÂ Mode 'SMART' looks into the source disc to find out the writing
ÂÂÂÂÂÂ areas. SMART is only avalable for ISO, CISO and WBFS file types. For
ÂÂÂÂÂÂ other file types ALL is used instead.
ÂÂÂÂÂÂÂÂ Mode 'ALL' preallocate the whole destination file. This is the
ÂÂÂÂÂÂ default for Cygwin. Because of the large holes in plain ISO images, the
ÂÂÂÂÂÂ SMART mode is used for ISOs instead.
 

W hat

Rhythm Heaven Fan
Member
Joined
Feb 28, 2007
Messages
632
Trophies
1
XP
697
Country
United States
I've used WIA enough to report in. After testing 12 games, I get compression ratios of 50-60% almost every time with 7-zip+LZMA1 (Ultra). Some games do still only compress to 80-90%, but that's still great. I love WIA.
 

Wiimm

Developer
OP
Member
Joined
Aug 11, 2009
Messages
2,292
Trophies
1
Location
Germany
Website
wiimmfi.de
XP
1,519
Country
Germany
wiimms-tools.png


WIT: Wiimms ISO Tools v1.23a.beta5 - 2010-12-03
Beta test version, for EXPERIMENTAL use only.

WIT (Wiimms ISO Tools) is a set of command line tools to manipulate Wii ISO images and WBFS containers. The two main tools are called wit (Wiimms ISO Tool) and wwt (Wiimms WBFS Tool, the WBFS manager).

For productive versions see the main WIT thread or visit http://wit.wiimm.de/.

Download this EXPERIMENTAL release:Highlights of this release:
  • Preallocation to avoid fragmentation
    I have implemented disc space preallocation. Under Windows it can avoid fragmentation. Therfor it is only enabled for Cygwin by default. It can be disabled with --prealloc=0 or enabled with --prealloc. See http://wit.wiimm.de/opt/prealloc for more details.
    .
  • ID wildcards for some wwt commands
    The wwt commands EXTRACT + REMOVE + TOUCH + VERIFY wil support now ID wildcards to select different discs. See http://wit.wiimm.de/info/disc-id.html for more details.
    The needed preallocation function is not avaialable for Mac. So Mac will ignore --prealloc.
    .
  • wwt EXTRACT rewritten
    "wwt EXTRACT" is now able to patch while extracting and to create WIA files like "wit COPY". It support also the options --fst --pmode --psel --raw --files and --sneek. http://wit.wiimm.de/wwt/cmd-extract.html for more details.
    .
Change log:


Code:
wit v1.23a.beta5 r2068 - 2010-12-03

- Preallocation of disc space implemented. It is only enabled for Cygwin by
ÂÂ default. It can be disabled with --prealloc=0 or enabled with --prealloc.
ÂÂ The needed preallocation function is not avaialable for mac.
- wwt EXTRACT + REMOVE + TOUCH + VERIFY:
ÂÂ These commands accept now ID6 pattern like '...P' or '+1'. Each '.' is a
ÂÂ wildcard that matches any single character. A '+' matches any number of
ÂÂ characters. For details see http://wit.wiimm.de/info/disc-id.html
- wwt EXTRACT is now able to patch while extracting and to create WIA files.
ÂÂ It support also the options --fst --pmode --psel --raw --files and --sneek.


Known bugs:

- The GameCube support is in development and EXPERIMENTAL!
- wit COPY --raw: Copying an image in raw mode to a WBFS file is buggy.
- There are new problems while reading from non seekable sources (e.g. pipes).
ÂÂ The reason is the complete new and more intelligent wiidisc lib which makes
ÂÂ more lookups ==> The best is to use version 1.01c for pipes.
- No WIA/bzip2 support for MAC/x86_64. (bug in the Mac bzip2 library)


Missing docu:

- wit commands FILES + EXTRACT => see built in help
- Tool wdf => see built in help

The source is published under the license of GPL2. Checkout the sources directly from the SVN repository or browse the source. These URLs are different from the product version. Visit http://wit.wiimm.de/ for more information.
 

Wiimm

Developer
OP
Member
Joined
Aug 11, 2009
Messages
2,292
Trophies
1
Location
Germany
Website
wiimmfi.de
XP
1,519
Country
Germany
wiimms-tools.png


WIT: Wiimms ISO Tools v1.26a.beta1 - 2011-01-11
Beta test version, for EXPERIMENTAL use only.

WIT (Wiimms ISO Tools) is a set of command line tools to manipulate Wii ISO images and WBFS containers. The two main tools are called wit (Wiimms ISO Tool) and wwt (Wiimms WBFS Tool, the WBFS manager).

For productive versions see the main WIT thread or visit http://wit.wiimm.de/.

Download this EXPERIMENTAL release:Highlights of this release:
  • I have made some changes in the make system to support local libraries with own makefiles (libbz2.a), option -static and cross compiling (Linux/i386 on Linux/x86_64). Hope that all work as usual.
    Please test it!
    .

    .

Change log:


Code:
wit v1.26a.beta1 r2156 - 2011-01-11

- I have made some changes in the make system to support local libraries with
ÂÂ own makefiles (libbz2.a), option -static and cross compiling (Linux/i386 on
ÂÂ Linux/x86_64). Hope that all work as usual.
ÂÂÂÂ- BZIP2 support is now included as source and should work for all systems.
ÂÂÂÂÂÂThis enables WIA/bzip2 support for Mac/x86_64.
ÂÂÂÂ- The distributions of Linux/i386 and Linux/X86_64 are linked static to
ÂÂÂÂÂÂto avoid GLIBC version issues (e.g. fallocate() support).
ÂÂÂÂ- On Linux/x86_64: Enter "M32=1 make ..." to create i386 binaries.
ÂÂÂÂÂÂI386 libraries and include files are needed.
- "wit COPY/CONVERT/EXTRACT/DIFF" support option --sections.


Known bugs:

- The GameCube support is in development and EXPERIMENTAL!
ÂÂÂÂ- GameCube conversion to WIA fails.
- There are new problems while reading from non seekable sources (e.g. pipes).
ÂÂ The reason is the complete new and more intelligent wiidisc lib which makes
ÂÂ more lookups ==> The best is to use version 1.01c for pipes.

The source is published under the license of GPL2. Checkout the sources directly from the SVN repository or browse the source. These URLs are different from the product version. Visit http://wit.wiimm.de/ for more information.
 

giantpune

Well-Known Member
Member
Joined
Apr 10, 2009
Messages
2,860
Trophies
0
XP
213
Country
United States
any chance you could change the makefile to support the jobserver? in prior revisions, "make -j4" works as expected. with whatever changes were made to add the bzib library, it causes make to spit out some crap about the jobserver.
Code:
$ make -j6
***ÂÂcreate templates.sedÂÂÂÂ 
***ÂÂÂÂ asm ssl-asm.oÂÂÂÂÂÂÂÂ 
*** ----------ÂÂENTER ./src/libbz2/ÂÂ----------ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ 
***ÂÂcreate install.shÂÂÂÂÂÂÂÂ
***ÂÂcreate load-titles.shÂÂÂÂ
make[1]: warning: jobserver unavailable: using -j1.ÂÂAdd `+' to parent make rule.
*****ÂÂobject blocksort.oÂÂÂÂÂÂ 
***ÂÂcreate version.hÂÂÂÂÂÂÂÂ 
***ÂÂobject gen-ui.oÂÂÂÂÂÂÂÂÂÂ
***ÂÂobject LzFind.oÂÂÂÂÂÂÂÂÂÂ
***ÂÂobject Lzma2Dec.oÂÂÂÂÂÂÂÂ
***ÂÂobject Lzma2Enc.oÂÂÂÂÂÂÂÂ
***ÂÂobject LzmaDec.oÂÂÂÂÂÂÂÂ 
***ÂÂobject LzmaEnc.oÂÂÂÂÂÂÂÂ 
***ÂÂobject cert.oÂÂÂÂÂÂÂÂÂÂÂÂ
***ÂÂobject dclib-utf8.oÂÂÂÂÂÂ
***ÂÂobject debug.oÂÂÂÂÂÂÂÂÂÂ 
***ÂÂobject file-formats.oÂÂÂÂ
***ÂÂobject iso-interface.oÂÂ 
***ÂÂobject lib-bzip2.oÂÂÂÂÂÂ 
*****ÂÂobject huffman.oÂÂÂÂÂÂÂÂ 
***ÂÂobject lib-ciso.oÂÂÂÂÂÂÂÂ
*****ÂÂobject crctable.oÂÂÂÂÂÂÂÂ
*****ÂÂobject randtable.oÂÂÂÂÂÂ 
***ÂÂobject lib-file.oÂÂÂÂÂÂÂÂ
*****ÂÂobject compress.oÂÂÂÂÂÂÂÂ
***ÂÂobject lib-lzma.oÂÂÂÂÂÂÂÂ
***ÂÂobject lib-sf.oÂÂÂÂÂÂÂÂÂÂ
***ÂÂobject lib-std.oÂÂÂÂÂÂÂÂ 
src/lib-std.c: In function ‘SetupLib’:
src/lib-std.c:472: warning: ignoring return value of ‘getcwd’, declared with attribute warn_unused_result
***ÂÂobject lib-wdf.oÂÂÂÂÂÂÂÂ 
***ÂÂobject lib-wia.oÂÂÂÂÂÂÂÂ 
*****ÂÂobject decompress.oÂÂÂÂÂÂ
***ÂÂobject libwbfs.oÂÂÂÂÂÂÂÂ 
***ÂÂobject match-pattern.oÂÂ 
***ÂÂobject patch.oÂÂÂÂÂÂÂÂÂÂ 
***ÂÂobject rijndael.oÂÂÂÂÂÂÂÂ
***ÂÂobject sha1_one.oÂÂÂÂÂÂÂÂ
***ÂÂobject sha1dgst.oÂÂÂÂÂÂÂÂ
***ÂÂobject titles.oÂÂÂÂÂÂÂÂÂÂ
***ÂÂobject tools.oÂÂÂÂÂÂÂÂÂÂ 
*****ÂÂobject bzlib.oÂÂÂÂÂÂÂÂÂÂ 
***ÂÂobject ui.oÂÂÂÂÂÂÂÂÂÂÂÂÂÂ
***ÂÂobject wbfs-interface.oÂÂ
src/wbfs-interface.c: In function ‘ScanPartitions’:
src/wbfs-interface.c:217: warning: ignoring return value of ‘fgets’, declared with attribute warn_unused_result
***ÂÂobject wiidisc.oÂÂÂÂÂÂÂÂ 
***ÂÂobject wit-mix.oÂÂÂÂÂÂÂÂ 
*****ÂÂÂÂÂÂar libbz2.aÂÂÂÂÂÂÂÂÂÂ
***ÂÂobject wtest.oÂÂÂÂÂÂÂÂÂÂ 
*****ÂÂranlib libbz2.aÂÂÂÂÂÂÂÂÂÂ
*** ----------ÂÂLEAVE ./src/libbz2/ÂÂ----------ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ 
***ÂÂhelper gen-uiÂÂÂÂÂÂÂÂÂÂÂÂ
*** +object wdf.oÂÂÂÂÂÂÂÂÂÂÂÂ 
*** +object wit.oÂÂÂÂÂÂÂÂÂÂÂÂ 
***ÂÂÂÂtool wtestÂÂÂÂÂÂÂÂÂÂÂÂ 
*** +object wwt.oÂÂÂÂÂÂÂÂÂÂÂÂ 
***ÂÂÂÂtool wdfÂÂÂÂÂÂÂÂÂÂÂÂÂÂ 
***ÂÂÂÂlink wdf -> wdf-catÂÂÂÂ
***ÂÂÂÂlink wdf -> wdf-dumpÂÂ 
***ÂÂÂÂtool witÂÂÂÂÂÂÂÂÂÂÂÂÂÂ wit-mix.o
***ÂÂÂÂtool wwtÂÂÂÂÂÂÂÂÂÂÂÂÂÂ 
HINT: try 'make help'

in your makefile, you can change this section
Code:
$(LIB_FILES):
ÂÂÂÂ@printf "$(LOGFORMAT)" "----------ÂÂENTER "./src/$(patsubst %.a,%,$@)/"ÂÂ----------" "" ""
ÂÂÂÂ@XFLAGS="$(XFLAGS)" $(MAKE) --no-print-directory -C "src/$(patsubst %.a,%,$@)"
ÂÂÂÂ@printf "$(LOGFORMAT)" "----------ÂÂLEAVE "./src/$(patsubst %.a,%,$@)/"ÂÂ----------" "" ""
ÂÂÂÂ@cp -p "src/$(patsubst %.a,%,$@)/$@" .

change "make" to "$(MAKE)" and the jobserver stuff is working again.

other than this, it seems to work good
smile.gif
 

giantpune

Well-Known Member
Member
Joined
Apr 10, 2009
Messages
2,860
Trophies
0
XP
213
Country
United States
oops... spoke too soon. it build fine on windows and linux x64. but it fails to build on OSx with this
Code:
l$ make -j2
***ÂÂcreate templates.sedÂÂÂÂ 
***ÂÂÂÂ asm ssl-asm.oÂÂÂÂÂÂÂÂ 
*** ----------ÂÂENTER ./src/libbz2/ÂÂ----------ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ 
*****ÂÂobject blocksort.oÂÂÂÂÂÂ 
***ÂÂcreate install.shÂÂÂÂÂÂÂÂ
*****ÂÂobject huffman.oÂÂÂÂÂÂÂÂ 
*****ÂÂobject crctable.oÂÂÂÂÂÂÂÂ
*****ÂÂobject randtable.oÂÂÂÂÂÂ 
*****ÂÂobject compress.oÂÂÂÂÂÂÂÂ
*****ÂÂobject decompress.oÂÂÂÂÂÂ
*****ÂÂobject bzlib.oÂÂÂÂÂÂÂÂÂÂ 
***ÂÂcreate load-titles.shÂÂÂÂ
***ÂÂcreate version.hÂÂÂÂÂÂÂÂ 
***ÂÂobject gen-ui.oÂÂÂÂÂÂÂÂÂÂ
*****ÂÂÂÂÂÂar libbz2.aÂÂÂÂÂÂÂÂÂÂ
*****ÂÂranlib libbz2.aÂÂÂÂÂÂÂÂÂÂ
*** ----------ÂÂLEAVE ./src/libbz2/ÂÂ----------ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ 
***ÂÂobject LzFind.oÂÂÂÂÂÂÂÂÂÂ
***ÂÂobject Lzma2Dec.oÂÂÂÂÂÂÂÂ
***ÂÂobject Lzma2Enc.oÂÂÂÂÂÂÂÂ
***ÂÂobject LzmaDec.oÂÂÂÂÂÂÂÂ 
***ÂÂobject LzmaEnc.oÂÂÂÂÂÂÂÂ 
***ÂÂobject cert.oÂÂÂÂÂÂÂÂÂÂÂÂ
***ÂÂobject dclib-utf8.oÂÂÂÂÂÂ
***ÂÂobject debug.oÂÂÂÂÂÂÂÂÂÂ 
***ÂÂobject file-formats.oÂÂÂÂ
***ÂÂobject iso-interface.oÂÂ 
***ÂÂobject lib-bzip2.oÂÂÂÂÂÂ 
***ÂÂobject lib-ciso.oÂÂÂÂÂÂÂÂ
***ÂÂobject lib-file.oÂÂÂÂÂÂÂÂ
***ÂÂobject lib-lzma.oÂÂÂÂÂÂÂÂ
***ÂÂobject lib-sf.oÂÂÂÂÂÂÂÂÂÂ
***ÂÂobject lib-std.oÂÂÂÂÂÂÂÂ 
***ÂÂobject lib-wdf.oÂÂÂÂÂÂÂÂ 
***ÂÂobject lib-wia.oÂÂÂÂÂÂÂÂ 
***ÂÂobject libwbfs.oÂÂÂÂÂÂÂÂ 
***ÂÂobject match-pattern.oÂÂ 
***ÂÂobject patch.oÂÂÂÂÂÂÂÂÂÂ 
***ÂÂobject rijndael.oÂÂÂÂÂÂÂÂ
***ÂÂobject sha1_one.oÂÂÂÂÂÂÂÂ
***ÂÂobject sha1dgst.oÂÂÂÂÂÂÂÂ
***ÂÂobject titles.oÂÂÂÂÂÂÂÂÂÂ
***ÂÂobject tools.oÂÂÂÂÂÂÂÂÂÂ 
***ÂÂobject ui.oÂÂÂÂÂÂÂÂÂÂÂÂÂÂ
***ÂÂobject wbfs-interface.oÂÂ
***ÂÂobject wiidisc.oÂÂÂÂÂÂÂÂ 
***ÂÂobject wit-mix.oÂÂÂÂÂÂÂÂ 
***ÂÂobject wtest.oÂÂÂÂÂÂÂÂÂÂ 
***ÂÂhelper gen-uiÂÂÂÂÂÂÂÂÂÂÂÂ
Undefined symbols:
ÂÂ"___strcat_chk", referenced from:
ÂÂÂÂÂÂ_bzopen_or_bzdopen in libbz2.a(bzlib.o)
ÂÂÂÂÂÂ_bzopen_or_bzdopen in libbz2.a(bzlib.o)
ÂÂ"___stack_chk_guard", referenced from:
ÂÂÂÂÂÂ___stack_chk_guard$non_lazy_ptr in libbz2.a(bzlib.o)
ÂÂÂÂÂÂ___stack_chk_guard$non_lazy_ptr in libbz2.a(compress.o)
ÂÂÂÂÂÂ___stack_chk_guard$non_lazy_ptr in libbz2.a(blocksort.o)
ÂÂÂÂ (maybe you meant: ___stack_chk_guard$non_lazy_ptr)
ÂÂ"___stack_chk_fail", referenced from:
ÂÂÂÂÂÂ_bzopen_or_bzdopen in libbz2.a(bzlib.o)
ÂÂÂÂÂÂ_BZ2_compressBlock in libbz2.a(compress.o)
ÂÂÂÂÂÂ_BZ2_blockSort in libbz2.a(blocksort.o)
ÂÂ"_fopen$UNIX2003", referenced from:
ÂÂÂÂÂÂ_bzopen_or_bzdopen in libbz2.a(bzlib.o)
ÂÂ"_fdopen$UNIX2003", referenced from:
ÂÂÂÂÂÂ_bzopen_or_bzdopen in libbz2.a(bzlib.o)
ld: symbol(s) not found
collect2: ld returned 1 exit status
make: *** [gen-ui] Error 1
make: *** Waiting for unfinished jobs....
 

giantpune

Well-Known Member
Member
Joined
Apr 10, 2009
Messages
2,860
Trophies
0
XP
213
Country
United States
mine says
Mac OS X 10.6.4 (10f569)
kernel version 10.0.0
if you have any ideas about what i may have done to break it, ill try them. but all i did is checkout the source and type make.
 

Site & Scene News

Popular threads in this forum

General chit-chat
Help Users
    SylverReZ @ SylverReZ: