Hacking WIT: Wiimms ISO Tools / beta test

[Wiimm]

./wit EXTRACT blabla.wbfs/#0/DATA/sys/main.dol --dest stdout > to/the/path/you/want/the/file.bin

This could be a nice feature:
CODE# copy a single file to dest, if dest is '-': copy to stdout
wit copy isoimage/isopath/isofile dest

# copy a complete dir tree to dest
wit copy isoimage/isopath/ dest

# concat all files to stdout
wit cat isoimage/isopath/isofile ...

# list all files
wit list isoimage/isopath/

Better is to write a FUSE module ...

 

[Wiimm]

Just trying to test this out...what am I doing wrong?

I confirm a bug for CONVERT a wbfs/wdf to wia with the cygwin release. COPY seems to work. I will make more tests for other param combinations in the next days.

 

[W hat]

I agree, cygwin Wit copy F:\foo.wdf F:\foo.wia seems to work fine.

 

[Knocks]

Great tool, thanks!

 

[Wiimm]

I have committed v.1.23.beta1 to the beta branch. Reason: I have detected the function posix_fallocate(). And now wit is able to preallocate disc space to minimize the fragmentation in 3 levels:

change log:
- Preallocation of disc space implemented. Set --sparse or --defrag to change
ÂÂ default behavior of ISO, CISO and WBFS files.

# wit help copy
...
ÂÂ--sparseÂÂÂÂÂÂÂÂTry to write the image with maximum sparse effect and
ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂdisables defragmentation optimization (option --defrag).
ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂThe default is a balance between --sparse and --defrag.
ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂThis option has only impact for ISO, CISO and WBFS files.
ÂÂ--defragÂÂÂÂÂÂÂÂTry to write the image with minimal number of fragments
ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂand disables sparse optimization (option --sparse). The
ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂdefault is a balance between --sparse and --defrag. This
ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂoption has only impact for ISO, CISO and WBFS files.
...

Preallocation is also enabled for WDF and WIA files, but both are compact (have no holes) and so the options are needless.

Some statistics on WinXP/NTFS with Wii Sports/scrubbed:

-------------------------------------------
FileÂÂÂÂÂÂÂÂÂÂsizeÂÂ disc usageÂÂFragments
-------------------------------------------
sparse.isoÂÂ 4482 MBÂÂ 632 MBÂÂÂÂÂÂÂÂ868
default.isoÂÂ4482 MBÂÂ 632 MBÂÂÂÂÂÂÂÂ164
defrag.isoÂÂ 4482 MBÂÂ 632 MBÂÂÂÂÂÂÂÂ 15
-------------------------------------------
sparse.cisoÂÂ 636 MBÂÂ 632 MBÂÂÂÂÂÂ 3209
default.cisoÂÂ636 MBÂÂ 635 MBÂÂÂÂÂÂÂÂÂÂ4
defrag.cisoÂÂ 636 MBÂÂ 636 MBÂÂÂÂÂÂÂÂÂÂ1
-------------------------------------------
sparse.wbfsÂÂ 638 MBÂÂ 632 MBÂÂÂÂÂÂ 3036
default.wbfsÂÂ638 MBÂÂ 637 MBÂÂÂÂÂÂÂÂÂÂ3
defrag.wbfsÂÂ 638 MBÂÂ 638 MBÂÂÂÂÂÂÂÂÂÂ1
-------------------------------------------

The source is published under the license of GPL2. Checkout the sources directly from the SVN repository or browse the source. These URLs are different from the product version. Visit http://wit.wiimm.de/ for more information.

 

[giantpune]

thats great. this will make a lot of people happy

but what does "sparse.wbfs 638 MB 632 MB 3.036" mean? how does it create a partial fragment? isnt it kind of an all-or-nothing deal?

 

[Wiimm]

thats great. this will make a lot of people happy

but what does "sparse.wbfs 638 MB 632 MB 3.036" mean? how does it create a partial fragment? isnt it kind of an all-or-nothing deal?

It means: option --sparse, output to wbfs, file size 638 MB, disc usage 632 MB, 3036 ntfs fragments. It was only a first fast test. And I don't know if both options are really usefull.

It is possible to preallocate different areas of the file. For NTFS wit must preallocate before writing anything to get good results. It seems to be good to prealloc the largest block first (not implemented yet). So wit must guess which blocks to preallocate and it only knows about scrubbed sectors but nothing know about additionally zeroed areas.

--sparse: preallocate nothing.

default: preallocate used wii sectors only.

--defrag: preallocate the complete wbfs/ciso file after estimating the size. For ISO files wit tries to combine more sectors into one preallocation area.

btw: if you compile with "make test debug all" wit print out the prealloc rabfes (and much more).

 

[giantpune]

ahh thats right. i forgot you were german. The '.' means something totally different now.

 

[Wiimm]

Yes, sorry, cut+paste of a localized output.

 

[Wiimm]

I have improved the preallocation handling and committed again. The difference between between default and --defrag is nearly zero; WBFS and CISO files have both only 1 fragment (and perhaps a sparse area at the end to fill the last block) .

A side effect of preallocaton: Writing to NTFS is much faster.

 

[giantpune]

im having some issue with r2032. i did "wit copy" using the default setting trying to see the speed increase when writing to NTFS. as soon as i did that, the lights on my HDD started lighting up like it is being written to.

Now its 10 minutes later, the lights on the HDD have been lighting up for the entire time, so wit is doing something on my HDD. after the 10 minutes, it starts printing stuff out stdout. it is writing at 1.3MBs. before it was writing to this same HDD at ~30MBs. its doing the same thing when writing to FAT32. I havent tested writing to ext yet.

 

[Wiimm]

@giantpune
2 days ago i have a similar but not reproducible effect while using a relative full drive (12 of 40G free). I worked with enabled trace and I know that the preallocation call of about 1.5 GB takes all the time. Immediately doing the same command again works fine and fast and it never happens again. It seems for me as an NTFS cleanup trigger.

My question to you: What happen if your are doing it again?

 

[giantpune]

i tried it again on the same NTFS HDD and i get the same result. It is a 1TB drive full of movies, so it is all files > 700MB. And i have 159GB of free space. So, does this mean that first it must move some of these full 700MB movies to create a block of 4GB before it can write the file? And judging be the time of it, it is having to move 3 or 4 movies to make room.

And does it start the time for calculating the transfer speed after it has started writing the file? or does it start the time before it tries to allocate the space on the drive? because it seems that when i try to copy a larger game, it takes longer to start the actual copy and when it does, the speed is slower ( 0.7 MBs reported for a 4GB game ).

and what happens to the allocated disc space if i forcibly interrupt wit? when is the space freed? and who is responsible for unallocating it?

 

[Wiimm]

The allocated disc space should be linked to the file and if the file is deleted the space become free.

Can you recompile it with "make debug all" and run again and send me via mail the trace file?

 

[giantpune]

heres teh trace file created when using wit copy to copy a 4GB game from ext4 to my NTFS partition that has all those movies on it. It took something like 15-20 minutes
http://www.multiupload.com/G8ENAJ0STF

 

[Wiimm]

I can't see not so much because there are to less trace commands in the preallocate functions. I will change that tomorrow. What happens if you split for example with 500m or 1g? If you do so i wit will not try to prealloc 4.7g at once.

 

[giantpune]

if i split the game into smaller parts, or if i just copy a smaller game ( 0.25 GB ) it happens very fast

 

[Wiimm]

My guess: Windows/NTFS tries to allocate that ordered space in 1 fragment and must sometimes move some other blocks. Under Linux such files are fragmented. Perhaps I must preallocate smaller peaces for windows. I will implement an option to control the max prealloc size for experiments to find out an optimal value.

 

[giantpune]

im doing all this testing using ubuntu 10.04, ntfs3g, and USB HDD. windows is not involved at all.

 

[Wiimm]

that means it is a ntfs problem!?