The preallocation consumes 88 seconds. The copy only time is in all 4 cases about 6:20. The number of fragments is always about 7800. This means that preallocation has no advantage on linux+ext3. I think, preallocation allocate disc space and fill it with zeros. Btw: fragmentation is not a ext2/ext3 problem because it is designed to handle with fragments.
Last not least the command I used:
CODEmake test debug all
for i in --sparse --pa-limit=100M --pa-limit=1G --pa-limit=10G; \
do \
ÂÂÂÂecho "##########ÂÂ$iÂÂ#########"; \
ÂÂÂÂrm -f pool/a.wbfs* pool/.a.wbfs*; \
ÂÂÂÂsync; \
ÂÂÂÂsleep 1; \
ÂÂÂÂtime ./wit cp -ovv --psel data srcdir/ssbb.wdf pool/a.wbfs $i; \
ÂÂÂÂ./print-disc-blocks.sh pool/a.wbfs \
done 2>&1 | tee a.log
# 'pool/' is my temp and working dir (it is ignored by svn)
# linux, 64 bit, copy ssbb.wdf to ext3
6902 MiB copied in 6:20, 18.1 MiB/secÂÂno prealloc
6902 MiB copied in 7:38, 15.0 MiB/secÂÂprealloc in chunks a 100 MiB
6902 MiB copied in 7:39, 15.0 MiB/secÂÂprealloc in chunks a 1 GiB
6902 MiB copied in 7:47, 14.8 MiB/secÂÂcomplete prealloc
Code:
# linux, 64 bit, copy tra.wdf to fat32
3168 MiB copied in 1:28, 35.8 MiB/secÂÂno prealloc
3168 MiB copied in 3:02, 17.4 MiB/secÂÂprealloc in chunks a 100 MiB
3168 MiB copied in 3:02, 17.3 MiB/secÂÂprealloc in chunks a 1 GiB
3168 MiB copied in 3:02, 17.3 MiB/secÂÂcomplete prealloc
Code:
# linux, 64 bit, copy tra.wdf to ntfs
3168 MiB copied in 0:40, 78.0 MiB/secÂÂno prealloc
3168 MiB copied in 1:15, 41.9 MiB/secÂÂprealloc in chunks a 100 MiB
3168 MiB copied in 1:15, 41.8 MiB/secÂÂprealloc in chunks a 1 GiB
3168 MiB copied in 1:16, 41.5 MiB/secÂÂcomplete prealloc
Code:
win XP, 32 bit, copy tra.wdf to fat32
3168 MiB copied in 2:22, 22.2 MiB/secÂÂno prealloc
3168 MiB copied in 2:41, 19.6 MiB/secÂÂprealloc in chunks a 100 MiB
3168 MiB copied in 2:17, 23.0 MiB/secÂÂprealloc in chunks a 1 GiB
3168 MiB copied in 2:10, 24.3 MiB/secÂÂcomplete prealloc
Code:
# win XP, 32 bit, copy tra.wdf to ntfs
3168 MiB copied in 5:54,ÂÂ8.9 MiB/secÂÂno prealloc
3168 MiB copied in 3:36, 14.7 MiB/secÂÂprealloc in chunks a 100 MiB
3168 MiB copied in 4:17, 12.3 MiB/secÂÂprealloc in chunks a 1 GiB
3168 MiB copied in 4:12, 12.5 MiB/secÂÂcomplete prealloc
Conclusion:
Preallocation seems only good if windows writes to NTFS.
On linux preallocation slows down the copy operation without any advantage.
If preallocating don't split the preallocation chunks.
Does anyone have tools that shows the number of FAT fragments?
In general I'm searching for non interactive tools analyzing the fragmentation for each os+filesystem combi.
this is the output from the perl script. it seems to find the exact same files as the one that allows 200 fragments/GB
Code:
$ sudo '/home/jenna/Desktop/perlFragChecker.pl' '/media/WiiFat500/wbfs'
/media/WiiFat500/wbfs/Rock Band Green Day Rock_[R36E69].wbfsÂÂÂÂ16
/media/WiiFat500/wbfs/RFPP01.wbfsÂÂÂÂ2
/media/WiiFat500/wbfs/Guitar Hero 5_[SXEE52].wbf1ÂÂÂÂ48
/media/WiiFat500/wbfs/Guitar Hero - Warriors of Rock_[SXIP52].wbfsÂÂÂÂ7732
/media/WiiFat500/wbfs/Guitar Hero 5_[SXEE52].wbfsÂÂÂÂ7685
/media/WiiFat500/wbfs/Pikmin 2_[R92P01].wbfsÂÂÂÂ4176
/media/WiiFat500/wbfs/New Super Mario Bros. Wii_[SMNE01].wbfsÂÂÂÂ4175
/media/WiiFat500/wbfs/James Bond - Golden Eye_[SJBE52].wbfsÂÂÂÂ2
/media/WiiFat500/wbfs/Guitar Hero Aerosmith Custom - Mini Concerts Edition_[CGVEMC].wbfsÂÂÂÂ4608
/media/WiiFat500/wbfs/Guitar Hero Aerosmith Custom - Mini Concerts Edition 2_[CGVEM2].wbfsÂÂÂÂ4608
/media/WiiFat500/wbfs/.____deleteme.wbfs.olOsd1.tmpÂÂÂÂ3
10.5769230769231% non contiguous files, 318.730769230769 average fragments.
does this mean that those 200 frags are really negligible?
I haven't read in detail the last couple of pages, but I guess you're having problem with preallocation on linux?
In wbfs_file, preallocation is #ifdefed only for windows, because that's the only os that needs such measures to prevent fragmentation, and on windows preallocating has no performance impact, only positive effects.
Here is my decision, discuss it if you like:[*] The test options --sparse --defrag and --pa-limit=size will be removed.[*] I implement a new option --prealloc with an optional parameter.[*] --prealloc=0 disables preallocation. This is the default for NON-CYGWIN.[*] --prealloc=1 enables an smart preallocation ignoring unused blocks.[*] --prealloc=2 or simple --prealloc preallocates the complete CISO or WBFS file. This is the default for CYGWIN because --prealloc=1 doesn't work like thought if the sparse holes are small.[*] Modes '1' and '2' are only different for WBFS and CISO files. ISO files will always use '1', and WDF and WIA files always '2' if nit disabled.[*] Later I will implement the preallocation for non image files (e.g. extracted files).
--prealloc=2 or simple --prealloc preallocates the complete CISO or WBFS file. This is the default for CYGWIN because --prealloc=1 doesn't work like thought if the sparse holes are small.
Ok, I have committed v1.23,beta4 (nice digit order) inti the beta branch.
New option: --prealloc
Code:
--prealloc [=mode]
ÂÂÂÂÂÂ This option enables or disables the disc space preallocation. If
ÂÂÂÂÂÂ enabled the tools try to allocate disc space for the new files before
ÂÂÂÂÂÂ writing the data. This reduces the fragmentation but also disables the
ÂÂÂÂÂÂ sparse effect for prealocated areas.
ÂÂÂÂÂÂÂÂ The optional parameter decides the preallocation modus: OFF (or 0),
ÂÂÂÂÂÂ SMART (or 1), ALL (or 2). If no parameter is set, ALL is used.
ÂÂÂÂÂÂÂÂ Mode 'OFF' disables the preallocation. This is the default for all
ÂÂÂÂÂÂ non Cygwin releases because preallocation has only advantages on
ÂÂÂÂÂÂ Windows systems.
ÂÂÂÂÂÂÂÂ Mode 'SMART' looks into the source disc to find out the writing
ÂÂÂÂÂÂ areas. SMART is only avalable for ISO, CISO and WBFS file types. For
ÂÂÂÂÂÂ other file types ALL is used instead.
ÂÂÂÂÂÂÂÂ Mode 'ALL' preallocate the whole destination file. This is the
ÂÂÂÂÂÂ default for Cygwin. Because of the large holes in plain ISO images, the
ÂÂÂÂÂÂ SMART mode is used for ISOs instead.
I've used WIA enough to report in. After testing 12 games, I get compression ratios of 50-60% almost every time with 7-zip+LZMA1 (Ultra). Some games do still only compress to 80-90%, but that's still great. I love WIA.
WIT: Wiimms ISO Tools v1.23a.beta5 - 2010-12-03 Beta test version, for EXPERIMENTAL use only.
WIT (Wiimms ISO Tools) is a set of command line tools to manipulate Wii ISO images and WBFS containers. The two main tools are called wit (Wiimms ISO Tool) and wwt (Wiimms WBFS Tool, the WBFS manager).
Preallocation to avoid fragmentation
I have implemented disc space preallocation. Under Windows it can avoid fragmentation. Therfor it is only enabled for Cygwin by default. It can be disabled with --prealloc=0 or enabled with --prealloc. See http://wit.wiimm.de/opt/prealloc for more details. .
ID wildcards for some wwt commands
The wwt commands EXTRACT + REMOVE + TOUCH + VERIFY wil support now ID wildcards to select different discs. See http://wit.wiimm.de/info/disc-id.html for more details.
The needed preallocation function is not avaialable for Mac. So Mac will ignore --prealloc. .
wwt EXTRACT rewritten
"wwt EXTRACT" is now able to patch while extracting and to create WIA files like "wit COPY". It support also the options --fst --pmode --psel --raw --files and --sneek. http://wit.wiimm.de/wwt/cmd-extract.html for more details. .
Change log:
Code:
wit v1.23a.beta5 r2068 - 2010-12-03
- Preallocation of disc space implemented. It is only enabled for Cygwin by
ÂÂ default. It can be disabled with --prealloc=0 or enabled with --prealloc.
ÂÂ The needed preallocation function is not avaialable for mac.
- wwt EXTRACT + REMOVE + TOUCH + VERIFY:
ÂÂ These commands accept now ID6 pattern like '...P' or '+1'. Each '.' is a
ÂÂ wildcard that matches any single character. A '+' matches any number of
ÂÂ characters. For details see http://wit.wiimm.de/info/disc-id.html
- wwt EXTRACT is now able to patch while extracting and to create WIA files.
ÂÂ It support also the options --fst --pmode --psel --raw --files and --sneek.
Known bugs:
- The GameCube support is in development and EXPERIMENTAL!
- wit COPY --raw: Copying an image in raw mode to a WBFS file is buggy.
- There are new problems while reading from non seekable sources (e.g. pipes).
ÂÂ The reason is the complete new and more intelligent wiidisc lib which makes
ÂÂ more lookups ==> The best is to use version 1.01c for pipes.
- No WIA/bzip2 support for MAC/x86_64. (bug in the Mac bzip2 library)
Missing docu:
- wit commands FILES + EXTRACT => see built in help
- Tool wdf => see built in help
The source is published under the license of GPL2. Checkout the sources directly from the SVN repository or browse the source. These URLs are different from the product version. Visit http://wit.wiimm.de/ for more information.
WIT: Wiimms ISO Tools v1.26a.beta1 - 2011-01-11 Beta test version, for EXPERIMENTAL use only.
WIT (Wiimms ISO Tools) is a set of command line tools to manipulate Wii ISO images and WBFS containers. The two main tools are called wit (Wiimms ISO Tool) and wwt (Wiimms WBFS Tool, the WBFS manager).
I have made some changes in the make system to support local libraries with own makefiles (libbz2.a), option -static and cross compiling (Linux/i386 on Linux/x86_64). Hope that all work as usual. Please test it! .
.
Change log:
Code:
wit v1.26a.beta1 r2156 - 2011-01-11
- I have made some changes in the make system to support local libraries with
ÂÂ own makefiles (libbz2.a), option -static and cross compiling (Linux/i386 on
ÂÂ Linux/x86_64). Hope that all work as usual.
ÂÂÂÂ- BZIP2 support is now included as source and should work for all systems.
ÂÂÂÂÂÂThis enables WIA/bzip2 support for Mac/x86_64.
ÂÂÂÂ- The distributions of Linux/i386 and Linux/X86_64 are linked static to
ÂÂÂÂÂÂto avoid GLIBC version issues (e.g. fallocate() support).
ÂÂÂÂ- On Linux/x86_64: Enter "M32=1 make ..." to create i386 binaries.
ÂÂÂÂÂÂI386 libraries and include files are needed.
- "wit COPY/CONVERT/EXTRACT/DIFF" support option --sections.
Known bugs:
- The GameCube support is in development and EXPERIMENTAL!
ÂÂÂÂ- GameCube conversion to WIA fails.
- There are new problems while reading from non seekable sources (e.g. pipes).
ÂÂ The reason is the complete new and more intelligent wiidisc lib which makes
ÂÂ more lookups ==> The best is to use version 1.01c for pipes.
The source is published under the license of GPL2. Checkout the sources directly from the SVN repository or browse the source. These URLs are different from the product version. Visit http://wit.wiimm.de/ for more information.
any chance you could change the makefile to support the jobserver? in prior revisions, "make -j4" works as expected. with whatever changes were made to add the bzib library, it causes make to spit out some crap about the jobserver.
mine says
Mac OS X 10.6.4 (10f569)
kernel version 10.0.0
if you have any ideas about what i may have done to break it, ill try them. but all i did is checkout the source and type make.
Sony made a shocking announcement today, revealing that the company plans to move away from physical game releases in the future. Citing claims of how the industry is...
After much speculation, a lot of which being caused by dbrand's unceremonious reveal of their Companion Cube casing, the Steam Machine is finally available to order...
Remember when you could get an Xbox Series S for $300? Those were the days. Microsoft has today announced the latest in their console price hikes, seeing their...
The delays may be behind us, but the news isn't all good for Grand Theft Auto VI. Rockstar have today announced that pre-orders for the game will go live tomorrow, on...
Last month we got confirmation of a new model of Switch 2 to better comply with upcoming EU regulations. With the legislation set to come into effect in February of...
The end has come for the PlayStation 3 and the PlayStation Vita. After supporting the PSN Store on the PS3 and PS Vita since 2006 and 2011 respectively, Sony has...
Tired of waiting for Game Freak to bring Pokemon Emerald to modern platforms? We've got you covered with a brand new port in the works. Currently available on GitHub...
In this time of economic uncertainty and rampant price hikes, the Steam sales stand as our final bastions of affordability for those opting to avoid the seas. The...
Apple have today announced price increases, primarily focused on their MacBook and iPad lines. These increases have already come into effect, with both prices and the...
The Switch 2 has been out for a year now, but you shouldn't count the original system out yet! Released a few days ago, popular PS2 emulator NetherSX2 has found its...
Sony made a shocking announcement today, revealing that the company plans to move away from physical game releases in the future. Citing claims of how the industry is...
After much speculation, a lot of which being caused by dbrand's unceremonious reveal of their Companion Cube casing, the Steam Machine is finally available to order...
Remember when you could get an Xbox Series S for $300? Those were the days. Microsoft has today announced the latest in their console price hikes, seeing their...
The delays may be behind us, but the news isn't all good for Grand Theft Auto VI. Rockstar have today announced that pre-orders for the game will go live tomorrow, on...
Last month we got confirmation of a new model of Switch 2 to better comply with upcoming EU regulations. With the legislation set to come into effect in February of...
The end has come for the PlayStation 3 and the PlayStation Vita. After supporting the PSN Store on the PS3 and PS Vita since 2006 and 2011 respectively, Sony has...
Tired of waiting for Game Freak to bring Pokemon Emerald to modern platforms? We've got you covered with a brand new port in the works. Currently available on GitHub...
Apple have today announced price increases, primarily focused on their MacBook and iPad lines. These increases have already come into effect, with both prices and the...
In this time of economic uncertainty and rampant price hikes, the Steam sales stand as our final bastions of affordability for those opting to avoid the seas. The...
Amidst news of layoffs and cancellations in the wake of Xbox's larger changes, Bethesda has today come out with a statement discussing their active projects. In this...