The preallocation consumes 88 seconds. The copy only time is in all 4 cases about 6:20. The number of fragments is always about 7800. This means that preallocation has no advantage on linux+ext3. I think, preallocation allocate disc space and fill it with zeros. Btw: fragmentation is not a ext2/ext3 problem because it is designed to handle with fragments.
Last not least the command I used:
CODEmake test debug all
for i in --sparse --pa-limit=100M --pa-limit=1G --pa-limit=10G; \
do \
ÂÂÂÂecho "##########ÂÂ$iÂÂ#########"; \
ÂÂÂÂrm -f pool/a.wbfs* pool/.a.wbfs*; \
ÂÂÂÂsync; \
ÂÂÂÂsleep 1; \
ÂÂÂÂtime ./wit cp -ovv --psel data srcdir/ssbb.wdf pool/a.wbfs $i; \
ÂÂÂÂ./print-disc-blocks.sh pool/a.wbfs \
done 2>&1 | tee a.log
# 'pool/' is my temp and working dir (it is ignored by svn)
# linux, 64 bit, copy ssbb.wdf to ext3
6902 MiB copied in 6:20, 18.1 MiB/secÂÂno prealloc
6902 MiB copied in 7:38, 15.0 MiB/secÂÂprealloc in chunks a 100 MiB
6902 MiB copied in 7:39, 15.0 MiB/secÂÂprealloc in chunks a 1 GiB
6902 MiB copied in 7:47, 14.8 MiB/secÂÂcomplete prealloc
Code:
# linux, 64 bit, copy tra.wdf to fat32
3168 MiB copied in 1:28, 35.8 MiB/secÂÂno prealloc
3168 MiB copied in 3:02, 17.4 MiB/secÂÂprealloc in chunks a 100 MiB
3168 MiB copied in 3:02, 17.3 MiB/secÂÂprealloc in chunks a 1 GiB
3168 MiB copied in 3:02, 17.3 MiB/secÂÂcomplete prealloc
Code:
# linux, 64 bit, copy tra.wdf to ntfs
3168 MiB copied in 0:40, 78.0 MiB/secÂÂno prealloc
3168 MiB copied in 1:15, 41.9 MiB/secÂÂprealloc in chunks a 100 MiB
3168 MiB copied in 1:15, 41.8 MiB/secÂÂprealloc in chunks a 1 GiB
3168 MiB copied in 1:16, 41.5 MiB/secÂÂcomplete prealloc
Code:
win XP, 32 bit, copy tra.wdf to fat32
3168 MiB copied in 2:22, 22.2 MiB/secÂÂno prealloc
3168 MiB copied in 2:41, 19.6 MiB/secÂÂprealloc in chunks a 100 MiB
3168 MiB copied in 2:17, 23.0 MiB/secÂÂprealloc in chunks a 1 GiB
3168 MiB copied in 2:10, 24.3 MiB/secÂÂcomplete prealloc
Code:
# win XP, 32 bit, copy tra.wdf to ntfs
3168 MiB copied in 5:54,ÂÂ8.9 MiB/secÂÂno prealloc
3168 MiB copied in 3:36, 14.7 MiB/secÂÂprealloc in chunks a 100 MiB
3168 MiB copied in 4:17, 12.3 MiB/secÂÂprealloc in chunks a 1 GiB
3168 MiB copied in 4:12, 12.5 MiB/secÂÂcomplete prealloc
Conclusion:
Preallocation seems only good if windows writes to NTFS.
On linux preallocation slows down the copy operation without any advantage.
If preallocating don't split the preallocation chunks.
Does anyone have tools that shows the number of FAT fragments?
In general I'm searching for non interactive tools analyzing the fragmentation for each os+filesystem combi.
this is the output from the perl script. it seems to find the exact same files as the one that allows 200 fragments/GB
Code:
$ sudo '/home/jenna/Desktop/perlFragChecker.pl' '/media/WiiFat500/wbfs'
/media/WiiFat500/wbfs/Rock Band Green Day Rock_[R36E69].wbfsÂÂÂÂ16
/media/WiiFat500/wbfs/RFPP01.wbfsÂÂÂÂ2
/media/WiiFat500/wbfs/Guitar Hero 5_[SXEE52].wbf1ÂÂÂÂ48
/media/WiiFat500/wbfs/Guitar Hero - Warriors of Rock_[SXIP52].wbfsÂÂÂÂ7732
/media/WiiFat500/wbfs/Guitar Hero 5_[SXEE52].wbfsÂÂÂÂ7685
/media/WiiFat500/wbfs/Pikmin 2_[R92P01].wbfsÂÂÂÂ4176
/media/WiiFat500/wbfs/New Super Mario Bros. Wii_[SMNE01].wbfsÂÂÂÂ4175
/media/WiiFat500/wbfs/James Bond - Golden Eye_[SJBE52].wbfsÂÂÂÂ2
/media/WiiFat500/wbfs/Guitar Hero Aerosmith Custom - Mini Concerts Edition_[CGVEMC].wbfsÂÂÂÂ4608
/media/WiiFat500/wbfs/Guitar Hero Aerosmith Custom - Mini Concerts Edition 2_[CGVEM2].wbfsÂÂÂÂ4608
/media/WiiFat500/wbfs/.____deleteme.wbfs.olOsd1.tmpÂÂÂÂ3
10.5769230769231% non contiguous files, 318.730769230769 average fragments.
does this mean that those 200 frags are really negligible?
I haven't read in detail the last couple of pages, but I guess you're having problem with preallocation on linux?
In wbfs_file, preallocation is #ifdefed only for windows, because that's the only os that needs such measures to prevent fragmentation, and on windows preallocating has no performance impact, only positive effects.
Here is my decision, discuss it if you like:[*] The test options --sparse --defrag and --pa-limit=size will be removed.[*] I implement a new option --prealloc with an optional parameter.[*] --prealloc=0 disables preallocation. This is the default for NON-CYGWIN.[*] --prealloc=1 enables an smart preallocation ignoring unused blocks.[*] --prealloc=2 or simple --prealloc preallocates the complete CISO or WBFS file. This is the default for CYGWIN because --prealloc=1 doesn't work like thought if the sparse holes are small.[*] Modes '1' and '2' are only different for WBFS and CISO files. ISO files will always use '1', and WDF and WIA files always '2' if nit disabled.[*] Later I will implement the preallocation for non image files (e.g. extracted files).
--prealloc=2 or simple --prealloc preallocates the complete CISO or WBFS file. This is the default for CYGWIN because --prealloc=1 doesn't work like thought if the sparse holes are small.
Ok, I have committed v1.23,beta4 (nice digit order) inti the beta branch.
New option: --prealloc
Code:
--prealloc [=mode]
ÂÂÂÂÂÂ This option enables or disables the disc space preallocation. If
ÂÂÂÂÂÂ enabled the tools try to allocate disc space for the new files before
ÂÂÂÂÂÂ writing the data. This reduces the fragmentation but also disables the
ÂÂÂÂÂÂ sparse effect for prealocated areas.
ÂÂÂÂÂÂÂÂ The optional parameter decides the preallocation modus: OFF (or 0),
ÂÂÂÂÂÂ SMART (or 1), ALL (or 2). If no parameter is set, ALL is used.
ÂÂÂÂÂÂÂÂ Mode 'OFF' disables the preallocation. This is the default for all
ÂÂÂÂÂÂ non Cygwin releases because preallocation has only advantages on
ÂÂÂÂÂÂ Windows systems.
ÂÂÂÂÂÂÂÂ Mode 'SMART' looks into the source disc to find out the writing
ÂÂÂÂÂÂ areas. SMART is only avalable for ISO, CISO and WBFS file types. For
ÂÂÂÂÂÂ other file types ALL is used instead.
ÂÂÂÂÂÂÂÂ Mode 'ALL' preallocate the whole destination file. This is the
ÂÂÂÂÂÂ default for Cygwin. Because of the large holes in plain ISO images, the
ÂÂÂÂÂÂ SMART mode is used for ISOs instead.
I've used WIA enough to report in. After testing 12 games, I get compression ratios of 50-60% almost every time with 7-zip+LZMA1 (Ultra). Some games do still only compress to 80-90%, but that's still great. I love WIA.
WIT: Wiimms ISO Tools v1.23a.beta5 - 2010-12-03 Beta test version, for EXPERIMENTAL use only.
WIT (Wiimms ISO Tools) is a set of command line tools to manipulate Wii ISO images and WBFS containers. The two main tools are called wit (Wiimms ISO Tool) and wwt (Wiimms WBFS Tool, the WBFS manager).
Preallocation to avoid fragmentation
I have implemented disc space preallocation. Under Windows it can avoid fragmentation. Therfor it is only enabled for Cygwin by default. It can be disabled with --prealloc=0 or enabled with --prealloc. See http://wit.wiimm.de/opt/prealloc for more details. .
ID wildcards for some wwt commands
The wwt commands EXTRACT + REMOVE + TOUCH + VERIFY wil support now ID wildcards to select different discs. See http://wit.wiimm.de/info/disc-id.html for more details.
The needed preallocation function is not avaialable for Mac. So Mac will ignore --prealloc. .
wwt EXTRACT rewritten
"wwt EXTRACT" is now able to patch while extracting and to create WIA files like "wit COPY". It support also the options --fst --pmode --psel --raw --files and --sneek. http://wit.wiimm.de/wwt/cmd-extract.html for more details. .
Change log:
Code:
wit v1.23a.beta5 r2068 - 2010-12-03
- Preallocation of disc space implemented. It is only enabled for Cygwin by
ÂÂ default. It can be disabled with --prealloc=0 or enabled with --prealloc.
ÂÂ The needed preallocation function is not avaialable for mac.
- wwt EXTRACT + REMOVE + TOUCH + VERIFY:
ÂÂ These commands accept now ID6 pattern like '...P' or '+1'. Each '.' is a
ÂÂ wildcard that matches any single character. A '+' matches any number of
ÂÂ characters. For details see http://wit.wiimm.de/info/disc-id.html
- wwt EXTRACT is now able to patch while extracting and to create WIA files.
ÂÂ It support also the options --fst --pmode --psel --raw --files and --sneek.
Known bugs:
- The GameCube support is in development and EXPERIMENTAL!
- wit COPY --raw: Copying an image in raw mode to a WBFS file is buggy.
- There are new problems while reading from non seekable sources (e.g. pipes).
ÂÂ The reason is the complete new and more intelligent wiidisc lib which makes
ÂÂ more lookups ==> The best is to use version 1.01c for pipes.
- No WIA/bzip2 support for MAC/x86_64. (bug in the Mac bzip2 library)
Missing docu:
- wit commands FILES + EXTRACT => see built in help
- Tool wdf => see built in help
The source is published under the license of GPL2. Checkout the sources directly from the SVN repository or browse the source. These URLs are different from the product version. Visit http://wit.wiimm.de/ for more information.
WIT: Wiimms ISO Tools v1.26a.beta1 - 2011-01-11 Beta test version, for EXPERIMENTAL use only.
WIT (Wiimms ISO Tools) is a set of command line tools to manipulate Wii ISO images and WBFS containers. The two main tools are called wit (Wiimms ISO Tool) and wwt (Wiimms WBFS Tool, the WBFS manager).
I have made some changes in the make system to support local libraries with own makefiles (libbz2.a), option -static and cross compiling (Linux/i386 on Linux/x86_64). Hope that all work as usual. Please test it! .
.
Change log:
Code:
wit v1.26a.beta1 r2156 - 2011-01-11
- I have made some changes in the make system to support local libraries with
ÂÂ own makefiles (libbz2.a), option -static and cross compiling (Linux/i386 on
ÂÂ Linux/x86_64). Hope that all work as usual.
ÂÂÂÂ- BZIP2 support is now included as source and should work for all systems.
ÂÂÂÂÂÂThis enables WIA/bzip2 support for Mac/x86_64.
ÂÂÂÂ- The distributions of Linux/i386 and Linux/X86_64 are linked static to
ÂÂÂÂÂÂto avoid GLIBC version issues (e.g. fallocate() support).
ÂÂÂÂ- On Linux/x86_64: Enter "M32=1 make ..." to create i386 binaries.
ÂÂÂÂÂÂI386 libraries and include files are needed.
- "wit COPY/CONVERT/EXTRACT/DIFF" support option --sections.
Known bugs:
- The GameCube support is in development and EXPERIMENTAL!
ÂÂÂÂ- GameCube conversion to WIA fails.
- There are new problems while reading from non seekable sources (e.g. pipes).
ÂÂ The reason is the complete new and more intelligent wiidisc lib which makes
ÂÂ more lookups ==> The best is to use version 1.01c for pipes.
The source is published under the license of GPL2. Checkout the sources directly from the SVN repository or browse the source. These URLs are different from the product version. Visit http://wit.wiimm.de/ for more information.
any chance you could change the makefile to support the jobserver? in prior revisions, "make -j4" works as expected. with whatever changes were made to add the bzib library, it causes make to spit out some crap about the jobserver.
mine says
Mac OS X 10.6.4 (10f569)
kernel version 10.0.0
if you have any ideas about what i may have done to break it, ill try them. but all i did is checkout the source and type make.
Today, April 8th, 2024, at 4PM PT, marks the day in which Nintendo permanently ends support for both the 3DS and the Wii U online services, which include co-op play...
A new Nintendo Switch firmware update is here. System software version 18.0.1 has been released. This update offers the typical stability features as all other...
With Apple having recently updated their guidelines for the App Store, iOS users have been left to speculate on specific wording and whether retro emulators as we...
The time has finally come, and after many, many years (if not decades) of Apple users having to side load emulator apps into their iOS devices through unofficial...
The highly popular and accurate FPGA hardware, MisterFGPA, has received today a brand new update with a long-awaited feature, or rather, a new core for hardcore...
Nintendo might just as well be a law firm more than a videogame company at this point in time, since they have yet again issued their now almost trademarked usual...
Another video game prototype has been found and preserved, and this time, it's none other than the game that spawned an entire franchise beloved by many, the very...
Nintendo has recently announced through their social media accounts that a new Indie World stream will be airing tomorrow, scheduled for April 17th, 2024 at 7 a.m. PT...
Anbernic is back with yet another retro handheld device. The upcoming RG28XX is another console sporting the quad-core H700 chip of the company's recent RG35XX 2024...
Homebrew games for retro consoles have seen a rise over the years thanks to new tools and community-made development kits, which help a lot in the process of creating...
Today, April 8th, 2024, at 4PM PT, marks the day in which Nintendo permanently ends support for both the 3DS and the Wii U online services, which include co-op play...
Nintendo might just as well be a law firm more than a videogame company at this point in time, since they have yet again issued their now almost trademarked usual...
With Apple having recently updated their guidelines for the App Store, iOS users have been left to speculate on specific wording and whether retro emulators as we...
The time has finally come, and after many, many years (if not decades) of Apple users having to side load emulator apps into their iOS devices through unofficial...
A new Nintendo Switch firmware update is here. System software version 18.0.1 has been released. This update offers the typical stability features as all other...
TheFlow has done it again--a new kernel exploit has been released for PlayStation 4 consoles. This latest exploit is called PPPwn, and works on PlayStation 4 systems...
Two classic titles join the Nintendo Switch Online Expansion Pack game lineup. Available starting April 24th will be the motorcycle racing game Extreme G and another...
The highly popular and accurate FPGA hardware, MisterFGPA, has received today a brand new update with a long-awaited feature, or rather, a new core for hardcore...
Nintendo has recently announced through their social media accounts that a new Indie World stream will be airing tomorrow, scheduled for April 17th, 2024 at 7 a.m. PT...
In 2017, the United States Federal Communications Commission (FCC) repealed net neutrality. At the time, it was a major controversy between internet service providers...