[giantpune]

im having some issue with r2032. i did "wit copy" using the default setting trying to see the speed increase when writing to NTFS. as soon as i did that, the lights on my HDD started lighting up like it is being written to.

Now its 10 minutes later, the lights on the HDD have been lighting up for the entire time, so wit is doing something on my HDD. after the 10 minutes, it starts printing stuff out stdout. it is writing at 1.3MBs. before it was writing to this same HDD at ~30MBs. its doing the same thing when writing to FAT32. I havent tested writing to ext yet.

Edited by giantpune, 26 November 2010 - 07:25 PM.

[Wiimm]

@giantpune
2 days ago i have a similar but not reproducible effect while using a relative full drive (12 of 40G free). I worked with enabled trace and I know that the preallocation call of about 1.5 GB takes all the time. Immediately doing the same command again works fine and fast and it never happens again. It seems for me as an NTFS cleanup trigger.

My question to you: What happen if your are doing it again?

Edited by Wiimm, 26 November 2010 - 08:23 PM.

[giantpune]

i tried it again on the same NTFS HDD and i get the same result. It is a 1TB drive full of movies, so it is all files > 700MB. And i have 159GB of free space. So, does this mean that first it must move some of these full 700MB movies to create a block of 4GB before it can write the file? And judging be the time of it, it is having to move 3 or 4 movies to make room.

And does it start the time for calculating the transfer speed after it has started writing the file? or does it start the time before it tries to allocate the space on the drive? because it seems that when i try to copy a larger game, it takes longer to start the actual copy and when it does, the speed is slower ( 0.7 MBs reported for a 4GB game ).

and what happens to the allocated disc space if i forcibly interrupt wit? when is the space freed? and who is responsible for unallocating it?

Edited by giantpune, 26 November 2010 - 08:22 PM.

[Wiimm]

The allocated disc space should be linked to the file and if the file is deleted the space become free.

Can you recompile it with "make debug all" and run again and send me via mail the trace file?

Edited by Wiimm, 26 November 2010 - 08:25 PM.

[giantpune]

heres teh trace file created when using wit copy to copy a 4GB game from ext4 to my NTFS partition that has all those movies on it. It took something like 15-20 minutes
http://www.multiupload.com/G8ENAJ0STF

[Wiimm]

I can't see not so much because there are to less trace commands in the preallocate functions. I will change that tomorrow. What happens if you split for example with 500m or 1g? If you do so i wit will not try to prealloc 4.7g at once.

[giantpune]

if i split the game into smaller parts, or if i just copy a smaller game ( 0.25 GB ) it happens very fast

[Wiimm]

My guess: Windows/NTFS tries to allocate that ordered space in 1 fragment and must sometimes move some other blocks. Under Linux such files are fragmented. Perhaps I must preallocate smaller peaces for windows. I will implement an option to control the max prealloc size for experiments to find out an optimal value.

[giantpune]

im doing all this testing using ubuntu 10.04, ntfs3g, and USB HDD. windows is not involved at all.

[Wiimm]

that means it is a ntfs problem!?

[Wiimm]

that means it is a ntfs problem!?

For more tests I have switched to linux. And there we have similar problems.

Some examples: copy ssbb data partition, image made with "make debug test all" (new private version of wit, i will commit it later)

CODE

# time ./wit cp -ovv --psel data ssbb.wdf pool/a.wbfs --sparse
...
     6902 MiB copied in 6:20, 18.1 MiB/sec            
...
real    6m20.381s
user    0m8.693s
sys    0m29.563s
pool/a.wbfs -> /dev/sdb3 <55257>, 6908/6910 MiB, frag=7824

CODE

# time ./wit cp -ovv --psel data ssbb.wdf pool/a.wbfs --pa-limit=100M
...
   0.930  CALL posix_fallocate(5,   400000,  6300000 [  99 MiB])
...
  88.142  CALL posix_fallocate(5,        0,     2800 [  10 KiB])
  88.143  ...
...
     6902 MiB copied in 7:38, 15.0 MiB/sec                

real    7m38.642s
user    0m9.964s
sys    0m38.083s
pool/a.wbfs -> /dev/sdb3 <55257>, 6908/6910 MiB, frag=7769

CODE

# time ./wit cp -ovv --psel data ssbb.wdf pool/a.wbfs --pa-limit=1G
...
   0.913  CALL posix_fallocate(5,   400000, 3a500000 [ 933 MiB])
   8.513  CALL posix_fallocate(5, 3a900000, 3a500000 [ 933 MiB])
  26.785  CALL posix_fallocate(5, 74e00000, 3a500000 [ 933 MiB])
  38.891  CALL posix_fallocate(5, af300000, 3a4c8000 [ 933 MiB])
  50.390  CALL posix_fallocate(5, eabf0000, 31500000 [ 789 MiB])
  60.408  CALL posix_fallocate(5,11c0f0000, 31500000 [ 789 MiB])
  68.615  CALL posix_fallocate(5,14d5f0000, 31500000 [ 789 MiB])
  78.573  CALL posix_fallocate(5,17eaf0000, 31260000 [ 786 MiB])
  88.381  CALL posix_fallocate(5, e9958000,  1088000 [  17 MiB])
  88.381  CALL posix_fallocate(5,   240000,    10000 [  64 KiB])
  88.381  CALL posix_fallocate(5,   200000,     8000 [  32 KiB])
  88.381  CALL posix_fallocate(5,        0,     2800 [  10 KiB])
  88.381  ...
....
     6902 MiB copied in 7:39, 15.0 MiB/sec                

real    7m39.483s
user    0m10.130s
sys    0m38.411s
pool/a.wbfs -> /dev/sdb3 <55257>, 6908/6910 MiB, frag=7769

CODE

# time ./wit cp -ovv --psel data ssbb.wdf pool/a.wbfs --pa-limit=10G
...
   0.717  CALL posix_fallocate(5,   400000, e93c8000 [3732 MiB])
  50.431  CALL posix_fallocate(5, eabf0000, c5160000 [3153 MiB])
  88.850  CALL posix_fallocate(5, e9958000,  1088000 [  17 MiB])
  88.850  CALL posix_fallocate(5,   240000,    10000 [  64 KiB])
  88.850  CALL posix_fallocate(5,   200000,     8000 [  32 KiB])
  88.850  CALL posix_fallocate(5,        0,     2800 [  10 KiB])
  88.850  ...
...
     6902 MiB copied in 7:47, 14.8 MiB/sec                

real    7m47.684s
user    0m10.179s
sys    0m38.757s
pool/a.wbfs -> /dev/sdb3 <55257>, 6908/6910 MiB, frag=7816

SUMMARY for Linux + ext3:

The preallocation consumes 88 seconds. The copy only time is in all 4 cases about 6:20. The number of fragments is always about 7800. This means that preallocation has no advantage on linux+ext3. I think, preallocation allocate disc space and fill it with zeros. Btw: fragmentation is not a ext2/ext3 problem because it is designed to handle with fragments.


Last not least the command I used:

Warning! Spoiler inside. 

CODE

make test debug all

for i in --sparse --pa-limit=100M --pa-limit=1G --pa-limit=10G; \
do \
    echo "##########  $i  #########"; \
    rm -f pool/a.wbfs* pool/.a.wbfs*; \
    sync; \
    sleep 1; \
    time ./wit cp -ovv --psel data srcdir/ssbb.wdf pool/a.wbfs $i; \
    ./print-disc-blocks.sh pool/a.wbfs \
done 2>&1 | tee a.log

# 'pool/' is my temp and working dir (it is ignored by svn)

[oggzee]

So, creating files under windows now works properly and doesn't create fragmented files anymore?
Very happy to hear that!
Good job wiimm!

and sorry if i was too annoying by constantly pointing this out, but it had to be fixed!

[Wiimm]

Here are some more statstics:

CODE

# linux, 64 bit, copy ssbb.wdf to ext3
6902 MiB copied in 6:20, 18.1 MiB/sec  no prealloc
6902 MiB copied in 7:38, 15.0 MiB/sec  prealloc in chunks a 100 MiB
6902 MiB copied in 7:39, 15.0 MiB/sec  prealloc in chunks a 1 GiB
6902 MiB copied in 7:47, 14.8 MiB/sec  complete prealloc

CODE

# linux, 64 bit, copy tra.wdf to fat32
3168 MiB copied in 1:28, 35.8 MiB/sec  no prealloc
3168 MiB copied in 3:02, 17.4 MiB/sec  prealloc in chunks a 100 MiB
3168 MiB copied in 3:02, 17.3 MiB/sec  prealloc in chunks a 1 GiB
3168 MiB copied in 3:02, 17.3 MiB/sec  complete prealloc

CODE

# linux, 64 bit, copy tra.wdf to ntfs
3168 MiB copied in 0:40, 78.0 MiB/sec  no prealloc
3168 MiB copied in 1:15, 41.9 MiB/sec  prealloc in chunks a 100 MiB
3168 MiB copied in 1:15, 41.8 MiB/sec  prealloc in chunks a 1 GiB
3168 MiB copied in 1:16, 41.5 MiB/sec  complete prealloc

CODE

win XP, 32 bit, copy tra.wdf to fat32
3168 MiB copied in 2:22, 22.2 MiB/sec  no prealloc
3168 MiB copied in 2:41, 19.6 MiB/sec  prealloc in chunks a 100 MiB
3168 MiB copied in 2:17, 23.0 MiB/sec  prealloc in chunks a 1 GiB
3168 MiB copied in 2:10, 24.3 MiB/sec  complete prealloc

CODE

# win XP, 32 bit, copy tra.wdf to ntfs
3168 MiB copied in 5:54,  8.9 MiB/sec  no prealloc
3168 MiB copied in 3:36, 14.7 MiB/sec  prealloc in chunks a 100 MiB
3168 MiB copied in 4:17, 12.3 MiB/sec  prealloc in chunks a 1 GiB
3168 MiB copied in 4:12, 12.5 MiB/sec  complete prealloc

Conclusion:

• Preallocation seems only good if windows writes to NTFS.
• On linux preallocation slows down the copy operation without any advantage.
• If preallocating don't split the preallocation chunks.

Does anyone have tools that shows the number of FAT fragments?
In general I'm searching for non interactive tools analyzing the fragmentation for each os+filesystem combi.

Edited by Wiimm, 27 November 2010 - 10:11 PM.

[giantpune]

maybe pyfragtools is what you are looking for?

http://ubuntuforums.org/showthread.php?t=169551

check a single file

CODE

$ sudo '/home/jenna/pyfragtools-0.1/defrag' --analyze '/media/WiiFat500/wbfs/Guitar Hero 5_[SXEE52].wbf1'
Building list of files to analyze... done!
                                                                              
Analyze finished. 100.0 % fragmentation (1 files), 8.0 average frags/MB
Fragmented files:
7.96    /media/WiiFat500/wbfs/Guitar Hero 5_[SXEE52].wbf1
========= COMPLETE ===========
Remaining Fragmented Files:
7.96    /media/WiiFat500/wbfs/Guitar Hero 5_[SXEE52].wbf1
Frags/MB Before:     7.96
Frags/MB After:      7.96
Improvement:         0.0 %
===============================

check a directory

CODE

jenna@pc:~/pyfragtools-0.1$ sudo '/home/jenna/pyfragtools-0.1/defrag' --analyze '/media/WiiFat500/wbfs/'
Building list of files to analyze... done!
                                                                              
Analyze finished. 10.6 % fragmentation (11 files), 2.2 average frags/MB
Fragmented files:
7.96    /media/WiiFat500/wbfs/Guitar Hero 5_[SXEE52].wbf1
7.73    /media/WiiFat500/wbfs/New Super Mario Bros. Wii_[SMNE01].wbfs
2.12    /media/WiiFat500/wbfs/Pikmin 2_[R92P01].wbfs
1.89    /media/WiiFat500/wbfs/Guitar Hero - Warriors of Rock_[SXIP52].wbfs
1.88    /media/WiiFat500/wbfs/Guitar Hero 5_[SXEE52].wbfs
1.49    /media/WiiFat500/wbfs/Guitar Hero Aerosmith Custom - Mini Concerts Edition 2_[CGVEM2].wbfs
1.28    /media/WiiFat500/wbfs/Guitar Hero Aerosmith Custom - Mini Concerts Edition_[CGVEMC].wbfs
0.00    /media/WiiFat500/wbfs/Rock Band Green Day Rock_[R36E69].wbfs
0.00    /media/WiiFat500/wbfs/RFPP01.wbfs
0.00    /media/WiiFat500/wbfs/.____deleteme.wbfs.olOsd1.tmp
...
========= COMPLETE ===========
Remaining Fragmented Files:
0.00    /media/WiiFat500/wbfs/James Bond - Golden Eye_[SJBE52].wbfs
0.00    /media/WiiFat500/wbfs/.____deleteme.wbfs.olOsd1.tmp
0.00    /media/WiiFat500/wbfs/RFPP01.wbfs
0.00    /media/WiiFat500/wbfs/Rock Band Green Day Rock_[R36E69].wbfs
1.28    /media/WiiFat500/wbfs/Guitar Hero Aerosmith Custom - Mini Concerts Edition_[CGVEMC].wbfs
1.49    /media/WiiFat500/wbfs/Guitar Hero Aerosmith Custom - Mini Concerts Edition 2_[CGVEM2].wbfs
1.88    /media/WiiFat500/wbfs/Guitar Hero 5_[SXEE52].wbfs
1.89    /media/WiiFat500/wbfs/Guitar Hero - Warriors of Rock_[SXIP52].wbfs
2.12    /media/WiiFat500/wbfs/Pikmin 2_[R92P01].wbfs
7.73    /media/WiiFat500/wbfs/New Super Mario Bros. Wii_[SMNE01].wbfs
7.96    /media/WiiFat500/wbfs/Guitar Hero 5_[SXEE52].wbf1
Frags/MB Before:     24.35
Frags/MB After:      24.35
Improvement:         0.0 %
===============================

Edited by giantpune, 27 November 2010 - 09:38 PM.

[Wiimm]

I have updated the beta branch. New features:

• If preallocating the largest block is preallocated first.
• The new option --pa-limit=size sets a limit for the allocation blocks. See "wit help copy" for details.

@giantpune:
thanx, but only if the tool can show the number of fragments without defragmenting

Edited by Wiimm, 27 November 2010 - 09:37 PM.