Making Sparse Files (for .ISO on NTFS)

Discussion in 'Wii - Backup Loaders' started by ClockWorK, Dec 23, 2009.

Dec 23, 2009
  1. ClockWorK
    OP

    Member ClockWorK GBAtemp Regular

    Joined:
    May 10, 2009
    Messages:
    150
    Country:
    United States
    I'm just going to post this and see if there's any interest.


    So, I recently made the jump from WBFS to FAT32 with .WBFS files, and then to NTFS with .ISO files.

    I learned a lot about sparse files along the way. I was not familiar with them at all. The basic concept is that the file system can encode a long string of zero-bytes in a condensed format to save space on disk. The down-side to sparse files is that they can become fragmented when they are modified, and have more overhead for operations that write to the file.

    Sparse files are useful for scrubbed Wii ISOs on NTFS because it allows the user to have a 4.3gig ISO on the drive but have it only take up it's scrubbed size on disk.

    Unfortunately, I've found that copying a sparse file from one NTFS partition to another NTFS partition creates a non-sparse file in the destination.

    So, I've written a program that will take a bunch of ISOs (or any files really), and copy them to a specified destination folder. But, instead of just doing a normal copy, it will re-write the destination file as a sparse file. This can be confirmed by right-clicking on the file and choosing 'Properties'. The .ISO file will show that it is 4,699,979,776 bytes. The size on disk will be less than this, possibly by a significant amount.

    Interestingly, VB.Net 2008 does not seem to expose the required methods to perform this task using managed code. I've included the source code with the app. Basically, if a string of zero-bytes is encountered that is more than 128,000 bytes (the threshold I came decided to use), the program encodes the string of zero-bytes as sparse zeros. Optionally, the program can also compare the content of the input and output files to ensure they match.

    To confirm that this works correctly, I've processed a few .ISO files and then run the input and the output through a file-compare. Everything has come out OK (100% match). However, if you are interested in trying this, I recommend keeping your original file and performing a compare with a third-party tool before getting rid of any of your original ISO files.

    Maybe there's already a tool for this that I'm not aware of. Either way, maybe the source will be useful to someone. (if you've never used CopyMemory in VB, this code will be a new experience for you). I'd like to know if there's any interest in this program.

    I haven't tested on Vista or Win7 yet.


    Application SourceCode and Executable:
    http://www.megaupload.com/?d=H0UMU6Y9
     


  2. Eddie_Brock

    Member Eddie_Brock GBAtemp Fan

    Joined:
    Apr 15, 2009
    Messages:
    341
    Location:
    Vancouver Island
    Country:
    Canada
    Cool concept! Thanks for the program [​IMG]

    I am currently waiting for more Loaders to support NTFS, and then I will gladly jump-ship back to NTFS, as its my preffered file system. This is a useful app for saving space, great work.

    [​IMG]
     
  3. oggzee

    Member oggzee GBAtemp Addict

    Joined:
    Apr 11, 2009
    Messages:
    2,335
    Country:
    Slovenia
    @ClockWorK:

    I would recommend that you verify that the created files are not fragmented.
    To prevent fragmentation you have to first pre-allocate file size and only then enable sparse mode.
    Look at wbfs_file source (libwbfs_win32.c) to see what i mean.
    To check the file fragmentation this is a good tool:
    http://www.mutexed.com/code/FragExt/#defrag-manager
    See this post for more info:
    http://gbatemp.net/index.php?s=&showto...t&p=2470297
    Also, in my opinion a good chunk size to check for zeroes is 32k (32768 bytes) since that is the wii sector size.
    And there's another issue which has to be considered, wiiscrubber will fill the unused data with 0xFF not zeroes, so before you can use your tool to copy and create sparse files the iso will have to be scrubbed using zeroes. (which wbfs_file and wwt does)
     
  4. ClockWorK
    OP

    Member ClockWorK GBAtemp Regular

    Joined:
    May 10, 2009
    Messages:
    150
    Country:
    United States
    @Oggzee:

    Thanks for the tip on setting the filesize first. That reduced my fragment count from 30+ to 7. Does that seem right, or are we looking for 1 single fragment?

    I set the chunk size to 128k to help prevent fragmentation. Since most of the sparse data in a Wii ISO is in one or two blocks, I don't think the larger chunksize will be any problem or cost much extra in terms of space. However, as mentioned, I'm still new to sparse files. If you say make it 32k, then 32k is what it will be.

    I'm not really familiar with Wii Scrubber, or most of the Wii ISO layout. I am quite competent writing programs that process and modify binary data, so it's feasible that I could add some other functions (for fixing the 0xFFs, as an example). But for now, this app is just a generic sparse file copier, nothing Wii-specific.

    If there's a need for any other tools, I am always willing to lend my coding skills.
     
  5. oggzee

    Member oggzee GBAtemp Addict

    Joined:
    Apr 11, 2009
    Messages:
    2,335
    Country:
    Slovenia
    Yes, that is ok, each sparse block will add up to the fragment count too.
    Usually there will be around 6 fragments because there will be 3 data blocks: header, update partition, game partition, and the sparse blocks around these.
     
  6. ClockWorK
    OP

    Member ClockWorK GBAtemp Regular

    Joined:
    May 10, 2009
    Messages:
    150
    Country:
    United States
    From looking at it, I had assumed that each change from sparse to non-sparse added a fragment.

    I'm feeling pretty confident that this works, and now the fragmentation is addressed.


    Now the real question is, is anybody actually going to want this?
     
  7. XFlak

    Member XFlak Wiitired but still kicking

    Joined:
    Sep 12, 2009
    Messages:
    9,122
    Location:
    Ontario
    Country:
    Canada
    I would like to have this, so once u post a link I will download for sure... but to be honest i doubt I will use it, I like using .wbfs files instead because then I can transfer them from 1 drive to another using windows explorer (without un-compressing/un-scrubbing it). Also, wii backup manager (by fig2k4) supports converting to sparse ISOs (this may not be explicitly stated, but I believe it is mentioned somewhere and it is either the only setting or the default setting for extracting an ISO).

    I maintain my own personal database of all wii-related computer programs, and I would love to add yours to the list.

    I don't care if some people might call this app 'redundant', the more options available the better, and the fact that you r releasing your source is also great! If u found a more efficient way to sparse ISO's (compared to other apps), it would be great for your code to be added to other programs (like wii backup manager)!

    All in all, great work, and thank you!
     
  8. ClockWorK
    OP

    Member ClockWorK GBAtemp Regular

    Joined:
    May 10, 2009
    Messages:
    150
    Country:
    United States
    Original Post updated with link to SourceCode and Executable.

    Thanks for the feedback, xflak40
     
  9. WiiFan2012

    Member WiiFan2012 GBAtemp Regular

    Joined:
    Dec 1, 2009
    Messages:
    177
    Country:
    United States
    I can't wait until USB Loaders support NTFS also! That, is also, my favorite preferred format. Mostly because it is the most common file format included in todays Windows operating systems.

    But, for now, I have to stick to the bottom of the Atlantic (WBFS), as it is the primary for Wii Backups today.
     
  10. zx3junglist

    Newcomer zx3junglist Member

    Joined:
    Dec 24, 2009
    Messages:
    24
    Country:
    United States
    I thought that USB loader GX is supporting ntfs since r867. http://sites.google.com/site/usbloadergui/news/ntfssupport

    I have formatted a fresh NTFS drive and dropped an untouched .ISO onto it, but even with r874, I get no joy. I'm probably doing something wrong, I'm thinking maybe I still need to convert to .wbfs.
     
  11. ClockWorK
    OP

    Member ClockWorK GBAtemp Regular

    Joined:
    May 10, 2009
    Messages:
    150
    Country:
    United States

    for NTFS, they must be named /wbfs/GAMEID.ISO or /wbfs/GAMEID_GameTitle/GAMEID.ISO
    (coincidentally, I wrote an app for this as well.)


    Similar to FAT32 with the .wbfs files.
     
  12. mousex

    Member mousex GBAtemp Advanced Fan

    Joined:
    Jan 23, 2009
    Messages:
    987
    Country:
    United States
    If you put the normal and sparse blocks directly behind each other shouldn't they be just one big fragment? I mean, this is what happens if you defrag the drive.
     
  13. ChaosEnergy

    Member ChaosEnergy GBAtemp Regular

    Joined:
    Jul 11, 2009
    Messages:
    186
    Country:
    Germany
    maybe you create a switch to choice the blocksize by each one
    this tool will do, what i asked already for
    another switch to scrub or not
    a universal sparse program...there are not only wii images on a disk [​IMG]
     
  14. zurcuer

    Newcomer zurcuer Newbie

    Joined:
    Aug 28, 2009
    Messages:
    5
    Country:
    Philippines
    I'm getting an unhandled exception

    CODESee the end of this message for details on invoking
    just-in-time (JIT) debugging instead of this dialog box.

    ************** Exception Text **************
    System.Exception: Application-defined or object-defined error.
    ÂÂ at Microsoft.VisualBasic.ErrObject.Raise(Int32 Number, Object Source, Object Description, Object HelpFile, Object HelpContext)
    ÂÂ at SparseCopier.modFileAPI.FileNum_SetFileSize(Int32 theFileNumber, Int64 fileLength) in C:\My Work\Programming\VB.Net\Test\SparseCopier\modFileAPI.vb:line 772
    ÂÂ at SparseCopier.modFileAPI.File_CopyToSparseNTFSfile(String inpFileName, String newFileName) in C:\My Work\Programming\VB.Net\Test\SparseCopier\modFileAPI.vb:line 1507
    ÂÂ at SparseCopier.Form1.Button2_Click(Object sender, EventArgs e) in C:\My Work\Programming\VB.Net\Test\SparseCopier\Form1.vb:line 89
    ÂÂ at System.Windows.Forms.Control.OnClick(EventArgs e)
    ÂÂ at System.Windows.Forms.Button.OnClick(EventArgs e)
    ÂÂ at System.Windows.Forms.Button.OnMouseUp(MouseEventArgs mevent)
    ÂÂ at System.Windows.Forms.Control.WmMouseUp(Message& m, MouseButtons button, Int32 clicks)
    ÂÂ at System.Windows.Forms.Control.WndProc(Message& m)
    ÂÂ at System.Windows.Forms.ButtonBase.WndProc(Message& m)
    ÂÂ at System.Windows.Forms.Button.WndProc(Message& m)
    ÂÂ at System.Windows.Forms.Control.ControlNativeWindow.OnMessage(Message& m)
    ÂÂ at System.Windows.Forms.Control.ControlNativeWindow.WndProc(Message& m)
    ÂÂ at System.Windows.Forms.NativeWindow.Callback(IntPtr hWnd, Int32 msg, IntPtr wparam, IntPtr lparam)


    ************** Loaded Assemblies **************
    mscorlib
    ÂÂÂÂAssembly Version: 2.0.0.0
    ÂÂÂÂWin32 Version: 2.0.50727.3603 (GDR.050727-3600)
    ÂÂÂÂCodeBase: file:///C:/WINDOWS/Microsoft.NET/Framework/v2.0.50727/mscorlib.dll
    ----------------------------------------
    SparseCopier
    ÂÂÂÂAssembly Version: 1.0.0.0
    ÂÂÂÂWin32 Version: 1.0.0.0
    ÂÂÂÂCodeBase: file:///D:/SparseCopier/bin/x86/Release/SparseCopier.exe
    ----------------------------------------
    Microsoft.VisualBasic
    ÂÂÂÂAssembly Version: 8.0.0.0
    ÂÂÂÂWin32 Version: 8.0.50727.3053 (netfxsp.050727-3000)
    ÂÂÂÂCodeBase: file:///C:/WINDOWS/assembly/GAC_MSIL/Microsoft.VisualBasic/8.0.0.0__b03f5f7f11d50a3a/Microsoft.VisualBasic.dll
    ----------------------------------------
    System
    ÂÂÂÂAssembly Version: 2.0.0.0
    ÂÂÂÂWin32 Version: 2.0.50727.3053 (netfxsp.050727-3000)
    ÂÂÂÂCodeBase: file:///C:/WINDOWS/assembly/GAC_MSIL/System/2.0.0.0__b77a5c561934e089/System.dll
    ----------------------------------------
    System.Windows.Forms
    ÂÂÂÂAssembly Version: 2.0.0.0
    ÂÂÂÂWin32 Version: 2.0.50727.3053 (netfxsp.050727-3000)
    ÂÂÂÂCodeBase: file:///C:/WINDOWS/assembly/GAC_MSIL/System.Windows.Forms/2.0.0.0__b77a5c561934e089/System.Windows.Forms.dll
    ----------------------------------------
    System.Drawing
    ÂÂÂÂAssembly Version: 2.0.0.0
    ÂÂÂÂWin32 Version: 2.0.50727.3053 (netfxsp.050727-3000)
    ÂÂÂÂCodeBase: file:///C:/WINDOWS/assembly/GAC_MSIL/System.Drawing/2.0.0.0__b03f5f7f11d50a3a/System.Drawing.dll
    ----------------------------------------
    System.Runtime.Remoting
    ÂÂÂÂAssembly Version: 2.0.0.0
    ÂÂÂÂWin32 Version: 2.0.50727.3053 (netfxsp.050727-3000)
    ÂÂÂÂCodeBase: file:///C:/WINDOWS/assembly/GAC_MSIL/System.Runtime.Remoting/2.0.0.0__b77a5c561934e089/System.Runtime.Remoting.dll
    ----------------------------------------
    System.Configuration
    ÂÂÂÂAssembly Version: 2.0.0.0
    ÂÂÂÂWin32 Version: 2.0.50727.3053 (netfxsp.050727-3000)
    ÂÂÂÂCodeBase: file:///C:/WINDOWS/assembly/GAC_MSIL/System.Configuration/2.0.0.0__b03f5f7f11d50a3a/System.Configuration.dll
    ----------------------------------------
    System.Xml
    ÂÂÂÂAssembly Version: 2.0.0.0
    ÂÂÂÂWin32 Version: 2.0.50727.3082 (QFE.050727-3000)
    ÂÂÂÂCodeBase: file:///C:/WINDOWS/assembly/GAC_MSIL/System.Xml/2.0.0.0__b77a5c561934e089/System.Xml.dll
    ----------------------------------------

    ************** JIT Debugging **************
    To enable just-in-time (JIT) debugging, the .config file for this
    application or computer (machine.config) must have the
    jitDebugging value set in the system.windows.forms section.
    The application must also be compiled with debugging
    enabled.

    For example:


    ÂÂÂÂ


    When JIT debugging is enabled, any unhandled exception
    will be sent to the JIT debugger registered on the computer
    rather than be handled by this dialog box.

    This tool might be very useful for me.. Thanks [​IMG]
     
  15. ChaosEnergy

    Member ChaosEnergy GBAtemp Regular

    Joined:
    Jul 11, 2009
    Messages:
    186
    Country:
    Germany
    it doe snot do anything for me
    i get the window, i can insert files and detsination
    click...it says done..but nothing has happend

    recompile is not possible for me, sln file is from 2008, my 2003 does not accept
     
  16. ClockWorK
    OP

    Member ClockWorK GBAtemp Regular

    Joined:
    May 10, 2009
    Messages:
    150
    Country:
    United States
    @chaosenergy
    hilite the files in the listbox that you want to copy.
    (also, I believe all of this code should work in VB.Net 2003 and 2005 as well.)


    @zurcuer:
    The error is occurring while trying to set the file size. Are you trying to copy to a drive that doesn't have 4.3 gigs free, or a FAT32 drive? Either would cause an error, but I wrote the app to check the file system type first.
     
  17. zx3junglist

    Newcomer zx3junglist Member

    Joined:
    Dec 24, 2009
    Messages:
    24
    Country:
    United States
    @ClockWorK
    I don't suppose you have any experience compiling for linux? I would assume we'd need mono to handle the .NET associations, but I'm no programmer. I am certainly interested in ways to fit more backups onto my relatively small USB drive. Currently I'm just dumping whole ISOs onto my NTFS partition, and at 4.7 Gb it fills up fast!

    Oh, and thanks for the heads up on the usb:/wbfs/GAMEID_title/GAMEID.iso structure, that worked straight away. I understand that NTFS support is in its relative infancy for the wii, so I'm sure there will be more developments as time goes on. Thanks ozzgee for your work on that!
     
  18. ClockWorK
    OP

    Member ClockWorK GBAtemp Regular

    Joined:
    May 10, 2009
    Messages:
    150
    Country:
    United States
    i think linux has a command-line option built in. but im not a linux guy.

    http://linux.about.com/od/commands/l/blcmdl1_cp.htm

    The code behind this program isn't that bad. It just requires some lower-level API calls, and it means the program is responsible for reading and writing every byte, including identifying areas in the file to write as sparse.

    Also, as oggzee mentioned, my code only works if the file was scrubbed with 00's (zeros) instead of FF's (255's). so, my code works on most ISO's coming out of a WBFS drive, but it doesn't work on every ISO you might download off the web.
     
  19. Wiimm

    Member Wiimm Developer

    Joined:
    Aug 11, 2009
    Messages:
    2,051
    Location:
    Germany
    Country:
    Germany
    yes, some linux command supports sparse files, like:
    cp --sparse=always
    rsync --sparse
     
  20. zx3junglist

    Newcomer zx3junglist Member

    Joined:
    Dec 24, 2009
    Messages:
    24
    Country:
    United States
    @Wiimm
    you're absolutely right, it looks like the GNU cp command that's included in my distro is version 7.2, and it looks like it has sparse destination files on by default. I sort of assumed that nautilus was using cp and mv to shuffle files around, but that might not be the case. All ISO files copied with nautilus were full size on my NTFS disk. I might try some tests just for curiosity's sale, but for now I've decided to use your wit tool (thanks!) to convert everything to .wbfs and place those on my NTFS partition. That way, I get the flexibility of being able to access that drive on a computer, but I also get the disk cost savings of a scrubbed wbfs file. I could always convert back if the need comes up. I'm hoping that in the near future, we can start loading your WDFs directly in the usb loader.
     

Share This Page