Hacking Making Sparse Files (for .ISO on NTFS)

[ClockWorK]

I'm just going to post this and see if there's any interest.


So, I recently made the jump from WBFS to FAT32 with .WBFS files, and then to NTFS with .ISO files.

I learned a lot about sparse files along the way. I was not familiar with them at all. The basic concept is that the file system can encode a long string of zero-bytes in a condensed format to save space on disk. The down-side to sparse files is that they can become fragmented when they are modified, and have more overhead for operations that write to the file.

Sparse files are useful for scrubbed Wii ISOs on NTFS because it allows the user to have a 4.3gig ISO on the drive but have it only take up it's scrubbed size on disk.

Unfortunately, I've found that copying a sparse file from one NTFS partition to another NTFS partition creates a non-sparse file in the destination.

So, I've written a program that will take a bunch of ISOs (or any files really), and copy them to a specified destination folder. But, instead of just doing a normal copy, it will re-write the destination file as a sparse file. This can be confirmed by right-clicking on the file and choosing 'Properties'. The .ISO file will show that it is 4,699,979,776 bytes. The size on disk will be less than this, possibly by a significant amount.

Interestingly, VB.Net 2008 does not seem to expose the required methods to perform this task using managed code. I've included the source code with the app. Basically, if a string of zero-bytes is encountered that is more than 128,000 bytes (the threshold I came decided to use), the program encodes the string of zero-bytes as sparse zeros. Optionally, the program can also compare the content of the input and output files to ensure they match.

To confirm that this works correctly, I've processed a few .ISO files and then run the input and the output through a file-compare. Everything has come out OK (100% match). However, if you are interested in trying this, I recommend keeping your original file and performing a compare with a third-party tool before getting rid of any of your original ISO files.

Maybe there's already a tool for this that I'm not aware of. Either way, maybe the source will be useful to someone. (if you've never used CopyMemory in VB, this code will be a new experience for you). I'd like to know if there's any interest in this program.

I haven't tested on Vista or Win7 yet.


Application SourceCode and Executable:
http://www.megaupload.com/?d=H0UMU6Y9

 

[Eddie_Brock]

Cool concept! Thanks for the program

I am currently waiting for more Loaders to support NTFS, and then I will gladly jump-ship back to NTFS, as its my preffered file system. This is a useful app for saving space, great work.

 

[oggzee]

@ClockWorK:

I would recommend that you verify that the created files are not fragmented.
To prevent fragmentation you have to first pre-allocate file size and only then enable sparse mode.
Look at wbfs_file source (libwbfs_win32.c) to see what i mean.
To check the file fragmentation this is a good tool:
http://www.mutexed.com/code/FragExt/#defrag-manager
See this post for more info:
http://gbatemp.net/index.php?s=&showto...t&p=2470297
Also, in my opinion a good chunk size to check for zeroes is 32k (32768 bytes) since that is the wii sector size.
And there's another issue which has to be considered, wiiscrubber will fill the unused data with 0xFF not zeroes, so before you can use your tool to copy and create sparse files the iso will have to be scrubbed using zeroes. (which wbfs_file and wwt does)

 

[ClockWorK]

@Oggzee:

Thanks for the tip on setting the filesize first. That reduced my fragment count from 30+ to 7. Does that seem right, or are we looking for 1 single fragment?

I set the chunk size to 128k to help prevent fragmentation. Since most of the sparse data in a Wii ISO is in one or two blocks, I don't think the larger chunksize will be any problem or cost much extra in terms of space. However, as mentioned, I'm still new to sparse files. If you say make it 32k, then 32k is what it will be.

I'm not really familiar with Wii Scrubber, or most of the Wii ISO layout. I am quite competent writing programs that process and modify binary data, so it's feasible that I could add some other functions (for fixing the 0xFFs, as an example). But for now, this app is just a generic sparse file copier, nothing Wii-specific.

If there's a need for any other tools, I am always willing to lend my coding skills.

 

[oggzee]

@Oggzee:
Thanks for the tip on setting the filesize first. That reduced my fragment count from 30+ to 7. Does that seem right, or are we looking for 1 single fragment?

Yes, that is ok, each sparse block will add up to the fragment count too.
Usually there will be around 6 fragments because there will be 3 data blocks: header, update partition, game partition, and the sparse blocks around these.

 

[ClockWorK]

From looking at it, I had assumed that each change from sparse to non-sparse added a fragment.

I'm feeling pretty confident that this works, and now the fragmentation is addressed.


Now the real question is, is anybody actually going to want this?

 

[XFlak]

I would like to have this, so once u post a link I will download for sure... but to be honest i doubt I will use it, I like using .wbfs files instead because then I can transfer them from 1 drive to another using windows explorer (without un-compressing/un-scrubbing it). Also, wii backup manager (by fig2k4) supports converting to sparse ISOs (this may not be explicitly stated, but I believe it is mentioned somewhere and it is either the only setting or the default setting for extracting an ISO).

I maintain my own personal database of all wii-related computer programs, and I would love to add yours to the list.

I don't care if some people might call this app 'redundant', the more options available the better, and the fact that you r releasing your source is also great! If u found a more efficient way to sparse ISO's (compared to other apps), it would be great for your code to be added to other programs (like wii backup manager)!

All in all, great work, and thank you!

 

[ClockWorK]

Original Post updated with link to SourceCode and Executable.

Thanks for the feedback, xflak40

 

[WiiFan2012]

I can't wait until USB Loaders support NTFS also! That, is also, my favorite preferred format. Mostly because it is the most common file format included in todays Windows operating systems.

But, for now, I have to stick to the bottom of the Atlantic (WBFS), as it is the primary for Wii Backups today.

 

[zx3junglist]

I thought that USB loader GX is supporting ntfs since r867. http://sites.google.com/site/usbloadergui/news/ntfssupport

I have formatted a fresh NTFS drive and dropped an untouched .ISO onto it, but even with r874, I get no joy. I'm probably doing something wrong, I'm thinking maybe I still need to convert to .wbfs.

 

[ClockWorK]

I thought that USB loader GX is supporting ntfs since r867. http://sites.google.com/site/usbloadergui/news/ntfssupport

I have formatted a fresh NTFS drive and dropped an untouched .ISO onto it, but even with r874, I get no joy. I'm probably doing something wrong, I'm thinking maybe I still need to convert to .wbfs.

for NTFS, they must be named /wbfs/GAMEID.ISO or /wbfs/GAMEID_GameTitle/GAMEID.ISO
(coincidentally, I wrote an app for this as well.)


Similar to FAT32 with the .wbfs files.

 

[mousex]

I thought that USB loader GX is supporting ntfs since r867. http://sites.google.com/site/usbloadergui/news/ntfssupport

I have formatted a fresh NTFS drive and dropped an untouched .ISO onto it, but even with r874, I get no joy. I'm probably doing something wrong, I'm thinking maybe I still need to convert to .wbfs.
Use cfg Loader, GX is just too buggy and they just use the code from cfg Loader. Oggzee's tool will always work better on FAT and NTFS because it is the first tool that get's fixes.

@ClockWorK: Is it a difference using your tool or cp with sparse option?
QUOTE(oggzee @ Dec 23 2009, 10:51 PM) Yes, that is ok, each sparse block will add up to the fragment count too.
Usually there will be around 6 fragments because there will be 3 data blocks: header, update partition, game partition, and the sparse blocks around these.

If you put the normal and sparse blocks directly behind each other shouldn't they be just one big fragment? I mean, this is what happens if you defrag the drive.

 

[ChaosEnergy]

maybe you create a switch to choice the blocksize by each one
this tool will do, what i asked already for
another switch to scrub or not
a universal sparse program...there are not only wii images on a disk

 

[zurcuer]

I'm getting an unhandled exception

CODESee the end of this message for details on invoking
just-in-time (JIT) debugging instead of this dialog box.

************** Exception Text **************
System.Exception: Application-defined or object-defined error.
ÂÂ at Microsoft.VisualBasic.ErrObject.Raise(Int32 Number, Object Source, Object Description, Object HelpFile, Object HelpContext)
ÂÂ at SparseCopier.modFileAPI.FileNum_SetFileSize(Int32 theFileNumber, Int64 fileLength) in C:\My Work\Programming\VB.Net\Test\SparseCopier\modFileAPI.vb:line 772
ÂÂ at SparseCopier.modFileAPI.File_CopyToSparseNTFSfile(String inpFileName, String newFileName) in C:\My Work\Programming\VB.Net\Test\SparseCopier\modFileAPI.vb:line 1507
ÂÂ at SparseCopier.Form1.Button2_Click(Object sender, EventArgs e) in C:\My Work\Programming\VB.Net\Test\SparseCopier\Form1.vb:line 89
ÂÂ at System.Windows.Forms.Control.OnClick(EventArgs e)
ÂÂ at System.Windows.Forms.Button.OnClick(EventArgs e)
ÂÂ at System.Windows.Forms.Button.OnMouseUp(MouseEventArgs mevent)
ÂÂ at System.Windows.Forms.Control.WmMouseUp(Message& m, MouseButtons button, Int32 clicks)
ÂÂ at System.Windows.Forms.Control.WndProc(Message& m)
ÂÂ at System.Windows.Forms.ButtonBase.WndProc(Message& m)
ÂÂ at System.Windows.Forms.Button.WndProc(Message& m)
ÂÂ at System.Windows.Forms.Control.ControlNativeWindow.OnMessage(Message& m)
ÂÂ at System.Windows.Forms.Control.ControlNativeWindow.WndProc(Message& m)
ÂÂ at System.Windows.Forms.NativeWindow.Callback(IntPtr hWnd, Int32 msg, IntPtr wparam, IntPtr lparam)


************** Loaded Assemblies **************
mscorlib
ÂÂÂÂAssembly Version: 2.0.0.0
ÂÂÂÂWin32 Version: 2.0.50727.3603 (GDR.050727-3600)
ÂÂÂÂCodeBase: file:///C:/WINDOWS/Microsoft.NET/Framework/v2.0.50727/mscorlib.dll
----------------------------------------
SparseCopier
ÂÂÂÂAssembly Version: 1.0.0.0
ÂÂÂÂWin32 Version: 1.0.0.0
ÂÂÂÂCodeBase: file:///D:/SparseCopier/bin/x86/Release/SparseCopier.exe
----------------------------------------
Microsoft.VisualBasic
ÂÂÂÂAssembly Version: 8.0.0.0
ÂÂÂÂWin32 Version: 8.0.50727.3053 (netfxsp.050727-3000)
ÂÂÂÂCodeBase: file:///C:/WINDOWS/assembly/GAC_MSIL/Microsoft.VisualBasic/8.0.0.0__b03f5f7f11d50a3a/Microsoft.VisualBasic.dll
----------------------------------------
System
ÂÂÂÂAssembly Version: 2.0.0.0
ÂÂÂÂWin32 Version: 2.0.50727.3053 (netfxsp.050727-3000)
ÂÂÂÂCodeBase: file:///C:/WINDOWS/assembly/GAC_MSIL/System/2.0.0.0__b77a5c561934e089/System.dll
----------------------------------------
System.Windows.Forms
ÂÂÂÂAssembly Version: 2.0.0.0
ÂÂÂÂWin32 Version: 2.0.50727.3053 (netfxsp.050727-3000)
ÂÂÂÂCodeBase: file:///C:/WINDOWS/assembly/GAC_MSIL/System.Windows.Forms/2.0.0.0__b77a5c561934e089/System.Windows.Forms.dll
----------------------------------------
System.Drawing
ÂÂÂÂAssembly Version: 2.0.0.0
ÂÂÂÂWin32 Version: 2.0.50727.3053 (netfxsp.050727-3000)
ÂÂÂÂCodeBase: file:///C:/WINDOWS/assembly/GAC_MSIL/System.Drawing/2.0.0.0__b03f5f7f11d50a3a/System.Drawing.dll
----------------------------------------
System.Runtime.Remoting
ÂÂÂÂAssembly Version: 2.0.0.0
ÂÂÂÂWin32 Version: 2.0.50727.3053 (netfxsp.050727-3000)
ÂÂÂÂCodeBase: file:///C:/WINDOWS/assembly/GAC_MSIL/System.Runtime.Remoting/2.0.0.0__b77a5c561934e089/System.Runtime.Remoting.dll
----------------------------------------
System.Configuration
ÂÂÂÂAssembly Version: 2.0.0.0
ÂÂÂÂWin32 Version: 2.0.50727.3053 (netfxsp.050727-3000)
ÂÂÂÂCodeBase: file:///C:/WINDOWS/assembly/GAC_MSIL/System.Configuration/2.0.0.0__b03f5f7f11d50a3a/System.Configuration.dll
----------------------------------------
System.Xml
ÂÂÂÂAssembly Version: 2.0.0.0
ÂÂÂÂWin32 Version: 2.0.50727.3082 (QFE.050727-3000)
ÂÂÂÂCodeBase: file:///C:/WINDOWS/assembly/GAC_MSIL/System.Xml/2.0.0.0__b77a5c561934e089/System.Xml.dll
----------------------------------------

************** JIT Debugging **************
To enable just-in-time (JIT) debugging, the .config file for this
application or computer (machine.config) must have the
jitDebugging value set in the system.windows.forms section.
The application must also be compiled with debugging
enabled.

For example:


ÂÂÂÂ


When JIT debugging is enabled, any unhandled exception
will be sent to the JIT debugger registered on the computer
rather than be handled by this dialog box.

This tool might be very useful for me.. Thanks

 

[ChaosEnergy]

it doe snot do anything for me
i get the window, i can insert files and detsination
click...it says done..but nothing has happend

recompile is not possible for me, sln file is from 2008, my 2003 does not accept

 

[ClockWorK]

@chaosenergy
hilite the files in the listbox that you want to copy.
(also, I believe all of this code should work in VB.Net 2003 and 2005 as well.)


@zurcuer:
The error is occurring while trying to set the file size. Are you trying to copy to a drive that doesn't have 4.3 gigs free, or a FAT32 drive? Either would cause an error, but I wrote the app to check the file system type first.

 

[zx3junglist]

@ClockWorK
I don't suppose you have any experience compiling for linux? I would assume we'd need mono to handle the .NET associations, but I'm no programmer. I am certainly interested in ways to fit more backups onto my relatively small USB drive. Currently I'm just dumping whole ISOs onto my NTFS partition, and at 4.7 Gb it fills up fast!

Oh, and thanks for the heads up on the usb:/wbfs/GAMEID_title/GAMEID.iso structure, that worked straight away. I understand that NTFS support is in its relative infancy for the wii, so I'm sure there will be more developments as time goes on. Thanks ozzgee for your work on that!

 

[ClockWorK]

i think linux has a command-line option built in. but im not a linux guy.

http://linux.about.com/od/commands/l/blcmdl1_cp.htm

The code behind this program isn't that bad. It just requires some lower-level API calls, and it means the program is responsible for reading and writing every byte, including identifying areas in the file to write as sparse.

Also, as oggzee mentioned, my code only works if the file was scrubbed with 00's (zeros) instead of FF's (255's). so, my code works on most ISO's coming out of a WBFS drive, but it doesn't work on every ISO you might download off the web.

 

[Wiimm]

yes, some linux command supports sparse files, like:
cp --sparse=always
rsync --sparse

 

[zx3junglist]

yes, some linux command supports sparse files, like:
cp --sparse=always
rsync --sparse

@Wiimm
you're absolutely right, it looks like the GNU cp command that's included in my distro is version 7.2, and it looks like it has sparse destination files on by default. I sort of assumed that nautilus was using cp and mv to shuffle files around, but that might not be the case. All ISO files copied with nautilus were full size on my NTFS disk. I might try some tests just for curiosity's sale, but for now I've decided to use your wit tool (thanks!) to convert everything to .wbfs and place those on my NTFS partition. That way, I get the flexibility of being able to access that drive on a computer, but I also get the disk cost savings of a scrubbed wbfs file. I could always convert back if the need comes up. I'm hoping that in the near future, we can start loading your WDFs directly in the usb loader.