Hardware NKit :: Recover to Redump Images :: Shrink and Preserve Wii and GameCube Disc Images as Playable

[nanook]

There are two key uses of NKit:

• Recovery: Rebuild and repair images to match Redump verified images. This is aided by use of Recovery files that can be extracted from good sources. Recovery uses these files to rebuild the iso and repair modified disc headers, unscrub blocks, reinsert missing Wii partitions, move and reorder GameCube filesystems
• Convert: Converts anything as is. When converting to the NKit Format the source data is preserved so that it can be restored back to the source iso

Wiki & Download
https://wiki.gbatemp.net/wiki/NKit

User Guide
https://wiki.gbatemp.net/wiki/NKit/UserGuide

=============================================================
Version 1.4

• Fixed Hash Preservation for images with corrupt FSTs
• Fixed "Truncated RVT-R image detected..." message being shown incorrectly
• FST filenames are encoded are now correctly encoded with Shift-JIS

There should be no breaking changes or conversion difference for any previously successfully verified images. Please report any issues.

Version 1.3

• Added RVT-H Detection. This is not currently supported - An error is thrown and logged
• Fixed NkitReencode. It was always reencoding and not copying the NKit file when converting nkit.iso <-> nkit.gcz
• Fixed Verify not outputting any log detail messages e.g. When a removed Update Partition is required to verify and was not present - no message was output, it just failed
• Moved the SummaryLog to the Processed folder rather than the app folder to make it more obvious
• Added settings home support. ~ at the start of a Path will insert the user's home directory (all OSs)
• Revert To .Net 4.6.1 - mono wasn't handling 4.7.2 very well. The app well be ported to dotnet core 3 soon
• ProcessorApp: Added failure message to log errors (same as cmldline exe output)
• ProcessorApp: Disabled the checkbox for 'Remove and Preserve Wii Update Partitions' when Recovering or Converting to iso
• ProcessorApp: Fixed checkbox for 'Remove and Preserve Wii Update Partitions' not being set on app load
• ProcessorApp: Fixed progress .1.2.3... text alignment in the log output
• ProcessorApp: Fixed the output log being truncated when a partition ID contained a null char

There should be no breaking changes or conversion difference for any previously successfully verified images. Please report any issues.

Version 1.2

• Updated to .Net 4.7.2 (Preparation for long file name support)
• Added support for images that a file ends on a non 4 byte alignment (for 1 shrunk GC demo)
• Added support for images that have no Magic (default to GC - for 1 GC early system disc)
• Fixed the ExtractionApp GC Region always showing Japan
• Corrected NKitExtracted path in config
• Added missing Wii Jakers config entry
• Fixed Double "Truncated RVT-R" message
• Fixed ProcessorApp Gui Exception not being captured
• Fixed Drag Drop message being cleared when a non supported file is dropped
• Modified some public class accessibility to make them internal

There should be no breaking changes or conversion difference for any previously successfully verified images. Please report any issues.

Version 1.1

• NKit format fix: Fixed a bug when converting Wii to NKit. This was introduced during the last 2 days of development while adding support for a Customs preservation bug.
• Fixed a minor ProcessingApp bug where the output path wouldn't be used.

Any Converted NKit Wii images from v1.0 are not corrupt. Just not 100% correct to the format. Reconverting will correct the issue. Make sure NKitReencode is set to true if reconverting images that were made with v1.0.

Version 1.0
A lot of effort has gone in to ensuring this version is as stable and reliable as possible. The NKit Format is stable and reliable. Always use FullVerify as it's always accurate as are the Hashes when CalculateHashes is on. Please report any issues and ask any questions.

• Fixed a bug in the GC Format (This alters 12 of the GC Redump images - just reencode them if you verify against NKit dats)
• Added Wii NKit Format (100% Redump tested, Scrubbed & Custom Support)
• 2 Windows (Sorry Linux & Mac users) GUI Apps added, including file extraction

Version 0.5
First stable version

 

[johnsanc]

For historical reference, the old thread leading up to v1.0 is here: https://gbatemp.net/threads/new-app...isc-images-in-playable-formerly-swiit.533402/

I'm sure @nanook was tired and just wanted to get this out, but a big thing here is that this app supports two LOSSLESS formats for GC and Wii which are playable out of the box in Dolphin.

READ THE WIKI - There is a lot of good stuff in there that isn't documented anywhere else.

 

[nanook]

Hi all. Very quiet here Which is good, no immediate bugs.

Please give any feedback good or bad. The Windows Apps are very new and feature requests are welcome. No promises, just interested in what ideas people have.

Has anyone read the user guide or wiki? Does anything need explaining further?

Cheers

 

[nanook]

Sorry for the double post. If you're using recover - a config entry is missing

<!-- Wii: Jakers! Kart Racing (Europe) (En,Fr,De,Es,It,Sv,No,Da,Fi) - 1 bit junk diff. Redump is 08 - possible Single Event fail before hashes and encryption were generated -->
<add key="SJ3PNL0000_68568d32" value="00" />

Add that to the <junkRedumpPatch> section or recovery will fail if the source is scrubbed/WBFS.

 

[Bent]

I've been waiting for this program to come out and I just found out you created a new thread, I was following the old one and had no idea it was out already. That could be why this thread is quiet, maybe a post in the old thread instead of just editing the first post?

 

[FMonty]

I’d just like some clarification. The best format to reduce space is NKit.iso while removing update partition.

Then afterwards you can add additional compression with GCZ through dolphin and then can 7z?

 

[Zarper]

I’d just like some clarification. The best format to reduce space is NKit.iso while removing update partition.

Then afterwards you can add additional compression with GCZ through dolphin and then can 7z?

You would likely get way better compression if you skip the GCZ compression and just do the 7zip. GCZ will compress the data making it more random and way harder for 7zip to do its compression

 

[nanook]

@FMonty @Zarper Yes for best space saving - nkit.iso - 7zipped

Best playable is nkit.gcz (made with nkit, not dolphin). Just because NKit will make a slightly different GCZ

Update Partition removed. Keep the recovery folders so you can always get back to the original image 1:1

 

[Bent]

Thank you for this program. As I breifly mentioned last night, I have been waiting for this release. Some years ago I dumped all my GC collection, at the time I had a smaller hard drive and needed the space, so I scrubbed and 7zipped them. I've never redumped them but always wished I could now go back to unscrubbed without redumping. From the two images I tested, this everything seems to be great. I have something like 50 images to throw at it when I have time. I'll save them all as nkit once I know they are all reversible.

Also, I did the opposite with my Wii collection, I dumped all my discs and saved them unmodified since I had the hard drive space, yet it always seemed wasteful to me, saving all that garbage data. Now that it appears to be 100% reversible, I'll be saving these as nkit as well.

I'll report back how well it worked after I get a chance to do this.

 

[SpiffyJUNIOR]

About fucking time.
Thanks for spending so much time on this and updating us so often during the development process.

 

[matt!]

Is this open source so support for the format can be added to loaders? is that even desirable?

 

[Bent]

So I'm still in the process of archiving all my rips, but I have enough of an impression to post my thoughts/questions.

Wii
I thought all my Wii rips were 1:1 verified, but I found a corrupt one, potentially just from bit rot on my old hard drive that failed at one point. I had moved everything off and thought it was ok, but it had something like 7 random bytes where it differed from my re-rip. This tool was great for detecting it, though I guess I could have done that from the redump dats too.

I am interested in archiving my rips, not necessarily playability right now, so I am recovering to nkit and 7zipping them. Initial results are showing about a 70% reduction in space used, which is wonderful.


Gamecube
When I run my GC rips through the extraction app to get the recovery files, all my rips say the region is Japan, which is definitely not true, they are all US. Is this a bug? I'm using the GUI app.

My GC rips were initially scrubbed years ago with an old GC iso management app (I don't remember which one) and then 7zipped. For these, I had a lot more fail to verify, something like 15 or so. Some comparing with a hex editor shows 0x00's where there is data in the re-ripped image. Reading the wiki, it seems like not everything that the scrubbers removed may have been junk that was able to be procedurally generated, some of it might have been other random data. Is this assumption correct?

It is interesting to see Mario Party 5 in the special cases section in the config file. This game would always fail my scrubbing attempts in the past. I don't remember details since I probably did this 10 years ago, but I think scrubbing it would either corrupt the image or fail to do anything, and it always bothered me.

I haven't started 7zipping my GC rips yet, but I'm expecting only marginal space savings over my old method of scrubbing/7zipping, since this tool is doing basically the same thing, only safer.

Thanks a ton for this tool. I've always hoped for a better way to store Wii rips especially, that could still be restored and verified.

 

[nanook]

So I'm still in the process of archiving all my rips, but I have enough of an impression to post my thoughts/questions.

Wii
I thought all my Wii rips were 1:1 verified, but I found a corrupt one, potentially just from bit rot on my old hard drive that failed at one point. I had moved everything off and thought it was ok, but it had something like 7 random bytes where it differed from my re-rip. This tool was great for detecting it, though I guess I could have done that from the redump dats too.

I am interested in archiving my rips, not necessarily playability right now, so I am recovering to nkit and 7zipping them. Initial results are showing about a 70% reduction in space used, which is wonderful.


Gamecube
When I run my GC rips through the extraction app to get the recovery files, all my rips say the region is Japan, which is definitely not true, they are all US. Is this a bug? I'm using the GUI app.

My GC rips were initially scrubbed years ago with an old GC iso management app (I don't remember which one) and then 7zipped. For these, I had a lot more fail to verify, something like 15 or so. Some comparing with a hex editor shows 0x00's where there is data in the re-ripped image. Reading the wiki, it seems like not everything that the scrubbers removed may have been junk that was able to be procedurally generated, some of it might have been other random data. Is this assumption correct?

It is interesting to see Mario Party 5 in the special cases section in the config file. This game would always fail my scrubbing attempts in the past. I don't remember details since I probably did this 10 years ago, but I think scrubbing it would either corrupt the image or fail to do anything, and it always bothered me.

I haven't started 7zipping my GC rips yet, but I'm expecting only marginal space savings over my old method of scrubbing/7zipping, since this tool is doing basically the same thing, only safer.

Thanks a ton for this tool. I've always hoped for a better way to store Wii rips especially, that could still be restored and verified.

I'll check that the GUI is handling the regions correctly. It's very likely to just be displaying the wrong thing. Your images will be correct. Interesting about the 0x00s. Some scrubbers may have had buggy releases. NKit is proven to be able to be able to generate every byte of junk reliably for GC. I'm interested in which scrubber did this.

--------------------- MERGED ---------------------------

Is this open source so support for the format can be added to loaders? is that even desirable?

Source will be released soon. It's not required though to understand the format. The format is just Wii decrypted and dehashed (RVT-H).

 

[matt!]

Source will be released soon. It's not required though to understand the format. The format is just Wii decrypted and dehashed (RVT-H).

Then can you please help by linking to docs for that (standard?) format?

 

[johnsanc]

@matt! Here is info on RVT-H courtesy of GerbilSoft (who also helped out on this project in the background for his incredible knowledge of dev formats)
https://github.com/GerbilSoft/rvthtool/blob/master/doc/disc_image.md

 

[Bent]

I'll check that the GUI is handling the regions correctly. It's very likely to just be displaying the wrong thing. Your images will be correct. Interesting about the 0x00s. Some scrubbers may have had buggy releases. NKit is proven to be able to be able to generate every byte of junk reliably for GC. I'm interested in which scrubber did this.

What I did was turn off the "Delete recovery match failures" option, then recover my rips to iso, they said Match Failed. I then split this iso and the re-rip iso into 50MB chunks, and compared these with a comparison tool I use called Beyond Compare. Some of the chunks matched 100%, some of the rips only differed in one chunk, some in almost all of them. I didn't do a thorough comparison, but I did usually see zeros in my restored rip, and data in the re-rip.

By my best guess, I probably used GC-Tool by Paradox. The release I still have buried on my hard drive says it was released in 2003. Though the dates on all my rips say 2008, I think it may be my original rips I made over the BBA adapter with a tool called Phoenix. I know years later I redid everything with CleanRip, but it looks like I didn't save those rips for whatever reason. So it's possible that I had some bad rips too.

EDIT: It turns out it is the tool I used to scrub that is corrupting them, using GC-Tool to scrub some of my verified good re-rips gives me the same CRC for the scrubbed rip that is unable to be restored correctly.

 

[nanook]

Nice work @Bent. I appreciate the thoroughness to prove your theory.

 

[GerbilSoft]

The RVT-H unencrypted disc image format is basically the same as Wii format, except partitions are not encrypted and don't have hashes. So it's more like GameCube format with partitions.

Retail IOSes don't support this format normally, so the cIOS would need to be updated to allow it to work.

On the plus side, this might actually reduce load times a bit, since the data won't need to be decrypted and hashed on load.

 

[fledge68]

The RVT-H unencrypted disc image format is basically the same as Wii format, except partitions are not encrypted and don't have hashes. So it's more like GameCube format with partitions.

Retail IOSes don't support this format normally, so the cIOS would need to be updated to allow it to work.

On the plus side, this might actually reduce load times a bit, since the data won't need to be decrypted and hashed on load.

so if i understand correctly. someone would have to update cIOS in order for usb loaders to support nkit iso's?

 

[johnsanc]

Correct. Know anyone that would be interested in tackling that? It would be awesome to see that kind of support added to cIOS... that would also mean prototypes could be played on retail hardware.