Hardware NKit :: Recover to Redump Images :: Shrink and Preserve Wii and GameCube Disc Images as Playable

nanook

Well-Known Member
OP
Member
Joined
Dec 17, 2008
Messages
115
Trophies
0
Website
www.scorehero.com
XP
923
Country
There are two key uses of NKit:
  • Recovery: Rebuild and repair images to match Redump verified images. This is aided by use of Recovery files that can be extracted from good sources. Recovery uses these files to rebuild the iso and repair modified disc headers, unscrub blocks, reinsert missing Wii partitions, move and reorder GameCube filesystems
  • Convert: Converts anything as is. When converting to the NKit Format the source data is preserved so that it can be restored back to the source iso
Wiki & Download
https://wiki.gbatemp.net/wiki/NKit

User Guide
https://wiki.gbatemp.net/wiki/NKit/UserGuide

=============================================================
Version 1.4
  • Fixed Hash Preservation for images with corrupt FSTs
  • Fixed "Truncated RVT-R image detected..." message being shown incorrectly
  • FST filenames are encoded are now correctly encoded with Shift-JIS
There should be no breaking changes or conversion difference for any previously successfully verified images. Please report any issues.

Version 1.3
  • Added RVT-H Detection. This is not currently supported - An error is thrown and logged
  • Fixed NkitReencode. It was always reencoding and not copying the NKit file when converting nkit.iso <-> nkit.gcz
  • Fixed Verify not outputting any log detail messages e.g. When a removed Update Partition is required to verify and was not present - no message was output, it just failed
  • Moved the SummaryLog to the Processed folder rather than the app folder to make it more obvious
  • Added settings home support. ~ at the start of a Path will insert the user's home directory (all OSs)
  • Revert To .Net 4.6.1 - mono wasn't handling 4.7.2 very well. The app well be ported to dotnet core 3 soon
  • ProcessorApp: Added failure message to log errors (same as cmldline exe output)
  • ProcessorApp: Disabled the checkbox for 'Remove and Preserve Wii Update Partitions' when Recovering or Converting to iso
  • ProcessorApp: Fixed checkbox for 'Remove and Preserve Wii Update Partitions' not being set on app load
  • ProcessorApp: Fixed progress .1.2.3... text alignment in the log output
  • ProcessorApp: Fixed the output log being truncated when a partition ID contained a null char
There should be no breaking changes or conversion difference for any previously successfully verified images. Please report any issues.

Version 1.2
  • Updated to .Net 4.7.2 (Preparation for long file name support)
  • Added support for images that a file ends on a non 4 byte alignment (for 1 shrunk GC demo)
  • Added support for images that have no Magic (default to GC - for 1 GC early system disc)
  • Fixed the ExtractionApp GC Region always showing Japan
  • Corrected NKitExtracted path in config
  • Added missing Wii Jakers config entry
  • Fixed Double "Truncated RVT-R" message
  • Fixed ProcessorApp Gui Exception not being captured
  • Fixed Drag Drop message being cleared when a non supported file is dropped
  • Modified some public class accessibility to make them internal
There should be no breaking changes or conversion difference for any previously successfully verified images. Please report any issues.

Version 1.1
  • NKit format fix: Fixed a bug when converting Wii to NKit. This was introduced during the last 2 days of development while adding support for a Customs preservation bug.
  • Fixed a minor ProcessingApp bug where the output path wouldn't be used.
Any Converted NKit Wii images from v1.0 are not corrupt. Just not 100% correct to the format. Reconverting will correct the issue. Make sure NKitReencode is set to true if reconverting images that were made with v1.0.

Version 1.0
A lot of effort has gone in to ensuring this version is as stable and reliable as possible. The NKit Format is stable and reliable. Always use FullVerify as it's always accurate as are the Hashes when CalculateHashes is on. Please report any issues and ask any questions.

  • Fixed a bug in the GC Format (This alters 12 of the GC Redump images - just reencode them if you verify against NKit dats)
  • Added Wii NKit Format (100% Redump tested, Scrubbed & Custom Support)
  • 2 Windows (Sorry Linux & Mac users) GUI Apps added, including file extraction
Version 0.5
First stable version
 
Last edited by nanook,

johnsanc

Well-Known Member
Newcomer
Joined
Sep 27, 2016
Messages
46
Trophies
0
Age
37
XP
777
Country
United States

nanook

Well-Known Member
OP
Member
Joined
Dec 17, 2008
Messages
115
Trophies
0
Website
www.scorehero.com
XP
923
Country
Hi all. Very quiet here :) Which is good, no immediate bugs.

Please give any feedback good or bad. The Windows Apps are very new and feature requests are welcome. No promises, just interested in what ideas people have.

Has anyone read the user guide or wiki? Does anything need explaining further?

Cheers
 

nanook

Well-Known Member
OP
Member
Joined
Dec 17, 2008
Messages
115
Trophies
0
Website
www.scorehero.com
XP
923
Country
Sorry for the double post. If you're using recover - a config entry is missing

<!-- Wii: Jakers! Kart Racing (Europe) (En,Fr,De,Es,It,Sv,No,Da,Fi) - 1 bit junk diff. Redump is 08 - possible Single Event fail before hashes and encryption were generated -->
<add key="SJ3PNL0000_68568d32" value="00" />


Add that to the <junkRedumpPatch> section or recovery will fail if the source is scrubbed/WBFS.
 

Bent

Well-Known Member
Member
Joined
Nov 6, 2002
Messages
191
Trophies
0
Age
39
Website
Visit site
XP
1,754
Country
United States
I've been waiting for this program to come out and I just found out you created a new thread, I was following the old one and had no idea it was out already. That could be why this thread is quiet, maybe a post in the old thread instead of just editing the first post?
 
  • Like
Reactions: nanook

FMonty

Member
Newcomer
Joined
Mar 17, 2019
Messages
7
Trophies
0
Age
30
XP
153
Country
United Kingdom
I’d just like some clarification. The best format to reduce space is NKit.iso while removing update partition.

Then afterwards you can add additional compression with GCZ through dolphin and then can 7z?
 

Zarper

Member
Newcomer
Joined
Sep 13, 2019
Messages
8
Trophies
0
Age
34
XP
719
Country
Sweden
I’d just like some clarification. The best format to reduce space is NKit.iso while removing update partition.

Then afterwards you can add additional compression with GCZ through dolphin and then can 7z?
You would likely get way better compression if you skip the GCZ compression and just do the 7zip. GCZ will compress the data making it more random and way harder for 7zip to do its compression
 

nanook

Well-Known Member
OP
Member
Joined
Dec 17, 2008
Messages
115
Trophies
0
Website
www.scorehero.com
XP
923
Country
@FMonty @Zarper Yes for best space saving - nkit.iso - 7zipped

Best playable is nkit.gcz (made with nkit, not dolphin). Just because NKit will make a slightly different GCZ

Update Partition removed. Keep the recovery folders so you can always get back to the original image 1:1
 
Last edited by nanook,

Bent

Well-Known Member
Member
Joined
Nov 6, 2002
Messages
191
Trophies
0
Age
39
Website
Visit site
XP
1,754
Country
United States
Thank you for this program. As I breifly mentioned last night, I have been waiting for this release. Some years ago I dumped all my GC collection, at the time I had a smaller hard drive and needed the space, so I scrubbed and 7zipped them. I've never redumped them but always wished I could now go back to unscrubbed without redumping. From the two images I tested, this everything seems to be great. I have something like 50 images to throw at it when I have time. I'll save them all as nkit once I know they are all reversible.

Also, I did the opposite with my Wii collection, I dumped all my discs and saved them unmodified since I had the hard drive space, yet it always seemed wasteful to me, saving all that garbage data. Now that it appears to be 100% reversible, I'll be saving these as nkit as well.

I'll report back how well it worked after I get a chance to do this.
 

Bent

Well-Known Member
Member
Joined
Nov 6, 2002
Messages
191
Trophies
0
Age
39
Website
Visit site
XP
1,754
Country
United States
So I'm still in the process of archiving all my rips, but I have enough of an impression to post my thoughts/questions.

Wii
I thought all my Wii rips were 1:1 verified, but I found a corrupt one, potentially just from bit rot on my old hard drive that failed at one point. I had moved everything off and thought it was ok, but it had something like 7 random bytes where it differed from my re-rip. This tool was great for detecting it, though I guess I could have done that from the redump dats too.

I am interested in archiving my rips, not necessarily playability right now, so I am recovering to nkit and 7zipping them. Initial results are showing about a 70% reduction in space used, which is wonderful.


Gamecube
When I run my GC rips through the extraction app to get the recovery files, all my rips say the region is Japan, which is definitely not true, they are all US. Is this a bug? I'm using the GUI app.

My GC rips were initially scrubbed years ago with an old GC iso management app (I don't remember which one) and then 7zipped. For these, I had a lot more fail to verify, something like 15 or so. Some comparing with a hex editor shows 0x00's where there is data in the re-ripped image. Reading the wiki, it seems like not everything that the scrubbers removed may have been junk that was able to be procedurally generated, some of it might have been other random data. Is this assumption correct?

It is interesting to see Mario Party 5 in the special cases section in the config file. This game would always fail my scrubbing attempts in the past. I don't remember details since I probably did this 10 years ago, but I think scrubbing it would either corrupt the image or fail to do anything, and it always bothered me.

I haven't started 7zipping my GC rips yet, but I'm expecting only marginal space savings over my old method of scrubbing/7zipping, since this tool is doing basically the same thing, only safer.

Thanks a ton for this tool. I've always hoped for a better way to store Wii rips especially, that could still be restored and verified.
 
  • Like
Reactions: nanook

nanook

Well-Known Member
OP
Member
Joined
Dec 17, 2008
Messages
115
Trophies
0
Website
www.scorehero.com
XP
923
Country
So I'm still in the process of archiving all my rips, but I have enough of an impression to post my thoughts/questions.

Wii
I thought all my Wii rips were 1:1 verified, but I found a corrupt one, potentially just from bit rot on my old hard drive that failed at one point. I had moved everything off and thought it was ok, but it had something like 7 random bytes where it differed from my re-rip. This tool was great for detecting it, though I guess I could have done that from the redump dats too.

I am interested in archiving my rips, not necessarily playability right now, so I am recovering to nkit and 7zipping them. Initial results are showing about a 70% reduction in space used, which is wonderful.


Gamecube
When I run my GC rips through the extraction app to get the recovery files, all my rips say the region is Japan, which is definitely not true, they are all US. Is this a bug? I'm using the GUI app.

My GC rips were initially scrubbed years ago with an old GC iso management app (I don't remember which one) and then 7zipped. For these, I had a lot more fail to verify, something like 15 or so. Some comparing with a hex editor shows 0x00's where there is data in the re-ripped image. Reading the wiki, it seems like not everything that the scrubbers removed may have been junk that was able to be procedurally generated, some of it might have been other random data. Is this assumption correct?

It is interesting to see Mario Party 5 in the special cases section in the config file. This game would always fail my scrubbing attempts in the past. I don't remember details since I probably did this 10 years ago, but I think scrubbing it would either corrupt the image or fail to do anything, and it always bothered me.

I haven't started 7zipping my GC rips yet, but I'm expecting only marginal space savings over my old method of scrubbing/7zipping, since this tool is doing basically the same thing, only safer.

Thanks a ton for this tool. I've always hoped for a better way to store Wii rips especially, that could still be restored and verified.

I'll check that the GUI is handling the regions correctly. It's very likely to just be displaying the wrong thing. Your images will be correct. Interesting about the 0x00s. Some scrubbers may have had buggy releases. NKit is proven to be able to be able to generate every byte of junk reliably for GC. I'm interested in which scrubber did this.

--------------------- MERGED ---------------------------

Is this open source so support for the format can be added to loaders? is that even desirable?

Source will be released soon. It's not required though to understand the format. The format is just Wii decrypted and dehashed (RVT-H).
 

Bent

Well-Known Member
Member
Joined
Nov 6, 2002
Messages
191
Trophies
0
Age
39
Website
Visit site
XP
1,754
Country
United States
I'll check that the GUI is handling the regions correctly. It's very likely to just be displaying the wrong thing. Your images will be correct. Interesting about the 0x00s. Some scrubbers may have had buggy releases. NKit is proven to be able to be able to generate every byte of junk reliably for GC. I'm interested in which scrubber did this.

What I did was turn off the "Delete recovery match failures" option, then recover my rips to iso, they said Match Failed. I then split this iso and the re-rip iso into 50MB chunks, and compared these with a comparison tool I use called Beyond Compare. Some of the chunks matched 100%, some of the rips only differed in one chunk, some in almost all of them. I didn't do a thorough comparison, but I did usually see zeros in my restored rip, and data in the re-rip.

By my best guess, I probably used GC-Tool by Paradox. The release I still have buried on my hard drive says it was released in 2003. Though the dates on all my rips say 2008, I think it may be my original rips I made over the BBA adapter with a tool called Phoenix. I know years later I redid everything with CleanRip, but it looks like I didn't save those rips for whatever reason. So it's possible that I had some bad rips too.

EDIT: It turns out it is the tool I used to scrub that is corrupting them, using GC-Tool to scrub some of my verified good re-rips gives me the same CRC for the scrubbed rip that is unable to be restored correctly.
 
Last edited by Bent,

GerbilSoft

Well-Known Member
Member
Joined
Mar 8, 2012
Messages
2,378
Trophies
1
Age
32
XP
3,915
Country
United States
The RVT-H unencrypted disc image format is basically the same as Wii format, except partitions are not encrypted and don't have hashes. So it's more like GameCube format with partitions.

Retail IOSes don't support this format normally, so the cIOS would need to be updated to allow it to work.

On the plus side, this might actually reduce load times a bit, since the data won't need to be decrypted and hashed on load.
 

fledge68

Well-Known Member
Member
Joined
Jan 30, 2012
Messages
2,220
Trophies
1
XP
3,774
Country
United States
The RVT-H unencrypted disc image format is basically the same as Wii format, except partitions are not encrypted and don't have hashes. So it's more like GameCube format with partitions.

Retail IOSes don't support this format normally, so the cIOS would need to be updated to allow it to work.

On the plus side, this might actually reduce load times a bit, since the data won't need to be decrypted and hashed on load.
so if i understand correctly. someone would have to update cIOS in order for usb loaders to support nkit iso's?
 
  • Like
Reactions: biggzsizzle

johnsanc

Well-Known Member
Newcomer
Joined
Sep 27, 2016
Messages
46
Trophies
0
Age
37
XP
777
Country
United States
Correct. Know anyone that would be interested in tackling that? It would be awesome to see that kind of support added to cIOS... that would also mean prototypes could be played on retail hardware.
 
  • Like
Reactions: ChibiMofo
General chit-chat
Help Users
  • No one is chatting at the moment.
    KennieDaMeanie @ KennieDaMeanie: lol someone's going wild with the +1s +1