NKit :: Recover to Redump Images :: Shrink and Preserve Wii and GameCube Disc Images as Playable

Discussion in 'Wii - Hardware, Devices and Utilities' started by nanook, Sep 20, 2019.

  1. nanook
    OP

    nanook Advanced Member

    Newcomer
    5
    Dec 17, 2008
    There are two key uses of NKit:
    • Recovery: Rebuild and repair images to match Redump verified images. This is aided by use of Recovery files that can be extracted from good sources. Recovery uses these files to rebuild the iso and repair modified disc headers, unscrub blocks, reinsert missing Wii partitions, move and reorder GameCube filesystems
    • Convert: Converts anything as is. When converting to the NKit Format the source data is preserved so that it can be restored back to the source iso
    Wiki & Download
    https://wiki.gbatemp.net/wiki/NKit

    User Guide
    https://wiki.gbatemp.net/wiki/NKit/UserGuide

    =============================================================

    Version 1.3
    • Added RVT-H Detection. This is not currently supported - An error is thrown and logged
    • Fixed NkitReencode. It was always reencoding and not copying the NKit file when converting nkit.iso <-> nkit.gcz
    • Fixed Verify not outputting any log detail messages e.g. When a removed Update Partition is required to verify and was not present - no message was output, it just failed
    • Moved the SummaryLog to the Processed folder rather than the app folder to make it more obvious
    • Added settings home support. ~ at the start of a Path will insert the user's home directory (all OSs)
    • Revert To .Net 4.6.1 - mono wasn't handling 4.7.2 very well. The app well be ported to dotnet core 3 soon
    • ProcessorApp: Added failure message to log errors (same as cmldline exe output)
    • ProcessorApp: Disabled the checkbox for 'Remove and Preserve Wii Update Partitions' when Recovering or Converting to iso
    • ProcessorApp: Fixed checkbox for 'Remove and Preserve Wii Update Partitions' not being set on app load
    • ProcessorApp: Fixed progress .1.2.3... text alignment in the log output
    • ProcessorApp: Fixed the output log being truncated when a partition ID contained a null char
    There should be no breaking changes or conversion difference for any previously successfully verified images. Please report any issues.

    Version 1.2
    • Updated to .Net 4.7.2 (Preparation for long file name support)
    • Added support for images that a file ends on a non 4 byte alignment (for 1 shrunk GC demo)
    • Added support for images that have no Magic (default to GC - for 1 GC early system disc)
    • Fixed the ExtractionApp GC Region always showing Japan
    • Corrected NKitExtracted path in config
    • Added missing Wii Jakers config entry
    • Fixed Double "Truncated RVT-R" message
    • Fixed ProcessorApp Gui Exception not being captured
    • Fixed Drag Drop message being cleared when a non supported file is dropped
    • Modified some public class accessibility to make them internal
    There should be no breaking changes or conversion difference for any previously successfully verified images. Please report any issues.

    Version 1.1
    • NKit format fix: Fixed a bug when converting Wii to NKit. This was introduced during the last 2 days of development while adding support for a Customs preservation bug.
    • Fixed a minor ProcessingApp bug where the output path wouldn't be used.
    Any Converted NKit Wii images from v1.0 are not corrupt. Just not 100% correct to the format. Reconverting will correct the issue. Make sure NKitReencode is set to true if reconverting images that were made with v1.0.

    Version 1.0
    A lot of effort has gone in to ensuring this version is as stable and reliable as possible. The NKit Format is stable and reliable. Always use FullVerify as it's always accurate as are the Hashes when CalculateHashes is on. Please report any issues and ask any questions.

    • Fixed a bug in the GC Format (This alters 12 of the GC Redump images - just reencode them if you verify against NKit dats)
    • Added Wii NKit Format (100% Redump tested, Scrubbed & Custom Support)
    • 2 Windows (Sorry Linux & Mac users) GUI Apps added, including file extraction
    Version 0.5
    First stable version

     
    Last edited by nanook, Oct 15, 2019 at 1:13 AM
  2. johnsanc

    johnsanc Member

    Newcomer
    5
    Sep 27, 2016
    United States
  3. nanook
    OP

    nanook Advanced Member

    Newcomer
    5
    Dec 17, 2008
    Hi all. Very quiet here :) Which is good, no immediate bugs.

    Please give any feedback good or bad. The Windows Apps are very new and feature requests are welcome. No promises, just interested in what ideas people have.

    Has anyone read the user guide or wiki? Does anything need explaining further?

    Cheers
     
  4. nanook
    OP

    nanook Advanced Member

    Newcomer
    5
    Dec 17, 2008
    Sorry for the double post. If you're using recover - a config entry is missing

    <!-- Wii: Jakers! Kart Racing (Europe) (En,Fr,De,Es,It,Sv,No,Da,Fi) - 1 bit junk diff. Redump is 08 - possible Single Event fail before hashes and encryption were generated -->
    <add key="SJ3PNL0000_68568d32" value="00" />


    Add that to the <junkRedumpPatch> section or recovery will fail if the source is scrubbed/WBFS.
     
  5. Bent

    Bent GBAtemp Regular

    Member
    6
    Nov 6, 2002
    United States
    I've been waiting for this program to come out and I just found out you created a new thread, I was following the old one and had no idea it was out already. That could be why this thread is quiet, maybe a post in the old thread instead of just editing the first post?
     
    nanook likes this.
  6. FMonty

    FMonty Newbie

    Newcomer
    1
    Mar 17, 2019
    United Kingdom
    I’d just like some clarification. The best format to reduce space is NKit.iso while removing update partition.

    Then afterwards you can add additional compression with GCZ through dolphin and then can 7z?
     
  7. Zarper

    Zarper Newbie

    Newcomer
    1
    Sep 13, 2019
    Sweden
    You would likely get way better compression if you skip the GCZ compression and just do the 7zip. GCZ will compress the data making it more random and way harder for 7zip to do its compression
     
  8. nanook
    OP

    nanook Advanced Member

    Newcomer
    5
    Dec 17, 2008
    @FMonty @Zarper Yes for best space saving - nkit.iso - 7zipped

    Best playable is nkit.gcz (made with nkit, not dolphin). Just because NKit will make a slightly different GCZ

    Update Partition removed. Keep the recovery folders so you can always get back to the original image 1:1
     
    Last edited by nanook, Sep 22, 2019
  9. Bent

    Bent GBAtemp Regular

    Member
    6
    Nov 6, 2002
    United States
    Thank you for this program. As I breifly mentioned last night, I have been waiting for this release. Some years ago I dumped all my GC collection, at the time I had a smaller hard drive and needed the space, so I scrubbed and 7zipped them. I've never redumped them but always wished I could now go back to unscrubbed without redumping. From the two images I tested, this everything seems to be great. I have something like 50 images to throw at it when I have time. I'll save them all as nkit once I know they are all reversible.

    Also, I did the opposite with my Wii collection, I dumped all my discs and saved them unmodified since I had the hard drive space, yet it always seemed wasteful to me, saving all that garbage data. Now that it appears to be 100% reversible, I'll be saving these as nkit as well.

    I'll report back how well it worked after I get a chance to do this.
     
    nanook likes this.
  10. SpiffyJUNIOR

    SpiffyJUNIOR GBAtemp Advanced Fan

    Member
    6
    Apr 13, 2016
    United States
    US
    About fucking time.
    Thanks for spending so much time on this and updating us so often during the development process.
     
    nanook likes this.
  11. matt!

    matt! GBAtemp Fan

    Member
    4
    Nov 20, 2009
    Is this open source so support for the format can be added to loaders? is that even desirable?
     
    Hakaisha and fledge68 like this.
  12. Bent

    Bent GBAtemp Regular

    Member
    6
    Nov 6, 2002
    United States
    So I'm still in the process of archiving all my rips, but I have enough of an impression to post my thoughts/questions.

    Wii
    I thought all my Wii rips were 1:1 verified, but I found a corrupt one, potentially just from bit rot on my old hard drive that failed at one point. I had moved everything off and thought it was ok, but it had something like 7 random bytes where it differed from my re-rip. This tool was great for detecting it, though I guess I could have done that from the redump dats too.

    I am interested in archiving my rips, not necessarily playability right now, so I am recovering to nkit and 7zipping them. Initial results are showing about a 70% reduction in space used, which is wonderful.


    Gamecube
    When I run my GC rips through the extraction app to get the recovery files, all my rips say the region is Japan, which is definitely not true, they are all US. Is this a bug? I'm using the GUI app.

    My GC rips were initially scrubbed years ago with an old GC iso management app (I don't remember which one) and then 7zipped. For these, I had a lot more fail to verify, something like 15 or so. Some comparing with a hex editor shows 0x00's where there is data in the re-ripped image. Reading the wiki, it seems like not everything that the scrubbers removed may have been junk that was able to be procedurally generated, some of it might have been other random data. Is this assumption correct?

    It is interesting to see Mario Party 5 in the special cases section in the config file. This game would always fail my scrubbing attempts in the past. I don't remember details since I probably did this 10 years ago, but I think scrubbing it would either corrupt the image or fail to do anything, and it always bothered me.

    I haven't started 7zipping my GC rips yet, but I'm expecting only marginal space savings over my old method of scrubbing/7zipping, since this tool is doing basically the same thing, only safer.

    Thanks a ton for this tool. I've always hoped for a better way to store Wii rips especially, that could still be restored and verified.
     
    nanook likes this.
  13. nanook
    OP

    nanook Advanced Member

    Newcomer
    5
    Dec 17, 2008
    I'll check that the GUI is handling the regions correctly. It's very likely to just be displaying the wrong thing. Your images will be correct. Interesting about the 0x00s. Some scrubbers may have had buggy releases. NKit is proven to be able to be able to generate every byte of junk reliably for GC. I'm interested in which scrubber did this.

    — Posts automatically merged - Please don't double post! —

    Source will be released soon. It's not required though to understand the format. The format is just Wii decrypted and dehashed (RVT-H).
     
  14. matt!

    matt! GBAtemp Fan

    Member
    4
    Nov 20, 2009
    Then can you please help by linking to docs for that (standard?) format?
     
  15. johnsanc

    johnsanc Member

    Newcomer
    5
    Sep 27, 2016
    United States
  16. Bent

    Bent GBAtemp Regular

    Member
    6
    Nov 6, 2002
    United States
    What I did was turn off the "Delete recovery match failures" option, then recover my rips to iso, they said Match Failed. I then split this iso and the re-rip iso into 50MB chunks, and compared these with a comparison tool I use called Beyond Compare. Some of the chunks matched 100%, some of the rips only differed in one chunk, some in almost all of them. I didn't do a thorough comparison, but I did usually see zeros in my restored rip, and data in the re-rip.

    By my best guess, I probably used GC-Tool by Paradox. The release I still have buried on my hard drive says it was released in 2003. Though the dates on all my rips say 2008, I think it may be my original rips I made over the BBA adapter with a tool called Phoenix. I know years later I redid everything with CleanRip, but it looks like I didn't save those rips for whatever reason. So it's possible that I had some bad rips too.

    EDIT: It turns out it is the tool I used to scrub that is corrupting them, using GC-Tool to scrub some of my verified good re-rips gives me the same CRC for the scrubbed rip that is unable to be restored correctly.
     
    Last edited by Bent, Sep 26, 2019
  17. nanook
    OP

    nanook Advanced Member

    Newcomer
    5
    Dec 17, 2008
    Nice work @Bent. I appreciate the thoroughness to prove your theory. :)
     
    Bent likes this.
  18. GerbilSoft

    GerbilSoft GBAtemp Addict

    Member
    13
    Mar 8, 2012
    United States
    The RVT-H unencrypted disc image format is basically the same as Wii format, except partitions are not encrypted and don't have hashes. So it's more like GameCube format with partitions.

    Retail IOSes don't support this format normally, so the cIOS would need to be updated to allow it to work.

    On the plus side, this might actually reduce load times a bit, since the data won't need to be decrypted and hashed on load.
     
    uyjulian, nanook and johnsanc like this.
  19. fledge68

    fledge68 GBAtemp Advanced Maniac

    Member
    10
    Jan 30, 2012
    United States
    so if i understand correctly. someone would have to update cIOS in order for usb loaders to support nkit iso's?
     
  20. johnsanc

    johnsanc Member

    Newcomer
    5
    Sep 27, 2016
    United States
    Correct. Know anyone that would be interested in tackling that? It would be awesome to see that kind of support added to cIOS... that would also mean prototypes could be played on retail hardware.
     
Quick Reply
Draft saved Draft deleted
Loading...