New Wii Unscrubber - SWiiT

Discussion in 'Wii - Hardware, Devices and Utilities' started by nanook, Jul 12, 2018.

  1. nanook
    OP

    nanook Member

    Newcomer
    3
    Dec 17, 2008
    SWiiT is a suite of tools for converting your Wii backup images from various scrubbed, shrunk and USB loader ripped formats back to the original
    disc image.

    It was based on the original Ultimate Unscrubber by Osupka. This version has been fully rewritten and many features added.

    • Unscrub Wii images from scene / scrubbed / wbfs (file) / iso.dec
    • Read the above images from Rar / Zip / 7zip etc
    • Create CRC / SHA1 / MD5 checksums
    • Rename restored images to match Redump dat crc
    • Rename restored images to match a secondary Redump compatible dat crc
    • Insert missing Update (inc rare extra data), Channel and VC partitions
    • Replace Brickblocked Update partitions
    • Auto fixes modified disc headers (where the Data header remains intact)
    • Fixes rare corrupt partition table (caused by WBM)
    • Fixes rare 0 byte file scrubbing (caused by WIT)
    • Fixes truncated images (where the Data partition is intact)
    • Fixes slightly overdumped images too (descrambled)
    • Fixes Data partitions moved before 0xF800000 to save space
    • All regions supported
    • Optimised for performance
    • Reusable library for use in other projects
    • Flexible custom post processing ability
    This is the first release, there are more to come. Enjoy,

    Nanook
     

    Attached Files:

  2. ca032769

    ca032769 GBAtemp Regular

    Member
    3
    Jan 11, 2012
    United States
    Riverside, CA
    Thank you!!
     
    nanook likes this.
  3. FancyNintendoGamer567

    FancyNintendoGamer567 GBAtemp Advanced Fan

    Member
    3
    Feb 13, 2017
    United States
    This looks cool.
     
    nanook likes this.
  4. johnsanc

    johnsanc Member

    Newcomer
    2
    Sep 27, 2016
    United States
    Fantastic work Nanook - you successfully resolved all of the quirks that Wii Ultimate Unscrubber had with not being able to unscrub a handful of games that had weird data after the update partition as well as the dozen or so games that made WUU choke with bogus H3 errors.

    For anyone new to using this tool, read the readme and the FAQ thoroughly.
    This tool is very powerful and flexible. Once you get it setup to your liking it will fix just about anything you throw at it.

    Also, all known good Update, Channel, and VC partitions can be found on the *snip* repository. These are specifically for use with this tool.
    I have also attached a recent redump dat file and a work-in-progress non-redump dat file. With these two dats and the partitions on mariocube just about every game can be restored.

    Enjoy!
     

    Attached Files:

    Last edited by Cyan, Jul 15, 2018 - Reason: sorry, link removed. as nice this repository could be for homebrew, it also contains piracy :(
    Wacko22 and nanook like this.
  5. azoreseuropa

    azoreseuropa GBAtemp Guru

    Member
    8
    Nov 6, 2002
    Portugal
    Proud to be Portuguese but I am in USA.
    Finally. I hate scrubber. Now I can get my backup games back to an original ones. Thanks. *Dig my external hard drive soon*
     
    nanook likes this.
  6. Wacko22

    Wacko22 Member

    Newcomer
    2
    Jun 6, 2015
    United States
    Congrats on release! I took the plunge and scrubbed my 55 1:1 ISOs. Running them through SWAT was a crucial first step; it identified one bad dump, and the partition extracts let me confidently remove the update partitions while scrubbing. Testing SWUU, I ran my new WBFSs through and got every 1:1 back, nice! Looking forward to that GC support. :)
     
    nanook likes this.
  7. FancyNintendoGamer567

    FancyNintendoGamer567 GBAtemp Advanced Fan

    Member
    3
    Feb 13, 2017
    United States
    @nanook @Wacko22 What program did you guys use for scrubbing in order to test this? I wanna try this.
     
  8. johnsanc

    johnsanc Member

    Newcomer
    2
    Sep 27, 2016
    United States
    You can use whatever program you like. This will rebuild any scrubbed iso/wbfs assuming you have the partitions and a dat file with a known good CRC of the full iso.

    That being said, most testing was done with images scrubbed with Wii Backup Manager or Wiimm's WIT toolset with various scrubbing options selected.
     
    Last edited by johnsanc, Jul 16, 2018
    Wacko22 likes this.
  9. FancyNintendoGamer567

    FancyNintendoGamer567 GBAtemp Advanced Fan

    Member
    3
    Feb 13, 2017
    United States
    @johnsanc You kinda confused, me, I understand the partitions but not anything else (especially the DAT file thing).
     
  10. johnsanc

    johnsanc Member

    Newcomer
    2
    Sep 27, 2016
    United States
    Ah ok - The dat files contain the hashes for the full ISOs. Redump.org is the "source of truth" so to speak for these hashes (Notice how CleanRip references redump.org?). Any hashes for games not in the redump.org dat were gathered from various places and verified against several people's collections. This is why i provided both a Redump and Non-Redump dat in my previous post. These two files should contain hash data for almost every known ISO with the exception of some Korean games.

    Now, the magic of this tool (as well osupka's original version) is that it uses an algorithm for combining CRCs. In very simple terms: Data CRC + Update CRC + Junk Data CRC = Full ISO CRC. Because we now know how to reconstruct the Junk Data algorithmically, the only common unknown to a scrubbed ISO is the update partition CRC. There are only a finite number of update partitions (~13gb total) since they are reused across games. This tool does the math and sees which CRC the update partition should have and then adds that partition in assuming you have it saved in your Partitions directory. Nanook's FAQ with the application explains in a little more detail.
     
  11. FancyNintendoGamer567

    FancyNintendoGamer567 GBAtemp Advanced Fan

    Member
    3
    Feb 13, 2017
    United States
    @johnsanc Thanks for the advice, but I got the stuff working. I was able to figure out what does the programs do via their full name in the Readme.
     
  12. nanook
    OP

    nanook Member

    Newcomer
    3
    Dec 17, 2008
    You can restore, wbfs then verify using swuu. This leaves you with a wbfs that has been verified to be restorable. It's just temporary until wbfs output is added.

    Back up your config files
    • Download and unzip wit
    • Edit swuu.exe.config - There are 2 post processing sections. Swap the comment from the bottom section to the top section (<!-- -->).
    • Edit SuperWiiDiscLibrary.dll.config - Set FullVerify to false (Verify will be done by the post processing)
    • Edit ToWbfs.bat inside the PostProcess folder - Set the line SET Wit="wit\wit.exe" to your wit path. Optionally edit call %wit% copy %1 --psel DATA %2 to scrub/wbfs the iso however you like (Only bad for Dragon Quest X backups). Use call %wit% copy %1 --psel -UPDATE %2 for 100% compatibility. This will preserve all extra channels
    Now when you run swuu it will rebuild the iso as normal. Upon completing a successful restore, wit will be ran to wbfs the iso. The restored iso will then be deleted and the wbfs verified with swiv.

    2 log files will be created in the swiv folder - PostProcessFailures.txt and PostProcessSuccess.txt. This allows large batches to be processed with swuu. ALWAYS check the 2 log files to ensure everything went well
     
    johnsanc likes this.
  13. Qriist

    Qriist Member

    Newcomer
    2
    Oct 24, 2010
    United States
    Awesome! This is a magic bullet at a really annoying aspect of a compression framework I've been developing.

    Can the same magic bullet be applied to GameCube games? They also have nonsense data - I just don't know if the data is programmatic.

    Also, where does one find the ReDump Wii dats? They are currently not publicly available on Redump.org.
     
    Last edited by Qriist, Aug 23, 2018
  14. GreyWolf

    GreyWolf GBAtemp Guru

    Member
    8
    Mar 2, 2015
    United States
    I think the GC used the same seed method for the padding. If you're looking for the Wii Redump dat file you can get it from https://www.gc-forever.com/datfile/wii.dat but it's from 2012 so it's likely to be missing some titles that you'll have to get individually from redump.org.
     
  15. Qriist

    Qriist Member

    Newcomer
    2
    Oct 24, 2010
    United States
    It's a start, thanks. ReDump doesn't even have the disc pages for wii accessible. I know they used to exist - and might, to a person with more privileges.
     
  16. GreyWolf

    GreyWolf GBAtemp Guru

    Member
    8
    Mar 2, 2015
    United States
    Hmm... I don't understand why they're such snobs about Wii data. You could also try getting the hashes for missing discs from GameTDB.
     
  17. Qriist

    Qriist Member

    Newcomer
    2
    Oct 24, 2010
    United States
    Thanks for the resource.

    I found a member sharing the ReDump dat here.
     
  18. johnsanc

    johnsanc Member

    Newcomer
    2
    Sep 27, 2016
    United States
    Some redump dat files are only available to people who contribute to the database. If you have any disc dump info you would like to contribute just request an account on their forums.

    I shared very up-to-date dats in this thread above. Also see attached for the latest as of today. I included both redump and "non-redump" (everything else).

    The GameTDB hashes should NOT be trusted. There is a ton of bad info in there right now. Eventually we will update all GameTDB hashes in bulk. Until then, the attached files are the most accurate information available.

    Cheers!
     

    Attached Files:

  19. migles

    migles All my gbatemp friends are now mods, except for me

    Member
    13
    GBAtemp Patron
    migles is a Patron of GBAtemp and is helping us stay independent!

    Our Patreon
    Sep 19, 2013
    China
    Earth-chan
    now make it compatible with gamecube games and i will love ya :D

    btw, where does it get the data thats erased from scrubbing? like that "rare extra data" you mention
     
    Last edited by migles, Aug 25, 2018
    qwertimus likes this.
  20. johnsanc

    johnsanc Member

    Newcomer
    2
    Sep 27, 2016
    United States
    Is scrubbing gamecube games very common?

    There are already programs out there to compress GC games down using the same basic junk data generating algorithm. NASOS (Not Another Shrinker or Scrubber) is what you are looking for. There are 3 main versions of that tool:

    1) NASOS - LedZeppelin68's (redump.org) original, slow and doesn't shrink Korean or dual disc games. Now obsolete
    2) nNASOS - edc's enhanced version, much faster and supports Korean and dual disc games for both GC and Wii. Most commonly used version.
    3) NASOS 2 - LedZeppelin68's updated version which is basically a consistent scrubbing / rebuilding tool. Output is directly playable in Dolphin. Only supports Gamecube games

    Regarding "rare extra data" - Nanook can explain better. However there are a few games that have unknown data after the update partition that almost all tools will scrub out. Rampage and Ant Bully are examples.
     
    Last edited by johnsanc, Aug 25, 2018
Loading...