Wii Ultimate Unscrubber

Discussion in 'Wii - Hardware, Devices and Utilities' started by osupka, Dec 5, 2016.

  1. osupka
    OP

    osupka Newbie

    Newcomer
    8
    7
    Feb 4, 2016
    Wii Ultimate Unscrubber

    The tool can unscrub a scrubbed Wii ISO to it's original state. That matches Redump.org checksum.

    The original discussion topic is at Romshepherd (category "tools")

    * Can unscrub ISOs scrubbed with many tools (wiiscrubber, wit, WiiBackUp Manager).
    * Can decrypt/encrypt ISOs

    Changelog

    v0.4 beta
    * Added support for WBFS. Can use everywhere the ISO is expected.
    * Can use UPDATE partitions from other ISOs, to restore ISOs that don't have UPDATE partition.
    * Can use Redump DAT file to verify the unscrubbed ISO. The redump DAT file is required to unscrub an ISO that has no UPDATE partition.
    * In some cases can restore a modified header. Works if the game title (name) is not too long.

    Ok, how it works. If your ISO/Wbfs has an UPDATE partition, then just drop the iso on the UltimateUnscrubber.exe to unscrub it. If it doesn't have an UPDATE partition then you need 2 things. First is a Redump DAT file (any name, make sure the extension is .dat) Put that file into UltimateUnscrubber_files folder which is located where the UltimateUnscrubber.exe is. I have already included such a DAT file into the archive, but if you want a more recent version, then replace it.
    Second you need a collection of extracted UPDATE partitions. Those should be placed at UltimateUnscrubber_files\update_partitions .
    Those partitions you can either get from somewhere (download from the internet, here). Or extract them yourself, if you have a big collection of good ISOs. To extract UPDATE partitions, drop a folder with your good ISOs or WBFSs on UltimateUnscrubber.exe.


    v0.3.2 changes
    * Added support for complex partition tables (the only known game using it "Super Smash Bros. Brawl")
    * Added support for unscrubbing ISOs with strange UPDATE padding, if the padding was not removed by scrubbing tools. (more info below)

    v0.3.1 changes
    * Added support for korean ISOs
    * Added support for multi-disc games (the only known game "Dragon Quest X: Mezameshi Itsutsu no Shuzoku Online")

    v0.3 changes
    * Unscrubber now takes into account "Keep Headers" option in wiiscrubber.

    v0.2 changes
    * always try to unscrub, even if the ISO does not seem to be scrubbed.
    * added "decrypt" switch, to decrypt the ISO instead of unscrub.


    H3 CHECKSUM ERROR

    If you get "H3 checksum error" in the unscrubber, then you should first verify if the ISO is damaged. Use wit tool: "wit verify game.iso". If the wit tool gives an error too, then the file is damaged, and there is no need to report this error here.


    WHY AN ISO CANNOT BE UNSCRUBBED?

    1. The often reason of an unsuccessful unscrub is the lack of an UPDATE partition inside the ISO or WBFS. The 0.4 beta version can integrate UPDATE partitions extracted from other ISOs.
    2. Another reason could be that the header was modified. Especially it's known that some scene releasers add their name to the game's title inside an ISO. :)
    3. The least probable reason is that the ISO is listed in the red list below


    ISOS THAT CANNOT NOT BE UNSCRUBBED (unless scrubbed with WBM, more info below)
    - Ant Bully (RI3E5D, RI3P5D)
    - Rampage: Total Destruction (RPGE5D, RPGP5D)
    - Link's Crossbow Training (only korean RZPK01)


    WHY THE ISOS LISTED IN THE RED LIST CANNOT BE UNSCRUBBED?

    Those ISOs contain some strange data after UPDATE partition. I didn't find what it is, and was not able to generate it. It is identical in USA and EUR discs for both games, but different in the korean one. Which makes me think, this could be data encrypted with an unknown encryption key (which are different for korean and non-korean). The "wii common AES key" for decrypting partitions, didn't work for that data.
    All those games are from the first generation games for wii. They were in development when the console was not even released at that time. So it makes me think, they used very old Wii SDK for development. And that SDK (maybe beta version) had some strange behavior (maybe a bug).

    Almost all scrubbing tools I tried, remove that data. The only way to preserve it when scrubbing is to use "Wii Backup Manager" with option "enable full scrubbing mode" disabled.

    I will try to somehow workaround this situation in 0.4 version, by extracting that data, and preserving it separately. But for now those discs might be impossible to unscrub.


    WHAT ARE PARTITIONS?

    almost all wii isos contain only 2 partitions: UPDATE and DATA. Some isos contain additional partitions. All partitions except DATA are mostly not needed for playing a game. So WBFSs oftenly contain only DATA partition.
    All these isos can be unscrubbed if the partitions were not removed.

    the following games contain a CHANNEL partition

    - Wii Fit
    - Wii Fit Plus
    - Rabbids Go Home
    - Mario Kart Wii

    the following game contains about 12 VC sample partitions. Their exact number depend on the region.

    - Super Smash Bros. Brawl

    the following japanese games contain a new partition type which actually contains most of game's data files. Without this partition the game does not work. These games are relatively new.

    - Dragon Quest X: Mezameshi Itsutsu no Shuzoku Online
    - Dragon Quest X: Inishie no Ryuu no Denshou Online


    WHY WOULD YOU NEED TO DECRYPT AN ISO

    All partitions inside a Wii ISO are AES encrypted. "UltimateUnscrubber decrypt <game.iso>" will decrypt an ISO. Decrypting an ISO makes it more compressible with ZIP or RAR. Up to 500mb regardless of scrubbing.
    But decrypted ISOs cannot be played on the console or the Dolphin emulator. They need to be encrypted back.
    My tool cannot scrub an ISO. So its "decrypt" feature is more for debugging purposes. If you want a more efficient decryption process, use the Wit tool. "wit convert <game.iso> --enc decrypt" , and to encrypt "wit convert <game.iso> --enc encrypt". My tool encrypts a decrypted iso automatically when unscrubbing.


    And for the last word

    All ISOs, if they have all their partitions in place and are not listed in the red list above, should be 100% unscrubable. If you found an ISO with an UPDATE partition and it didn't unscrub, please let me know.
     

    Attached Files:

    Last edited by osupka, Dec 8, 2016
    johnsanc, sat0shii, Zonta85 and 4 others like this.
  2. GerbilSoft

    GerbilSoft GBAtemp Addict

    Member
    2,103
    2,341
    Mar 8, 2012
    United States
    Interesting that the "junk" data isn't just random garbage; it's based on the game ID and some magic numbers.

    Some suggestions for a future release:
    • Support for CISO images in addition to WBFS. Main difference is instead of the drive information block and a block table, there's a 32 KB "block present" table.
    • Support for RVT-R images. These use a different common key, and can be identified by the signature issuer "Root-CA00000002-XS00000006". (Retail discs [and I think Korean also] have the signature issuer "Root-CA00000001-XS00000003".) I don't know if the garbage algorithm is the same on RVT-R, but it probably is.
    • Support for GameCube images in both plain (ISO/GCM) and CISO format. Googling for the magic number (0x1ef29123) brought up a not-working redump.org forum link that seems to imply GameCube uses the same algorithm.
    You might also want to consider hosting this tool on GitHub or a similar code hosting site.

    @Wiimm would it be possible to update wit to preserve the extra data after the UPDATE partitions for these games?

    EDIT: The function a10002710() looks familiar. Searching for its magic number (0x5d588b65) brings up references to C#'s own string hash function (Perl implementation), as well as the random number generator in Paper Mario 64. (The C# one looks closer to this function, though it's still a bit different.)
     
    Last edited by GerbilSoft, Dec 6, 2016
  3. osupka
    OP

    osupka Newbie

    Newcomer
    8
    7
    Feb 4, 2016
    That's where I took that function from. I didn't write it myself. The guy on the redump forum said that he decompiled that function from the Nintendo SDK mastering tools. Later he used that code to write his tool NASOS compression for wii isos.

    GerbilSoft thank you for your proposals, but I will unlikely implement those. I have very little motivation to continue development.
     
    Last edited by Costello, Feb 24, 2017
  4. gamecaptor

    gamecaptor R.I.P. Densetsu. Legends never die!

    Member
    GBAtemp Patron
    gamecaptor is a Patron of GBAtemp and is helping us stay independent!

    Our Patreon
    550
    308
    Jan 22, 2004
    United States
    Hyrule most of the time
    This is actually something I was just recently looking for. Very cool. Now hopefully Redump comes back up so I can submit some new stuff to help your project along :glare:
     
  5. Larsenv

    Larsenv RiiConnect24 Developer

    Member
    548
    465
    Sep 28, 2013
    United States
    Texas
    Nice.

    I guess it's good so Redump can have more checksums.
     
  6. GerbilSoft

    GerbilSoft GBAtemp Addict

    Member
    2,103
    2,341
    Mar 8, 2012
    United States
    I might try it myself then. In particular, I'm looking for CISO, GameCube, and RVT-R support so I can reduce the size of some of my download directories.
     
  7. GerbilSoft

    GerbilSoft GBAtemp Addict

    Member
    2,103
    2,341
    Mar 8, 2012
    United States
    Update: I started doing some hax on the codebase. https://github.com/GerbilSoft/UltimateUnscrubber

    Initial set of changes was fixing command line support. (I usually run command line programs from the command line instead of drag & drop; the command line handling was a bit screwy.)

    Currently testing an RVT-R image that I shrunk using a modified version of wit.

    EDIT: RVT-R unscrubbing works. :) Only one minor issue: RVT-R discs are 7 MB larger than regular Wii discs. This 7 MB isn't usable, though; it's always 00. (Not even encrypted garbage.)
     
    Last edited by GerbilSoft, Dec 7, 2016 - Reason: RVT-R works.
  8. Wiimm

    Wiimm Developer

    Member
    2,170
    379
    Aug 11, 2009
    Gambia, The
    Germany
    My thoughts and also a question:
    If we know, that the garbage data can be recreated by an algorithm, it is very clear, that it can be scrubbed without disadvantages. The experience told us this before. So what is the sense of unscrubbing?
     
  9. GerbilSoft

    GerbilSoft GBAtemp Addict

    Member
    2,103
    2,341
    Mar 8, 2012
    United States
    It's mostly so you *can* get the original 1:1 rip if you need to for some reason. (mostly historical preservation)
     
  10. ssjkakaroto

    ssjkakaroto Advanced Member

    Newcomer
    86
    0
    Dec 14, 2005
    @Wiimm It would be nice to be able to convert from a scrubbed WDF/WIA image to an unscrubbed ISO using wit and be able to match a known good dump.
     
  11. Wiimm

    Wiimm Developer

    Member
    2,170
    379
    Aug 11, 2009
    Gambia, The
    Germany
    I have no plans to implement this unscrub feature, because I don't see any sense to restore garbage.
     
  12. GreyWolf

    GreyWolf GBAtemp Psycho!

    Member
    4,156
    801
    Mar 2, 2015
    United States
    Interesting... 0x5d588b65 is a standard seed for the Mersenne Twister psudo-random number generator.
     
  13. Boink

    Boink Newbie

    Newcomer
    5
    0
    May 12, 2004
    United States
  14. Boink

    Boink Newbie

    Newcomer
    5
    0
    May 12, 2004
    United States
    +1 for ssjkakaroto's suggestion.

    Being able to use WIT for everything would be nice, rather than having to rely on other tools like Wii Ultimate Unscrubber.

    If the user wants to archive/preserve their original disc collection, they'll want to keep (or have a way to recreate) the virgin/untouched images. Being able to unscrub means they can greatly reduce the amount of disk space used since scrubbed images compress much better.

    An unmodified image is the only reliable way to tell you have a good one. You can compare with other people's SHA1 hashes. Different scrubbing software leaves varying amounts of padding/junk data in scrubbed ISOs, so there's no single good/canonical SHA1 for a scrubbed image.

    It seems most scrubbers including WIT actually leave some padding data behind. Here are two specific examples. These were the only two ISOs I checked so far, so this issue probably applies to almost all scrubbed-by-WIT ISOs.

    GameCube Animal Crossing NTSC-U. Wit scrubbing doesn't remove this padding:
    offset 0xFE620, 0x19E0 bytes
    offset 0x55708000, 0x2838 bytes

    GameCube Zelda Wind Waker NTSC-U. Wit scrubbing doesn't remove this padding:
    offset 0x3D1DFC, 0x6204 bytes
    offset 0x16848000, 0x1160 bytes
    offset 0x3A92257C, 0x5A84 bytes
    offset 0x3EC7CD5C, 0x6C4 bytes
     
  15. GreyWolf

    GreyWolf GBAtemp Psycho!

    Member
    4,156
    801
    Mar 2, 2015
    United States
    I'm not too sure about this as it was reverse-engineered directly from the SDK. :/
     
  16. Wiimm

    Wiimm Developer

    Member
    2,170
    379
    Aug 11, 2009
    Gambia, The
    Germany
    One question:
    If anyone of you buys a new harddrive/sdcard will you then make a copy of the initial unused blocks to be able to restore them?

    I have to spent time for Implementing this unscrub feature without any benefit. But there are more interesting things to do (mkw-fun, mkw-ana, Wiimmfi, Wii U, ...) . And so my decision is clear. If ever we found out, that the unscrubbing has any advantages, the we can discuss it again.
     
    Last edited by Wiimm, Jan 17, 2017
  17. johnsanc

    johnsanc Member

    Newcomer
    14
    0
    Sep 27, 2016
    United States
    I can think of a few good use cases to consider:
    1. Converting back to a full 1:1 ISO to be burned and played on a Wii
    2. Some people are interested in the archival of full ISOs (Redump.org). The hash data is useful for people to validate that their own dumps are good. Being able to unscrub back to a known good 1:1 ISO provides piece of mind that people can store their backups in a much more space-friendly way while still being able to restore the full ISO if they ever wanted to.
    3. Ability to create a consistently scrubbed set of backups from various sources. E.g. WBFS files made from various programs can be unscrubbed and rescrubbed with WIT so they are consistent.
    4. IMO, its always preferable to have a way to convert back and forth between formats. Its just a matter of convenience if a single tool could handle this.
    I completely understand you probably have more interesting things to tackle, but just sharing my 2 cents.
     
  18. johnsanc

    johnsanc Member

    Newcomer
    14
    0
    Sep 27, 2016
    United States
    Is there a version of this that supports use by command line? GerbilSoft's modifications to fix this won't compile properly for me.

    Edit - Nevermind. Had to use the preview build of Visual Studio and now it works.
     
    Last edited by johnsanc, Jul 12, 2017
  19. GerbilSoft

    GerbilSoft GBAtemp Addict

    Member
    2,103
    2,341
    Mar 8, 2012
    United States
    I used VS2008 for editing Ultimate Unscrubber (since that's what I had set up). You shouldn't need to use a preview build...

    I may consider rewriting it in C++ and adding a Qt UI, but that takes effort.
     
  20. johnsanc

    johnsanc Member

    Newcomer
    14
    0
    Sep 27, 2016
    United States
    i used the latest stable build of VS2017 and it gave me CS0307 error around line 76 of CRC.cs and one other I don't recall when trying to build. I guess it was a reported issue with VS2017 that appears to be resolved in the preview.