Hardware Wii Ultimate Unscrubber

osupka

Member
OP
Newcomer
Joined
Feb 4, 2016
Messages
8
Trophies
0
Age
37
XP
60
Country
Wii Ultimate Unscrubber

The tool can unscrub a scrubbed Wii ISO to it's original state. That matches Redump.org checksum.

The original discussion topic is at Romshepherd (category "tools")

* Can unscrub ISOs scrubbed with many tools (wiiscrubber, wit, WiiBackUp Manager).
* Can decrypt/encrypt ISOs

Changelog

v0.4 beta
* Added support for WBFS. Can use everywhere the ISO is expected.
* Can use UPDATE partitions from other ISOs, to restore ISOs that don't have UPDATE partition.
* Can use Redump DAT file to verify the unscrubbed ISO. The redump DAT file is required to unscrub an ISO that has no UPDATE partition.
* In some cases can restore a modified header. Works if the game title (name) is not too long.

Ok, how it works. If your ISO/Wbfs has an UPDATE partition, then just drop the iso on the UltimateUnscrubber.exe to unscrub it. If it doesn't have an UPDATE partition then you need 2 things. First is a Redump DAT file (any name, make sure the extension is .dat) Put that file into UltimateUnscrubber_files folder which is located where the UltimateUnscrubber.exe is. I have already included such a DAT file into the archive, but if you want a more recent version, then replace it.
Second you need a collection of extracted UPDATE partitions. Those should be placed at UltimateUnscrubber_files\update_partitions .
Those partitions you can either get from somewhere (download from the internet, here). Or extract them yourself, if you have a big collection of good ISOs. To extract UPDATE partitions, drop a folder with your good ISOs or WBFSs on UltimateUnscrubber.exe.


v0.3.2 changes
* Added support for complex partition tables (the only known game using it "Super Smash Bros. Brawl")
* Added support for unscrubbing ISOs with strange UPDATE padding, if the padding was not removed by scrubbing tools. (more info below)

v0.3.1 changes
* Added support for korean ISOs
* Added support for multi-disc games (the only known game "Dragon Quest X: Mezameshi Itsutsu no Shuzoku Online")

v0.3 changes
* Unscrubber now takes into account "Keep Headers" option in wiiscrubber.

v0.2 changes
* always try to unscrub, even if the ISO does not seem to be scrubbed.
* added "decrypt" switch, to decrypt the ISO instead of unscrub.


H3 CHECKSUM ERROR

If you get "H3 checksum error" in the unscrubber, then you should first verify if the ISO is damaged. Use wit tool: "wit verify game.iso". If the wit tool gives an error too, then the file is damaged, and there is no need to report this error here.


WHY AN ISO CANNOT BE UNSCRUBBED?

1. The often reason of an unsuccessful unscrub is the lack of an UPDATE partition inside the ISO or WBFS. The 0.4 beta version can integrate UPDATE partitions extracted from other ISOs.
2. Another reason could be that the header was modified. Especially it's known that some scene releasers add their name to the game's title inside an ISO. :)
3. The least probable reason is that the ISO is listed in the red list below


ISOS THAT CANNOT NOT BE UNSCRUBBED (unless scrubbed with WBM, more info below)
- Ant Bully (RI3E5D, RI3P5D)
- Rampage: Total Destruction (RPGE5D, RPGP5D)
- Link's Crossbow Training (only korean RZPK01)


WHY THE ISOS LISTED IN THE RED LIST CANNOT BE UNSCRUBBED?

Those ISOs contain some strange data after UPDATE partition. I didn't find what it is, and was not able to generate it. It is identical in USA and EUR discs for both games, but different in the korean one. Which makes me think, this could be data encrypted with an unknown encryption key (which are different for korean and non-korean). The "wii common AES key" for decrypting partitions, didn't work for that data.
All those games are from the first generation games for wii. They were in development when the console was not even released at that time. So it makes me think, they used very old Wii SDK for development. And that SDK (maybe beta version) had some strange behavior (maybe a bug).

Almost all scrubbing tools I tried, remove that data. The only way to preserve it when scrubbing is to use "Wii Backup Manager" with option "enable full scrubbing mode" disabled.

I will try to somehow workaround this situation in 0.4 version, by extracting that data, and preserving it separately. But for now those discs might be impossible to unscrub.


WHAT ARE PARTITIONS?

almost all wii isos contain only 2 partitions: UPDATE and DATA. Some isos contain additional partitions. All partitions except DATA are mostly not needed for playing a game. So WBFSs oftenly contain only DATA partition.
All these isos can be unscrubbed if the partitions were not removed.

the following games contain a CHANNEL partition

- Wii Fit
- Wii Fit Plus
- Rabbids Go Home
- Mario Kart Wii

the following game contains about 12 VC sample partitions. Their exact number depend on the region.

- Super Smash Bros. Brawl

the following japanese games contain a new partition type which actually contains most of game's data files. Without this partition the game does not work. These games are relatively new.

- Dragon Quest X: Mezameshi Itsutsu no Shuzoku Online
- Dragon Quest X: Inishie no Ryuu no Denshou Online


WHY WOULD YOU NEED TO DECRYPT AN ISO

All partitions inside a Wii ISO are AES encrypted. "UltimateUnscrubber decrypt <game.iso>" will decrypt an ISO. Decrypting an ISO makes it more compressible with ZIP or RAR. Up to 500mb regardless of scrubbing.
But decrypted ISOs cannot be played on the console or the Dolphin emulator. They need to be encrypted back.
My tool cannot scrub an ISO. So its "decrypt" feature is more for debugging purposes. If you want a more efficient decryption process, use the Wit tool. "wit convert <game.iso> --enc decrypt" , and to encrypt "wit convert <game.iso> --enc encrypt". My tool encrypts a decrypted iso automatically when unscrubbing.


And for the last word

All ISOs, if they have all their partitions in place and are not listed in the red list above, should be 100% unscrubable. If you found an ISO with an UPDATE partition and it didn't unscrub, please let me know.
 

Attachments

  • UltimateUnscrubber 0.3.2.zip
    14.2 KB · Views: 3,214
  • UltimateUnscrubber 0.4 beta 251016.zip
    126 KB · Views: 2,583
  • wii_iso_docs.7z
    333.7 KB · Views: 1,767
Last edited by osupka,

GerbilSoft

Well-Known Member
Member
Joined
Mar 8, 2012
Messages
2,395
Trophies
2
Age
34
XP
4,249
Country
United States
Interesting that the "junk" data isn't just random garbage; it's based on the game ID and some magic numbers.

Some suggestions for a future release:
  • Support for CISO images in addition to WBFS. Main difference is instead of the drive information block and a block table, there's a 32 KB "block present" table.
  • Support for RVT-R images. These use a different common key, and can be identified by the signature issuer "Root-CA00000002-XS00000006". (Retail discs [and I think Korean also] have the signature issuer "Root-CA00000001-XS00000003".) I don't know if the garbage algorithm is the same on RVT-R, but it probably is.
  • Support for GameCube images in both plain (ISO/GCM) and CISO format. Googling for the magic number (0x1ef29123) brought up a not-working redump.org forum link that seems to imply GameCube uses the same algorithm.
You might also want to consider hosting this tool on GitHub or a similar code hosting site.

@Wiimm would it be possible to update wit to preserve the extra data after the UPDATE partitions for these games?

EDIT: The function a10002710() looks familiar. Searching for its magic number (0x5d588b65) brings up references to C#'s own string hash function (Perl implementation), as well as the random number generator in Paper Mario 64. (The C# one looks closer to this function, though it's still a bit different.)
 
Last edited by GerbilSoft,
  • Like
Reactions: DarthDub

osupka

Member
OP
Newcomer
Joined
Feb 4, 2016
Messages
8
Trophies
0
Age
37
XP
60
Country
Googling for the magic number (0x1ef29123) brought up a not-working redump.org forum link that seems to imply GameCube uses the same algorithm.

That's where I took that function from. I didn't write it myself. The guy on the redump forum said that he decompiled that function from the Nintendo SDK mastering tools. Later he used that code to write his tool NASOS compression for wii isos.

GerbilSoft thank you for your proposals, but I will unlikely implement those. I have very little motivation to continue development.
 
Last edited by Costello,

GerbilSoft

Well-Known Member
Member
Joined
Mar 8, 2012
Messages
2,395
Trophies
2
Age
34
XP
4,249
Country
United States
GerbilSoft thank you for your proposals, but I will unlikely implement those. I have very little motivation to continue development.
I might try it myself then. In particular, I'm looking for CISO, GameCube, and RVT-R support so I can reduce the size of some of my download directories.
 

GerbilSoft

Well-Known Member
Member
Joined
Mar 8, 2012
Messages
2,395
Trophies
2
Age
34
XP
4,249
Country
United States
Update: I started doing some hax on the codebase. https://github.com/GerbilSoft/UltimateUnscrubber

Initial set of changes was fixing command line support. (I usually run command line programs from the command line instead of drag & drop; the command line handling was a bit screwy.)

Currently testing an RVT-R image that I shrunk using a modified version of wit.

EDIT: RVT-R unscrubbing works. :) Only one minor issue: RVT-R discs are 7 MB larger than regular Wii discs. This 7 MB isn't usable, though; it's always 00. (Not even encrypted garbage.)
 
Last edited by GerbilSoft, , Reason: RVT-R works.
  • Like
Reactions: DarthDub

Wiimm

Developer
Member
Joined
Aug 11, 2009
Messages
2,292
Trophies
1
Location
Germany
Website
wiimmfi.de
XP
1,509
Country
Germany
My thoughts and also a question:
If we know, that the garbage data can be recreated by an algorithm, it is very clear, that it can be scrubbed without disadvantages. The experience told us this before. So what is the sense of unscrubbing?
 

GerbilSoft

Well-Known Member
Member
Joined
Mar 8, 2012
Messages
2,395
Trophies
2
Age
34
XP
4,249
Country
United States
My thoughts and also a question:
If we know, that the garbage data can be recreated by an algorithm, it is very clear, that it can be scrubbed without disadvantages. The experience told us this before. So what is the sense of unscrubbing?
It's mostly so you *can* get the original 1:1 rip if you need to for some reason. (mostly historical preservation)
 
  • Like
Reactions: DarthDub

Boink

Member
Newcomer
Joined
May 12, 2004
Messages
5
Trophies
0
XP
276
Country
United States

Boink

Member
Newcomer
Joined
May 12, 2004
Messages
5
Trophies
0
XP
276
Country
United States
I have no plans to implement this unscrub feature, because I don't see any sense to restore garbage.
+1 for ssjkakaroto's suggestion.

Being able to use WIT for everything would be nice, rather than having to rely on other tools like Wii Ultimate Unscrubber.

If the user wants to archive/preserve their original disc collection, they'll want to keep (or have a way to recreate) the virgin/untouched images. Being able to unscrub means they can greatly reduce the amount of disk space used since scrubbed images compress much better.

An unmodified image is the only reliable way to tell you have a good one. You can compare with other people's SHA1 hashes. Different scrubbing software leaves varying amounts of padding/junk data in scrubbed ISOs, so there's no single good/canonical SHA1 for a scrubbed image.

It seems most scrubbers including WIT actually leave some padding data behind. Here are two specific examples. These were the only two ISOs I checked so far, so this issue probably applies to almost all scrubbed-by-WIT ISOs.

GameCube Animal Crossing NTSC-U. Wit scrubbing doesn't remove this padding:
offset 0xFE620, 0x19E0 bytes
offset 0x55708000, 0x2838 bytes

GameCube Zelda Wind Waker NTSC-U. Wit scrubbing doesn't remove this padding:
offset 0x3D1DFC, 0x6204 bytes
offset 0x16848000, 0x1160 bytes
offset 0x3A92257C, 0x5A84 bytes
offset 0x3EC7CD5C, 0x6C4 bytes
 
  • Like
Reactions: aphirst

Wiimm

Developer
Member
Joined
Aug 11, 2009
Messages
2,292
Trophies
1
Location
Germany
Website
wiimmfi.de
XP
1,509
Country
Germany
One question:
If anyone of you buys a new harddrive/sdcard will you then make a copy of the initial unused blocks to be able to restore them?

I have to spent time for Implementing this unscrub feature without any benefit. But there are more interesting things to do (mkw-fun, mkw-ana, Wiimmfi, Wii U, ...) . And so my decision is clear. If ever we found out, that the unscrubbing has any advantages, the we can discuss it again.
 
Last edited by Wiimm,

johnsanc

Well-Known Member
Newcomer
Joined
Sep 27, 2016
Messages
46
Trophies
0
Age
40
XP
812
Country
United States
I can think of a few good use cases to consider:
  1. Converting back to a full 1:1 ISO to be burned and played on a Wii
  2. Some people are interested in the archival of full ISOs (Redump.org). The hash data is useful for people to validate that their own dumps are good. Being able to unscrub back to a known good 1:1 ISO provides piece of mind that people can store their backups in a much more space-friendly way while still being able to restore the full ISO if they ever wanted to.
  3. Ability to create a consistently scrubbed set of backups from various sources. E.g. WBFS files made from various programs can be unscrubbed and rescrubbed with WIT so they are consistent.
  4. IMO, its always preferable to have a way to convert back and forth between formats. Its just a matter of convenience if a single tool could handle this.
I completely understand you probably have more interesting things to tackle, but just sharing my 2 cents.
 
  • Like
Reactions: aphirst

johnsanc

Well-Known Member
Newcomer
Joined
Sep 27, 2016
Messages
46
Trophies
0
Age
40
XP
812
Country
United States
Is there a version of this that supports use by command line? GerbilSoft's modifications to fix this won't compile properly for me.

Edit - Nevermind. Had to use the preview build of Visual Studio and now it works.
 
Last edited by johnsanc,

GerbilSoft

Well-Known Member
Member
Joined
Mar 8, 2012
Messages
2,395
Trophies
2
Age
34
XP
4,249
Country
United States
Is there a version of this that supports use by command line? GerbilSoft's modifications to fix this won't compile properly for me.

Edit - Nevermind. Had to use the preview build of Visual Studio and now it works.
I used VS2008 for editing Ultimate Unscrubber (since that's what I had set up). You shouldn't need to use a preview build...

I may consider rewriting it in C++ and adding a Qt UI, but that takes effort.
 

johnsanc

Well-Known Member
Newcomer
Joined
Sep 27, 2016
Messages
46
Trophies
0
Age
40
XP
812
Country
United States
i used the latest stable build of VS2017 and it gave me CS0307 error around line 76 of CRC.cs and one other I don't recall when trying to build. I guess it was a reported issue with VS2017 that appears to be resolved in the preview.
 

Site & Scene News

Popular threads in this forum

General chit-chat
Help Users
    Sicklyboy @ Sicklyboy: Switching my home routing/firewall solution from pfsense to OPNsense. Boy how I hate networking...