Hardware Wii Ultimate Unscrubber

johnsanc · Jul 13, 2017

@GerbilSoft - The program is so simple i don't know if a gui is even needed... but it would be nice to add in a command line switch to allow it proceed without a keypress after completion so this can be used easily in a batch file.

GerbilSoft · Jul 13, 2017

GUI would be useful to show other aspects of the disc image, e.g. title and (for GameCube, which isn't supported yet) banners. Also to handle batch processing more easily.

Either way, I have a few other projects I'm working on right now, so don't expect any major updates anytime soon.

johnsanc · Jul 13, 2017

Ah I didn't know you were thinking of enhancements like that, very cool. For now I just took the quickest and dirtiest workaround possible by just commenting out the console.ReadKey references.

MaDTi · Jul 23, 2017

Hi. Small request:
Add (return) support for command line and disable keypress after "finished"
Don't delete non-redump (unverified) isos after update recovery process

johnsanc · Jul 26, 2017

Wii Ultimate Unscrubber cannot unscrub some games which pass wit verify without issue. These always get an H3 checksum error.

Calvin Tucker's Redneck Racing
Cheggars Party Quiz
Harvest Moon - Tree of Tranquility
Inazuma Elevent Strikers
Formula 1 2009
Fire Emblem Akatsuki no Megami
Kateikyou Hitman Reborn! Dream Hyper Battle! Wii
MadWorld (Japan)
SAFAR'Wii
Suzumiya Haruhi no Heiretsu
Tornado Outbreak
WSC Real 08 World Snooker Championship
WSC Real 09 World Snooker Championship

I will modify this list as I find more.

johnyept · Nov 22, 2017

I just signed just to thank osupka for this great tool! I already unscrubbed a bunch of scrubbed ISOs/WBFS and all returned CRCs from redump or group releases!

Ok, I didn't just signed up for that... I also compiled GerbilSoft's version with VS2008 to comment the readkey lines but for some reason the new EXE fails when trying to unscrub an ISO/WBFS without an update partition by demanding a DAT and an extracted partition, even if they're already placed in the "UltimateUnscrubber_files" and "update_partitions" folders. It works fine with the original EXE and both EXEs work if the ISO/WBFS includes the update partition. Any tips?

EDIT: tried a WBFS with update partition, it correctly unscrubbed to ISO and said "no redump match found", but I checked the ISO's crc and it's listed in the DAT, so I'm guessing it's just not reading anything from the subfolders for some reason that escapes my very limited c++ knowledge.

EDIT2: Problem solved, the program was looking for the DAT in the wrong folder! Just to go line 127 (the one with "*.dat"), and replace "exe_dir" with "additional_dir" (without the quotes).
Here's some newbie tips for anyone interested:
- If you don't want the program to pause at the end, replace all "Console.ReadKey();" with "// Console.ReadKey();", without the quotes.
- If you don't want to delete the newly created file after the message "No redump match found. The ISO cannot be restored", replace "File.Delete(target_path);" with "// File.Delete(target_path);", without the quotes.

johnyept · Dec 30, 2017

Ultimate Unscrubber v0.4 failed to unscrub Smash Bros Brawl (PAL RSBP01). This is what I did:
- Converted original ISO (CRC 6757C5F2) to WBFS+WBF1 with Wii Backup Manager and "keep all partitions" option enabled;
- Converted WBFS+WBF1 back to ISO with WBM (UU unfortunately doesn't recognize WBF1);
- Unscrubbed ISO with UU.

Final ISO has CRC 634CE1D8, there were 49 bytes with different values between 0x40003 and 0x4009E. Merging WBFS+WBF1 with copy /b and then unscrubbing the combined WBFS results in the same CRC.

The v0.3.2 included "support for complex partition tables (the only known game using it "Super Smash Bros. Brawl")". Was this tested with both PAL and NTSC versions?

GreyWolf · Dec 30, 2017

Wii Backup Manager corrupts the partition table for SSBB.

johnyept · Jan 2, 2018

GreyWolf said:
Wii Backup Manager corrupts the partition table for SSBB.

Thanks for the info! I kept the backup in WBFS format and made a PPF patch file with a very old tool (Playstation Patch File) which corrects the wrong bytes of the unscrubbed ISO. All other WBFS I tested either had the same CRC from the dat file or from a release listed in the srrdb site, so the tool works perfectly.

If I knew how to program in C#, the only 2 fixes I would make is adding support for WBF1 files (probably very easy), and include the CRC of the unscrubbed ISO in it's own name, for ex. "Game (CRC).iso". The CRC is specially useful when you're trying to unscrub a game without the update partition, and you know the final CRC but it isn't in the dat file. To work around this, I made a batch file that asks for the CRC, runs UU (a modified version with no pauses nor ISO deletion) with an update partition file, and uses an external program to calculate the unscrubbed ISO's CRC, repeating the process until it finds the correct CRC or runs out of partition files. At least it gets the job done

By the way, does each update partition correspond to a certain time period? For example, can an update partition from a 2007 game appear in a 2011 game? Can that info be checked with perhaps an hex editor? If so, it would make it easier to find a correct update partition instead of trying to unscrub through 74 update partitions

johnsanc · Feb 14, 2018

If its not in a redump.org dat file I wouldn't trust it.

WBF1 is easily solved, just concatenate the files together first. I created a simple batch file to do exactly this.

@Echo OFF
CD /D "%~dp0"
FOR /R %1 %%I IN ("*.wbf1") DO (
COPY /B "%%~dpnI.wbfs" + "%%~dpnI.wbf1" "%%~dpnI-merged.wbfs"
DEL "%%~dpnI.wbfs"
DEL "%%~dpnI.wbf1"
)
PAUSE

Just drag and drop the folder of wbfs + wbf1 files onto the batch file.

Also, it would be great if someone could weigh into why these games get an H3 error with WUU but pass WIT verify without any issues:

Cabela's Survival: Shadows of Katmai (Europe)
Calvin Tucker's Redneck Racing (USA)
Cheggars Party Quiz (Europe)
Harvest Moon - Tree of Tranquility (USA / Europe / Japan)
Inazuma Eleven Strikers (Europe / Japan)
Formula 1 2009 (Europe)
Fire Emblem Akatsuki no Megami (Japan)
Jakers Kart Racing (Europe)
Kateikyou Hitman Reborn! Dream Hyper Battle! Wii (Japan)
MadWorld (Japan)
Minute to Win It (USA)
SAFAR'Wii (Europe)
Suzumiya Haruhi no Heiretsu (Japan)
Tornado Outbreak (USA)
Truth or Lies (USA)
Victorious: Taking the Lead (Europe)
WSC Real 08 World Snooker Championship (Europe)
WSC Real 09 World Snooker Championship (Europe)

johnyept · Apr 3, 2018

I think I found an "unscrubbable" one: Wild_West_Shootout_PAL_Wii-WiiERD contains w-wws.iso with CRC 45FE999A. After converting to WBFS with WiiBackupManager then back to ISO with UU, it results in CRC 01419D0A. Same result with or without update partition in the WBFS. If comparing both ISOs in a HexEditor, they're identical up to 4.699.914.240, then completely different.

EDIT:
Found 2 more that unscrub to a different CRC, Dino_Strike_PAL_Wii-WiiERD and Lego_Rock_Band_PAL_Wii-WiiERD. I'm starting to see a "WiiERD" pattern here

from these 3 games only Lego Rock Band is listed in my redump dat file. WiiERD's ISO has CRC 7654B48C, and after ISO->WBFS->UU it has CRC AA760894, same as the redump dat file.

So, it's not a UU problem, but "WiiERD" dumps that UU is somehow.., fixing? I'll take it!

johnsanc · Apr 4, 2018

Pretty much all WiiERD releases between 11/2009 and 1/2010 are bad. In fact... there are MANY more than the ones you listed that can be fixed to match redump (330+).

nanook · Apr 19, 2018

I'm actively working on wuu at the moment with johnsanc advising and testing.

Much progress has been made. PM me if you wish to test when the time comes for a wider test audience.

johnsanc · Apr 20, 2018

Just to chime in about the mysterious H3 errors referenced in my posts above in case anyone else is following this thread and is curious:

Scrubbing with WBM with full scrubbing mode DISABLED = can be unscrubbed properly
Scrubbing with WBM with full scrubbing mode enabled = cannot be unscrubbed
Scrubbing with WIT using "--psel ALL" = cannot be unscrubbed

Also, when scrubbing to ISO it appears that WBM with full scrubbing mode enabled saving to .iso is exactly the same as scrubbing with WIT with "--psel ALL" to ISO.

Does anyone know the difference between the two scrubbing methods? There is something that gets scrubbed out that prevents WUU from working properly for these titles. Perhaps this is similar to some of the mystery data found in the following games:

Ant Bully (RI3E5D, RI3P5D)
Rampage: Total Destruction (RPGE5D, RPGP5D)
Grim Adventures of Billy and Mandy (RGME5D, RGMP5D)
Link's Crossbow Training (only korean RZPK01)

If any scrubbing experts are watching, feel free to chime in!

nanook · Apr 25, 2018

The issue was to do with zero byte files within the partition filesystem. The app had no support for the filesystem so it took a bit of work to resolve. Further testing is required to ensure the issue is fixed for all valid discs.

aphirst · Jun 29, 2018

Wiimm said:
I have no plans to implement this unscrub feature, because I don't see any sense to restore garbage.

Boink said:
+1 for ssjkakaroto's suggestion.

Being able to use WIT for everything would be nice, rather than having to rely on other tools like Wii Ultimate Unscrubber.

If the user wants to archive/preserve their original disc collection, they'll want to keep (or have a way to recreate) the virgin/untouched images. Being able to unscrub means they can greatly reduce the amount of disk space used since scrubbed images compress much better.

An unmodified image is the only reliable way to tell you have a good one. You can compare with other people's SHA1 hashes. Different scrubbing software leaves varying amounts of padding/junk data in scrubbed ISOs, so there's no single good/canonical SHA1 for a scrubbed image.

Wiimm said:
One question:
If anyone of you buys a new harddrive/sdcard will you then make a copy of the initial unused blocks to be able to restore them?

I have to spent time for Implementing this unscrub feature without any benefit. But there are more interesting things to do (mkw-fun, mkw-ana, Wiimmfi, Wii U, ...) . And so my decision is clear. If ever we found out, that the unscrubbing has any advantages, the we can discuss it again.

I think the usefulness for historical preservation has been woefully understated here. Thanks to the great efforts of those at redump.org there exists a (mostly complete) database of checksums and other metadata for Wii games. Of course, these checksums have been generated against clean, unscrubbed disc images, and it wouldn't make sense to do it any other way.

Integration of unscrubbing into WIT could in principle marry the best of two worlds - efficient storage space of ones own archive of disc images, AND safety in the knowledge that ones own collection can always be re-verified in accordance with the historical-preservation data.

I am aware that Wii game data has its own verifiability measures; the problem is that games which have had the UPDATE partition (and others) removed may still "pass" this test, though of course are guaranteed to fail comparisons to the databases.

If one cares about such historical things, keeping ones own collection as raw images can therefore be an incredible waste of space, since we know that somewhere from 1/4 to 1/3 of that data is literally just reproducible, nigh-incompressible noise - an astonishing amount of needless redundancy.

aphirst · Jun 29, 2018

Funnily enough, today I had some interesting adventures in unscrubbing.

I found an ISO on an old hard disk of a game in my Wii collection, which I must have made a while ago. It seemed I didn't use CleanRip since its checksum didn't match a more recent rip I made of the same game (whose checksums can be verified against redump.org). Since no partitions had been deleted, I decided to see whether the only difference was some sort of scrubbing, so ran WUU on it, yielding a new checksum, but still not the correct one.

Any sane person would have stopped here, figuring the dump must just have been bad. However, it seemed to run fine in dolphin, and more importantly, `wit verify` (on both the original ISO, and its unscrubbed version) reported no issues, suggesting the "actual data" was fine. So I instead persisted.

I then tried using `wit copy` on that original old dump to make a new ISO - I figured this might reformat the content somehow, allowing the unscrubbing to work as it's supposed to. The ISO output from `wit copy` was indeed different, but unscrubbing this yielded the same incorrect checksum as before. I guess all that `wit copy` was doing here was moving the "real data" directly, and generating padding data which was somehow different to before, though still "scrubbed". (I'm sure of that, since the ISO compressed well using standard algorithms; and its compressed xdelta against the original ISO was mere kilobytes)

Finally, I tried something more drastic. I first used `wit copy` to generate not an ISO, but a WBFS file. I then used `wit copy` again on this to generate a totally new ISO, which was, surprisingly enough, different again. THIS TIME, running WUU yielded a result with the desired checksums, matching my newer dump, and as mentioned already, the redump.org database. I get the same behaviour if I use "Wii Backup Manager" to generate the WBFS. And, this output must similarly be "scrubbed", since like the previous comparison, the compressed xdeltas are mere kilobytes).

What I don't understand is how "ISO -> ISO -> Unscrub" can yield a different result to "ISO -> WBFS -> ISO -> Unscrub", where only the latter works. Indeed, I'm also not sure what exactly was used on that original dump, to make it how it is. Perhaps older scrubbing tools did something quirky-but-nondestructive? And, perhaps there's a `wit` option which would make "ISO -> ISO -> Unscrub" give the same output as "ISO -> WBFS -> ISO -> Unscrub", which I'm just ignorant of. (IIRC, @Wiimm said something once about how it uses something timestamp related for its padding, which would mean that it would *not* be reasonable to expect "ISO -> ISO" to match "ISO -> WBFS -> ISO", even if both the Unscrubbed results would be identical)

Lots of time spent, but I think it's all very interesting! What would be even better, though, is if someone could advise me on how I could better "analyse" or "visualise" how exactly these ISOs are differing each time. I can generate xdelta diffs, but I don't really know how to make heads or tails of them.

tswntk · Jun 30, 2018

aphirst said:
What I don't understand is how "ISO -> ISO -> Unscrub" can yield a different result to "ISO -> WBFS -> ISO -> Unscrub", where only the latter works. Indeed, I'm also not sure what exactly was used on that original dump, to make it how it is. Perhaps older scrubbing tools did something quirky-but-nondestructive? And, perhaps there's a `wit` option which would make "ISO -> ISO -> Unscrub" give the same output as "ISO -> WBFS -> ISO -> Unscrub", which I'm just ignorant of. (IIRC, @Wiimm said something once about how it uses something timestamp related for its padding, which would mean that it would *not* be reasonable to expect "ISO -> ISO" to match "ISO -> WBFS -> ISO", even if both the Unscrubbed results would be identical)

Lots of time spent, but I think it's all very interesting! What would be even better, though, is if someone could advise me on how I could better "analyse" or "visualise" how exactly these ISOs are differing each time. I can generate xdelta diffs, but I don't really know how to make heads or tails of them.

that probably reflects how redump.org database was created or the way how cleanrip work. The database is not public any way.
Just so you know, the content of Wii retail game disks are encrypted. Modern Wii dumping tools decrypt the data on the fly, and there are areas on the disk that are not included in the iso (e.g. BCA?}so they are not true "raw images". It all depends on how you want to do something for historical preservation.

Wiimm · Jun 30, 2018

There are 3 ways of scrubbing (read this). And the third way differ by tool and wbfs chunk size. So you get different results. Also way 2 is done or not.

I don't see any benefit for an unscrub feature of wit. If you want to archive an original dump, just do it. The is no need to scrub and then unscrub an image. Scrubbing way 2 make unscrubbing imposible, not sure about way 3. But both are usually used to keep the compact image (WBFS/WDF/CISO) small.

aphirst · Jun 30, 2018

tswntk said:
that probably reflects how redump.org database was created or the way how cleanrip work. The database is not public any way.
Just so you know, the content of Wii retail game disks are encrypted. Modern Wii dumping tools decrypt the data on the fly, and there are areas on the disk that are not included in the iso (e.g. BCA?}so they are not true "raw images". It all depends on how you want to do something for historical preservation.

Wiimm said:
There are 3 ways of scrubbing (read this). And the third way differ by tool and wbfs chunk size. So you get different results. Also way 2 is done or not.

I don't see any benefit for an unscrub feature of wit. If you want to archive an original dump, just do it. The is no need to scrub and then unscrub an image. Scrubbing way 2 make unscrubbing imposible, not sure about way 3. But both are usually used to keep the compact image (WBFS/WDF/CISO) small.

AFAIK the only reason the database isn't public *yet* is because the Wii is still "too new". I'm sure that in another couple of years it will be public.
I'm still inclined to believe that comparisons against the CleanRip results, including the original junk data, is the "right" way to do it - the fact that there are several different ways to "scrub" an ISO (even when not removing any partitions, and even when using the same padding constant) only seems to support this. If there were some "obvious", "natural" way to generate a "full data" WBFS which was both reproducible, *and* didn't blitz any of the info needed to unscrub in principle (taking note of the handful of "unscrubbable" games), then perhaps checksums of these could also be used? However I don't see that ever happening, at least not in a way that convinces the archiving community of its stability.

If after all this, @Wiimm still doesn't see the potential merit of being able to perform after-the-fact verification of one's own scrubbed ISOs, or WBFS-packed games, against the database of confirmed full-disc checksums - and for this to not merely be an awkward isolated Windows binary, but integrated into the cross-platform and otherwise truly excellent ISO toolset he maintains - I don't see any way to ever convince him. I mean, in my world-view, the merit is screamingly self-evident. But obviously not for everyone.

Hardware Wii Ultimate Unscrubber

Well-Known Member

Well-Known Member

Well-Known Member

New Member

Well-Known Member

Member

Member

Well-Known Member

Member

Well-Known Member

Member

Well-Known Member

Well-Known Member

Well-Known Member

Well-Known Member

Well-Known Member

Well-Known Member

Well-Known Member

Developer

Well-Known Member

Similar threads

Popular threads in this forum