Homebrew Why are decrypted Wii U game files smaller than encrypted ones?

[SquishyPug]

Hello all,

I've recently been messing around with the source files of various Wii U games,
and while it has been successful so far, there is something that I find strange.
As an example, I obtained the official NUS version of Minecraft: Wii U Edition, and its encrypted files
(.app and .h3 files) total to 591MB in size. However, their unencrypted counterparts (code, content, meta folders)
only equate to 464MB.

I decided to encrypt the latter using NUSPacker to see if their total size would match that of the original, yet they stay the same.
I also did the opposite to the original encrypted files using CDecrypt v4.7, and to my surprise (not) they shrunk to exactly 477MB.
Although the difference in size varies, this seems to be consistent with other games as well.
While this isn't much of an issue itself when it comes to playing it, I find it rather odd that over 100MB can just disappear
after a game is decrypted. If anything, shouldn't it be the very opposite?

Is there something I'm missing or just don't know?
Any help is appreciated.

 

[Deleted User]

Hello all,

I've recently been messing around with the source files of various Wii U games,
and while it has been successful so far, there is something that I find strange.
As an example, I obtained the official NUS version of Minecraft: Wii U Edition, and its encrypted files
(.app and .h3 files) total to 591MB in size. However, their unencrypted counterparts (code, content, meta folders)
only equate to 464MB.

I decided to encrypt the latter using NUSPacker to see if their total size would match that of the original, yet they stay the same.
I also did the opposite to the original encrypted files using CDecrypt v4.7, and to my surprise (not) they shrunk to exactly 477MB.
Although the difference in size varies, this seems to be consistent with other games as well.
While this isn't much of an issue itself when it comes to playing it, I find it rather odd that over 100MB can just disappear
after a game is decrypted. If anything, shouldn't it be the very opposite?

Is there something I'm missing or just don't know?
Any help is appreciated.

Encrypting stuff costs some memory dude. It´s not magic.

So when you decrypt it like that tool does it, the encryptioni s just cut away.

It´s hacker-basics. It´s the same like "cracks" do it on PC-stuff. When they crack things the exe-files also get smaller. Since the encryption is gone there too.

It has to do with overhead. Whenever data is encrypted you need a command to do it. This command is the "Overhead" here. This tells the processor to encrypt the file. There allways has to be command for such functions.

And unencrypted data don´t contain these commands any longer. The headers of these commands are taken out of the file.

This can take from 10-30% less storage depending on your filesystem and data-size you have to encrypt/decrypt.

Or in short for all this technical blabla: Encryption/security costs performance. In this case it costs bandwidth.

Your file gets bigger. You lose bandwidth. Bandwidth is one type of performance, just like Gflops are a measure of performance.

 

[ber71]

Encrypting stuff costs some memory dude. It´s not magic.

So when you decrypt it like that tool does it, the encryptioni s just cut away.

It´s hacker-basics. It´s the same like "cracks" do it on PC-stuff. When they crack things the exe-files also get smaller. Since the encryption is gone there too.

It has to do with overhead. Whenever data is encrypted you need a command to do it. This command is the "Overhead" here. This tells the processor to encrypt the file. There allways has to be command for such functions.

And unencrypted data don´t contain these commands any longer. The headers of these commands are taken out of the file.

This can take from 10-30% less storage depending on your filesystem and data-size you have to encrypt/decrypt.

Or in short for all this technical blabla: Encryption/security costs performance. In this case it costs bandwidth.

Your file gets bigger. You lose bandwidth. Bandwidth is one type of performance, just like Gflops are a measure of performance.

Not true

 

[GerbilSoft]

Not true

Encryption has processing overhead, but not storage overhead. And on modern systems, the processing overhead is minimal, thanks to things like AES-NI. In the case of Wii U, Nintendo's I/O-Processor has a hardware AES engine.

The reason why Wii U game files are smaller when decrypted is because there's two processing methods for contents: "standard" and "H3". Standard stores a single hash within the TMD, so the entire content has to be checked before use, but it takes up no extra space. The H3 method breaks the file up into chunks of 0xFC00 bytes of actual data and 0x400 bytes of hashes. This allows verified random access to 64,512-byte chunks of data at a time.

Hence, out of every 64 KB in an H3-hashed content, 1 KB is hash overhead. That's 1/64 overhead.

Do note that this isn't the only storage overhead. An H3-hashed content may be an archive, which could have padding within the archive. I'm sure Nintendo had a particular reason for including padding, but I wouldn't be able to say why at the moment.

tl;dr encryption does not have storage overhead; encryption with *verification* does, and it's not nearly as much as TheDunningKruger would like you to believe.

Sidenote: I've been looking into using cryptsetup's authenticated encryption modes, but it's still not considered stable. It requires either 4 or 8 bytes of metadata per logical sector, so with 4 KB sectors, 8/4096 = 1.95% overhead. With a 500 GB (477 GiB) SSD, that's around 0.93 GiB.

 

[EelBubble]

Hey, I just registered as I found the same oddity and looked for answers. The replies you got so far make no sense and are all wrong. After digging into it a bit more, I found the actual answer. I don't fully understand the details, but it should clarify it for you.

So first off, `.app` files are only encrypted and don't have compression themselves. In turn they contain different parts of the `FST` "filesystem table" structure, which in turn describes and contains the actual game files.

Now if you decrypt the .app files, without simultaneously extracting the game files from the FST, which is possible with `wiiu-things`, you can see that the FST structures have large portions of empty space with `0x00`. Now why this is the case, I don't know, maybe reduced seek latency for disc access and they just use the same files for the NUS download servers?

Now normally these empty areas should still results in vastly different encrypted output in the final .app files, but for some reason they don't. As far as I understand, .app files are encrypted with the decrypted title key from the title.tik plus the .app index used as an IV. The use of an Initialization Vector is normally meant to protect exactly against these scenarios of the same plaintext resulting in identical ciphertext, but for some reason it doesn't here. This results in the encrypted output for these 0x00 sections being identical. When you compress these encrypted .app files now, these parts effectively get deduplicated by your compression algorithm as they are small enough to fit into the compression dictionary size. Normally compression encrypted data would be nonsensical as encrypted data should be random.

If you recreate .app files yourself from the raw game files using NUSPacker, it creates it own FST likely without this padding.