nsZip - NSP compressor/decompressor to reduce storage

Reploid · Feb 4, 2019

ZachyCatGames said:
Being able to run compressed titles doesn't automatically make it better than the Switch lol

It makes it better at at least one thing, so this not out of the question, not it?

ZachyCatGames · Feb 4, 2019

Reploid said:
It makes it better at at least one thing, so this not out of the question, not it?

I honestly couldn't care less since all of my games are physical. But whatever

TotalJustice · Feb 5, 2019

Minor suggestion: Delete the contents of the extract folder after decrypting. Saves space during the process, useful for those with limited space on the c drive.

Extracted 13.5gb => decrypted 13.5gb(27gb total) => compressed 12gb (39gb total)

Deleted User · Feb 5, 2019

So, basically this program is like the GC shrinker..? does it rip some stuff of the game...?

nicoboss · Feb 6, 2019

dAVID_ said:
LZMA Soon.

The NSZ format supports every compression algorithm you like by its design. LZMA even was registered as 0x02 since its creation. However, ZStandard has a far higher compression/decompression speed and less RAM usage by nearly the same compression ratio and so was chosen as default and the only one supported by nsZip for now. Another very important reason why I've chosen ZStandard over LZMA was its way lower compression ratio decrease when splitted into 256 KB chunks to allow random read access which is required to directly play compressed games in Yuzu/Switch without decompressing the whole game first. This is because unlike LZMA ZStandard doesn’t use a dictionary by default and even if used ZStandard in in dictionary mode one dictionary could be shared across all blocks while LZMA will save it in every block. But in the end adding LZMA support seems not to be that unlikely especially with NSZ type 0 support in mind which won't allow random read access however LZMA definitely doesn't fit to the currently used NSZ type 1 files used to create NSPZ/XCIZ containers and the adding of it would only slow down homebrew integration for absolutely no benefit. In case you're interested here some single threaded compression algorithm compression here you fine one I made for me and the Yuzu team to find the best fitting compression algorithm for NSZ type 1: https://gist.github.com/nicoboss/064e37753de007ea95a29f7fcf8bb3d0 and https://gist.github.com/nicoboss/62ae92c5dd0d44088323dc8b37859abf

Reploid said:
It makes it better at at least one thing, so this not out of the question, not it?

Well for now it’s definitely is something PSP can better but also something that will likely be possible in the future for the Switch too. It's kind of sad Nintendo doesn't use NCA instead of something similar to NSZ. It would on average saved Nintendo around 40% bandwidth and storage which would a lot of games allow to be on a smaller less expensive game cards and let the user install more titles on the storage he has and all this for only a very small increase of random-access time. To be fair a lot of other gaming consoles downloads and stored their games uncompressed too and Nintendo is sadly well known for not being that innovative with such things. Generally, compression sadly is extremely underestimated - probably due to the huge internet speed increase and Price/TB drop in the past decades.

TotalJustice said:
Minor suggestion: Delete the contents of the extract folder after decrypting. Saves space during the process, useful for those with limited space on the c drive.

Extracted 13.5gb => decrypted 13.5gb(27gb total) => compressed 12gb (39gb total)

I completely agree. It's just stupid to save every step on HDD instead of always using streams but it really made things way easier to develop and debug. I'll get rid of a lot of this when improving performance and for the next version I will delete temporary data immediately if no longer needed instead of cleaning things up after the task.

Elalexyzoner said:
So, basically this program is like the GC shrinker..? does it rip some stuff of the game...?

No, this definitely isn't like any shrinker/trimmer. Trimming just removes zero data used to fill the gamecard's physical storage not used by any game data which has nothing to do with compression. nsZip will further reduce the size even if you already trimmed the input files before by around 40% using compression. As a highly oversimplified explanation how compression works: In real world compressing a book would be to write a list with the 999 most used words and replace all of them with that number to save space. In reality file compression is obviously way more complex but basically it tries to reduce redundancy to save as much storage as possible while being able to recreate the exact same file after decompressing. Because normally compressed files don't allow random read access the NCA files are compressed in 256 KB blocks. In addition, only the header of every fragment files is stored as their content can be exactly recreated. In the end you will have the exact same game after decompressing as before compressing as everything is exactly reversable. All NCA files will have the exact same hash then before too and so are exactly the same. As long you let “verify after compress” enabled nsZip makes sure that after decompressing your game will be exactly the same as before compressing and will error if anything went wrong.

dAVID_ · Feb 6, 2019

nicoboss said:
The NSZ format supports every compression algorithm you like by its design. LZMA even was registered as 0x02 since its creation. However, ZStandard has a far higher compression/decompression speed and less RAM usage by nearly the same compression ratio and so was chosen as default and the only one supported by nsZip for now. Another very important reason why I've chosen ZStandard over LZMA was its way lower compression ratio decrease when splitted into 256 KB chunks to allow random read access which is required to directly play compressed games in Yuzu/Switch without decompressing the whole game first. This is because unlike LZMA ZStandard doesn’t use a dictionary by default and even if used ZStandard in in dictionary mode one dictionary could be shared across all blocks while LZMA will save it in every block. But in the end adding LZMA support seems not to be that unlikely especially with NSZ type 0 support in mind which won't allow random read access however LZMA definitely doesn't fit to the currently used NSZ type 1 files used to create NSPZ/XCIZ containers and the adding of it would only slow down homebrew integration for absolutely no benefit. In case you're interested here some single threaded compression algorithm compression here you fine one I made for me and the Yuzu team to find the best fitting compression algorithm for NSZ type 1: https://gist.github.com/nicoboss/064e37753de007ea95a29f7fcf8bb3d0 and https://gist.github.com/nicoboss/62ae92c5dd0d44088323dc8b37859abf

Well for now it’s definitely is something PSP can better but also something that will likely be possible in the future for the Switch too. It's kind of sad Nintendo doesn't use NCA instead of something similar to NSZ. It would on average saved Nintendo around 40% bandwidth and storage which would a lot of games allow to be on a smaller less expensive game cards and let the user install more titles on the storage he has and all this for only a very small increase of random-access time. To be fair a lot of other gaming consoles downloads and stored their games uncompressed too and Nintendo is sadly well known for not being that innovative with such things. Generally, compression sadly is extremely underestimated - probably due to the huge internet speed increase and Price/TB drop in the past decades.

I completely agree. It's just stupid to save every step on HDD instead of always using streams but it really made things way easier to develop and debug. I'll get rid of a lot of this when improving performance and for the next version I will delete temporary data immediately if no longer needed instead of cleaning things up after the task.

No, this definitely isn't like any shrinker/trimmer. Trimming just removes zero data used to fill the gamecard's physical storage not used by any game data which has nothing to do with compression. nsZip will further reduce the size even if you already trimmed the input files before by around 40% using compression. As a highly oversimplified explanation how compression works: In real world compressing a book would be to write a list with the 999 most used words and replace all of them with that number to save space. In reality file compression is obviously way more complex but basically it tries to reduce redundancy to save as much storage as possible while being able to recreate the exact same file after decompressing. Because normally compressed files don't allow random read access the NCA files are compressed in 256 KB blocks. In addition, only the header of every fragment files is stored as their content can be exactly recreated. In the end you will have the exact same game after decompressing as before compressing as everything is exactly reversable. All NCA files will have the exact same hash then before too and so are exactly the same. As long you let “verify after compress” enabled nsZip makes sure that after decompressing your game will be exactly the same as before compressing and will error if anything went wrong.

Is it really that much faster?

68747470733a2f2f6d636d696c6b2e64652f70726f6a656374732f372d5a69702d7a7374642f646c2f636f6d70722d763132302e706e67

mrmole · Feb 6, 2019

dAVID_ said:
Is it really that much faster?

Please quote your sources, as everybody can make a graph. This particular graph comes from here: https://github.com/mcmilk/7-Zip-zstd and just above it it says: "please note, that the benchmarks are outdated now". Why would you post it, knowing that it is outdated? Simple wikipedia search will tell you that LZMA and and LZMA2 are not exactly the same and there seem to be some arguments that LZMA2 is in fact unsafe for long term archiving: http://lzip.nongnu.org/xz_inadequate.html. Your graph however, does not show the 7zip LZMA2, but Fast LZMA2, which is a 7zip LZMA2, with slightly lower compression ratio that uses parallel buffered radix matchfinder and some optimizations from Zstandard. You should have stated that when presenting the graph, as that is an important information. And lastly the version of ZStandard showed in this graph is from nearly 2 years ago; the newest version is 1.3.8 from about two weeks ago (graph is clearly outdated).

I really know nothing about archiving, but I am all for good research and the transparency of the sources.

dAVID_ · Feb 6, 2019

mrmole said:
Please quote your sources, as everybody can make a graph. This particular graph comes from here: https://github.com/mcmilk/7-Zip-zstd and just above it it says: "please note, that the benchmarks are outdated now". Why would you post it, knowing that it is outdated? Simple wikipedia search will tell you that LZMA and and LZMA2 are not exactly the same and there seem to be some arguments that LZMA2 is in fact unsafe for long term archiving: http://lzip.nongnu.org/xz_inadequate.html. Your graph however, does not show the 7zip LZMA2, but Fast LZMA2, which is a 7zip LZMA2, with slightly lower compression ratio that uses parallel buffered radix matchfinder and some optimizations from Zstandard. You should have stated that when presenting the graph, as that is an important information. And lastly the version of ZStandard showed in this graph is from nearly 2 years ago; the newest version is 1.3.8 from about two weeks ago (graph is clearly outdated).

I really know nothing about archiving, but I am all for good research and the transparency of the sources.

Sorry, I really did just grab one of the first images.

TotalJustice · Feb 7, 2019

Thought you'd like to know, I compressed 517gb of NSP's. After compression the size was 415gb.
Very happy with the 102gb saved!

Nazosan · Feb 7, 2019

I really don't understand why CSO keeps getting brought up. It was always a bit tricky. Some things were fine, but I always found more games would run really badly than those which would not. I ended up with just a few CSOs on my PSP versus almost all the rest being just a standard ISO. This should really go without saying -- especially for more modern games -- but doing on-the-fly decompression requires resources. CPU and RAM (but CPU being the biggie I imagine.) And today games for consoles like these are far less optimized than they were in the PSP's era. The percentage of games that would play well with transparent compression is going to be, if anything, even lower. I mean, it will be great if some simpler games do well with this, but overall I really don't think this should be the big push and such a big deal made about it. As for "the PSP could do it so it was better," if you want to be technical, both are computers. They both can do it. Heck, you can use compression on a NES if you really wanted to. The question isn't whether it is physically possible but A. whether it is viable and. B. worth it. Why should battery life be considerably worse, lots of frameskips, etc etc to save a bit here and there? Like I said, in a few games it would be fine I'm sure, but generally speaking the advantage here is not in any potential to run directly on the system itself. Unfortunately, even without any added compression I'm seeing a lot burn through the battery and struggle to maintain a decent framerate as it is because they are just so poorly optimized. One even was probably largely responsible for the stock cooling fan in my system burning out from ramping up to maximum speed continuously for hours at a time (yeah, I learned my lesson from that and use external cooling now.) In the end that's an awful lot of work for probably a lot more problems than the benefits.

Which comes to the real point. The true value of this is absolutely the compression itself being better. Less storage space being used for NSPs is huge. They add up and fill up backup media quite easily -- especially if you have other stuff backed up. (Heck, even just staying on-system, a full NAND backup is quite large, nevermind considering any other things.) It could also definitely help with installation. It's generally easiest by far to put the NSPs on the memory card and install directly. But this comes at the cost of needing double the space free to start. I can't think of a single reason this wouldn't actually be very tinfoil/goldleaf/etc friendly -- all they have to do is integrate block decompression on such a file source and the rest is likely just about the same from there (does it even need to reencrypt? If it did surely even a zero key or something would be sufficient wouldn't it? Nothing complicated needed before the system takes over and does its own thing.) I honestly think that as soon as they become aware of this they will 100% support the idea of it. The size difference on an individual file isn't huge, but it will help -- particularly when one does a bunch at once. Thanks to their encryption it has been impossible to use normal compression techniques, so this is going to help a lot. I think long-term the best thing to do is implement multiple compression techniques and have some sort of mechanism (header or whatever) that specifies which is used (maybe what it already does?) I suspect that the best tradeoff lies in that point where installation doesn't take significantly longer but trading just a bit of install time for overall best compression ratio. I just don't think a "CSO" like thing is all that viable, but if it supports a number of different compression formats and levels the user will be able to try that if they really want to.

nicoboss, I absolutely want to thank you for your work on this. You have a good idea and it's going to help the community.

PS. As for "Nintendo should do something like this to save themselves bandwidth" they still haven't even integrated basic system themes beyond the default not even very good two or given us any form of organizing stuff despite how much of the Switch's OS being based on the 3DS'. They can't even catch up to where they were before. I don't know what the heck is going on there honestly, but clearly optimization and features aren't a focus. I guess they've switched 100% to marketing and game development cycles or something with a bit of anti-piracy measures on the side. Whatever they're doing, it isn't actually working on actually achieving anything useful or nice so much as just maintain status quo. Oh. And "general system stability improvements to enhance the user's experience" of course (I wasn't aware that previous firmwares were so massively unstable until they told me. Phew. Thank goodness for the warning!)

sageharpuiahx · Feb 8, 2019

It struggles to compress anything over a gig, so compressing my triple A titles is out of the question for now, but it's definitely a start

corim123 · Feb 12, 2019

Reploid said:
It makes it better at at least one thing, so this not out of the question, not it?

No. Can you go be dumbass somewhere else? In the context of the discussion it doesn’t even make sense and is hardly an apples to apples comparison you fucking twat.

nicoboss · Feb 14, 2019

nsZip version 1.1.0 released: https://github.com/nicoboss/nsZip/releases
I finally implemented over multiple NCA’s splitted NDV0 partitions trimming/untrimming. It took much longer than expected as things were quite complex. This should fix all the update NCAs that throw an error in v1.0.0 and got stored uncompressed after showing a warning message box in v1.0.1. nsZip is written to always throw an error if something went wrong to ensure decompression will give the exact same NCAs as before compressing if verification is on, so please report all errors as I can only test games I own. However, 2 weeks after release without hearing of any other unsupported NSPs it seems as if all are supported by now.
Now that this is done I can finally full focus on your suggestions. In addition, I’ll post a writeup how NDV0 trimming/untrimming works in the next few days.

orangpelupa · Feb 15, 2019

nicoboss said:
How compressing works: NSP => extracted NSP => decrypted NCAs => trim fragments => compressing to NSZ => verify correctness => repacking NSPZ file

Is it viable to add checkboxes for the decrypted NCA to stay on hard-drive?

Arnold101 · Feb 15, 2019

the program crashed and left all the temp data on my c drive, now is full, what folder it is? i want to delete temp files.

TotalJustice · Feb 15, 2019

Arnold101 said:
the program crashed and left all the temp data on my c drive, now is full, what folder it is? i want to delete temp files.

Can’t remember off the top of my head but the folder is called 2.0 which is located in the users/*username*/app folder.

Idk if the latest version changed the folder name so if it’s not there, search for nsZip.exe.

Once you find that folder, you can copy it to another drive with more space and run the exe from there. I had to do that as I didn’t have enough space on the c drive either.

Arnold101 · Feb 15, 2019

TotalJustice said:
Can’t remember off the top of my head but the folder is called 2.0 which is located in the users/*username*/app folder.

Idk if the latest version changed the folder name so if it’s not there, search for nsZip.exe.

Once you find that folder, you can copy it to another drive with more space and run the exe from there. I had to do that as I didn’t have enough space on the c drive either.

not found.

TotalJustice · Feb 15, 2019

Arnold101 said:
not found.

Users/*username*/appdata/local/apps/

2.0 will be there

Arnold101 · Feb 15, 2019

TotalJustice said:
Users/*username*/appdata/local/apps/

2.0 will be there

thx, now how i change it? if i can

TotalJustice · Feb 15, 2019

Arnold101 said:
thx, now how i change it? if i can

Change what? Remove the temp files or change its directory?

If it’s temp files, locate the folders compressed, decrypted and extracted (it’s tge folder that has the nszip.exe) delete all files in those folders.

To move it to another drive (like D: drive), just move it and make a shortcut to the nszip.exe.

Well-Known Member

Well-Known Member

Well-Known Member

Deleted User

Guest

Well-Known Member

Well-Known Member

Active Member

Well-Known Member

Well-Known Member

Well-Known Member

Member

Well-Known Member

Well-Known Member

Well-Known Member

Well-Known Member

Well-Known Member

Well-Known Member

Well-Known Member

Well-Known Member

Well-Known Member

Similar threads

Popular threads in this forum