nsZip is an open source PC tool to lossless compress/decompress NSP files in order to save a lot of storage while all NCA files keep their exact same hash. [prebreak]1[/prebreak]
But it’s more than just a compression. A lot of data like NDV0 partitions or fragment NCA3 files will be removed while compressing and exactly recreated when decompressed which saves even more space especially on updates higher than v65536.
In addition, the NSZ format was designed with emulators in mind so adding NSZ support to Yuzu will be possible in the future and because NSZ contains decrypted NCAs no keys would be needed to only extract game files. As compression algorithm Zstandard is used to multithreaded compress 256 KB chunks while uncompressible chunks are stored uncompressed. That way NSPZ/XCIZ allows random read access. Zstandard has a 43MB/s compression and 7032MB/s decompression speed on an 8 threaded CPU at level 18 while having one of the best compression ratios compared to other compression algorithms.

How compressing works: NSP => extracted NSP => decrypted NCAs => trim fragments => compressing to NSZ => verify correctness => repacking NSPZ file
How decompressing works: NSPZ file => extracted NSPZ => decompress NSZ => untrim fragments => encrypt NCAs => verify correctness => repack as NSP

Check out my GitHub page to report bugs, follow this project, post your suggestions or contribute.
GitHub link: https://github.com/nicoboss/nsZip/releases

For the new homebrew compatible NSZ format please see https://github.com/nicoboss/nsz
NSZ thread: https://gbatemp.net/threads/nsz-homebrew-compatible-nsp-xci-compressor-decompressor.550556/
Until NSZ support is implemented in nsZip I don't see why you want to use it instead of NSZ

Block compressed NSZ is very similar to the beautiful NSPZ format just without all the unnecessarily complexity which made NSPZ unfeasible for other software to implement. I might remove NSPZ/XCIZ support in the future if I see that there's no point in keeping it to not confuse people.

Differences between NSZ and NSPZ:

NSPZ/XCIZ:
- GitHub Project: https://github.com/nicoboss/nsZip
- Always uses Block compression allowing random read access to play compressed games in the future
- Decrypts the whole NCA
- Trims NDV0 fragments to their header and reconstructs them
- Only supported by nsZip and unfortunately doesn't really have a future

NSZ/XCZ:
- GitHub Project: https://github.com/nicoboss/nsz
- Uses solid compression by default. Block compression can be enabled using the -B option. Block compression will be the default for XCZ
- Decrypts all sections while keeping the first 0x4000 bytes encrypted. Puts informations needed to encrypt inside the header.
- Deleted NDV0 fragments as they have no use for end users as they only exist to save CDN bandwidth
- Already widely used. Supported by Tinfoil, SX Installer v3.0.0 and probably a lot of other software in the future


nsZip v2.0.0 preview 2:

• Fixed XCI to XCIZ compression

nsZip v2.0.0 preview 1:

• New UI made using WPF instead of WinForms
  
• Huge performance improvements
  • Directly decrypt the NSP without extracting it first
  • Directly decompressing NSPZ files without extracting them first
  • Huge encrypting speed increase
  • Huge SHA-256 verification speed increase
  • Enhanced compressing multithreading
  • Support for up to 400 CPU Cores
• Improved game compatibility
• Command Line support
• More settings

Version 1.1.0 Changelog:

• Added support for over multiple NCAs splitted NDV0 fragment trimming

Version 1.0.1 Changelog:

• Let the user continue when detecting a yet unimplemented multifile NDV0 fragment. I'll add proper support for that unimplemented multifile NDV0 fragment format in the following days.
• The compression level can now be changed (tradeoff between speed and compression ratio)
• The block size can now be changed (tradeoff between random access time and compression ratio)

Future planes:

• NSZ implementation

FAQ:
Q: It prod.keys not found!
A: Dump them using Lockpick and copy them to %userprofile%/.switch/prod.keys
Q: How much storage does this save?
A: It depends on the game. The compressed size is usually around 40-80% of the original size.
Q: What about XCI?
A: Compression to XCIZ is supported, but XCIZ to XCI recreation is still in development.
Q: The program throws an error or seems to behaves not as intended!
A: Open an issue on my GitHub page where you give me exact steps how to reproduce.
Q: Does this compress installed games?
A: Not now but it's planned for the far future.

Screenshot:

 

[SushiKing]

how much space does it save if it's a big game like 10GB+?

 

[nicoboss]

how much space does it save if it's a big game like 10GB+?

It doesn't matter how big a game is but how game data is stored which again depends mostly on the game engine used to develop that game. As more data are stored in a compressible file format as higher the compression ratio. Most of the time this would lead to around 2 to 5 GB space reduction for a 10 GB game. Oleboy555 tested Mario Kart 8 and got a reduction from 7.43 GB to 5.27 GB which would save 3 GB upscaled to a 10 GB game.

 

[stick267]

Great work. Interesting to hear that running/installing compressed files directly could be possible in the future.

 

[Stash23]

Excuse me if I missed this, but if I compressed it to .NSPZ, could I think use Tinfoil/Goldleaf to install it just fine?

 

[nicoboss]

Excuse me if I missed this, but if I compressed it to .NSPZ, could I think use Tinfoil/Goldleaf to install it just fine?

The NSPZ/XCIZ format is only a few days old so nobody really supports it at the moment. It wouldn't be hard to add NSPZ support to Tinfoil/Goldleaf but such things take time. If the format gets popular, we will probably see homebrew support soon. After improving nsZip I might write a library to make supporting this format even easier. I’m willing to help every developer that wants to support the NSZ format in their software.

 

[blawar]

The NSPZ/XCIZ format is only a few days old so nobody really supports it at the moment. It wouldn't be hard to add NSPZ support to Tinfoil/Goldleaf but such things take time. If the format gets popular, we will probably see homebrew support soon. After improving nsZip I might write a library to make supporting this format even easier. I’m willing to help every developer that wants to support the NSZ format in their software.

You should write it in c++, and only use libraries available in DKP if you want homebrew support. Interesting idea.

 

[mrmole]

First of all, this is a really fantastic app that seems to be working pretty great. It would be absolutely amazing if homebrew devs would adopt your compression formats.

Nonetheless, I have some small suggestions from the user stand point:

• These type of apps just ask for drag and drop interface
• There is no cancel button, so essentially we can't stop it running
• Maybe a very rough estimation on the compression ratio?
• Similarly time estimation would be superb
• Why not make the keys be in the same folder as the app?
• An option to set the default output dictionary would be handy
• This is nitpicking in its finest, but the colours of the buttons are quite saturated and random. Check this tool makes picking colours a breeze: https://color.adobe.com/create/color-wheel/

I hope any of these suggestions help. This is one great app!

 

[Don Jon]

Yes, NSZ is a compression format specialized to compress the normally encrypted and incompressible NCA files (the content of NSP/XCI files). Unlike winzip/winrar NSZ allows random read access and so would be able to be directly played on Yuzu and even the Switch if somebody bothers writing a custom sysmodule for this.

nsz loading?
no need for installing nsps?
possible?

 

[jscjml]

I would be careful with this, depending on the game it could make it very annoying to play. If you tried compressing any GTA games back on the PSP they would become unplayable, since everything is compressed it takes longer to load so the game would constantly stutter.

At the very least keep a backup of the uncompressed game just incase. So you dont have to go and redownload it if something goes wrong.

 

[TheRealNGB]

I would be careful with this, depending on the game it could make it very annoying to play. If you tried compressing any GTA games back on the PSP they would become unplayable, since everything is compressed it takes longer to load so the game would constantly stutter.

At the very least keep a backup of the uncompressed game just incase. So you dont have to go and redownload it if something goes wrong.

Well fist we would have to be able to launch the format, which from what I understand is not possible as of yet, compatibility list with the format would come after, for now its just to save space before installation.

 

[jscjml]

Well fist we would have to be able to launch the format, which from what I understand is not possible as of yet, compatibility list with the format would come after, for now its just to save space before installation.

I know this may be irrelevant cause its not the PSP, but even after reverting the file to normal (after compressing) those said games would still have the same problem. Hopefully not the case, its been like 10 years people probably got better at making these things.

Will it still keep the same size when you install the game? Like if compressed to 2GB, will it stay 2GB when you tinfoil it in?

 

[Kubas_inko]

The only way to really obtain NCA2 files is by restoring improperly deleted factory NCA files on a 1.0.0 console and all of them together are only a few hundred MB. There is barely any person that would even be able to obtain them as only a small amount even owns an untouched 1.0.0 switch. I have my factory NCA2 files dumped but it definitely not worth the efford supporting them with nsZip.

--------------------- MERGED ---------------------------


No this isn't trimming but compressing. When trimming an XCI only useless zeros at the end gets removed.


Yes XCIZ will further reduce the size of a trimmed XCI and TX could implement the format if they like.

nsZip can already compress XCI to XCIZ but not the opposite way so feels free to try it out if you like. Trimmed XCI compression is as good as NSP compression as they booth are file containers containing NCA files.

What are NCA2s? Or why are they special (except being only on 1.0.0 and lower units)?

 

[ZachyCatGames]

What are NCA2s? Or why are they special (except being only on 1.0.0 and lower units)?

An earlier version of the NCA format, there’s also NCA1s and NCA0s which are even earlier versions

Edit: (Also they aren’t even on retail 1.0.0, only pre 1.0.0 firmwares and titles)

 

[TheRealNGB]

I know this may be irrelevant cause its not the PSP, but even after reverting the file to normal (after compressing) those said games would still have the same problem. Hopefully not the case, its been like 10 years people probably got better at making these things.

Will it still keep the same size when you install the game? Like if compressed to 2GB, will it stay 2GB when you tinfoil it in?

I don't think it works like that (you can not even install them through tinfoil yet) it simply a compression/decompression program at this point, I was a bit confused about the functionality as well. Someone please correct me if I'm wrong.

 

[The Real Jdbye]

Nice work, this will be very useful for those uploading nsp's.
Surely a custom sysmodule wouldn't be needed for Switch support though, wouldn't it be much easier and give the same end result to just add on-the-fly decompression to Goldleaf?

 

[nicoboss]

You should write it in c++, and only use libraries available in DKP if you want homebrew support. Interesting idea.

Yes, the library will be in C++ for sure and beside Zstandard (which is BSD licensed) I’ll try to make it as independent from 3rd party libraries as possible.

First of all, this is a really fantastic app that seems to be working pretty great. It would be absolutely amazing if homebrew devs would adopt your compression formats.
Nonetheless, I have some small suggestions from the user stand point:
These type of apps just ask for drag and drop interface
There is no cancel button, so essentially we can't stop it running
Maybe a very rough estimation on the compression ratio?
Similarly time estimation would be superb
Why not make the keys be in the same folder as the app?
An option to set the default output dictionary would be handy
This is nitpicking in its finest, but the colours of the buttons are quite saturated and random. Check this tool makes picking colours a breeze: https://color.adobe.com/create/color-wheel/
I hope any of these suggestions help. This is one great app!

You're right the UI is very bad. I cared way more about the functionality then making a user-friendly well-designed UI for its first version and will completely redo the whole UI in the near future which then will also save all the settings like output directory. With the new UI I will also add a Cancel button but until then feel free to just kill it using Task Manager. Great idea of adding estimated time/ratio while compressing – I’ll add that soon. It's already possible to put keys.txt in the same folder but by default I recommend the standardized key location as most other switch related tool use that one too.

nsz loading?
no need for installing nsps?
possible?

Technically possible but probably something for the far future if TX doesn't implement it in thair XCI loader as no open source system module is available to do something like this.

I would be careful with this, depending on the game it could make it very annoying to play. If you tried compressing any GTA games back on the PSP they would become unplayable, since everything is compressed it takes longer to load so the game would constantly stutter.
At the very least keep a backup of the uncompressed game just incase. So you dont have to go and redownload it if something goes wrong.

NSPZ is completely reversable. The decompressed NCA files have the exact same NCA hash as before compressing them and if "verify while compressing" is enabled there shouldn't be any risk of corrupted NSPZ file and you should always be able to decompress it back to an NSP and never need to redownload anything. Zstandard has a decompression speed way faster than SD-Card read speed and the NSZ format allows random read access so speed won't be a problem but more the lack of somebody willing to write homebrew to even play them on real hardware.

Nice work, this will be very useful for those uploading nsp's.
Surely a custom sysmodule wouldn't be needed for Switch support though, wouldn't it be much easier and give the same end result to just add on-the-fly decompression to Goldleaf?

Making Tinfoil/Godleaf NSPZ compatible won't require any custom system models and will likely happen after I wrote an NSZ C++ library. But after installing them their size will be the same as when installing normal uncompressed NSP files. As soon you want a solution that can directly launch NSZ files (NSZ is the file format of compressed NCA files) a custom system model will be required.

 

[dAVID_]

LZMA Soon.

 

[jakkal]

thats why psp is better

Stop talking trash
Psp better than the switch?

 

[Reploid]

Stop talking trash
Psp better than the switch?

It can run compressed games, can it?

 

[ZachyCatGames]

It can run compressed games, can it?

Being able to run compressed titles doesn't automatically make it better than the Switch lol