nsZip is an open source PC tool to lossless compress/decompress NSP files in order to save a lot of storage while all NCA files keep their exact same hash. [prebreak]1[/prebreak]
But it’s more than just a compression. A lot of data like NDV0 partitions or fragment NCA3 files will be removed while compressing and exactly recreated when decompressed which saves even more space especially on updates higher than v65536.
In addition, the NSZ format was designed with emulators in mind so adding NSZ support to Yuzu will be possible in the future and because NSZ contains decrypted NCAs no keys would be needed to only extract game files. As compression algorithm Zstandard is used to multithreaded compress 256 KB chunks while uncompressible chunks are stored uncompressed. That way NSPZ/XCIZ allows random read access. Zstandard has a 43MB/s compression and 7032MB/s decompression speed on an 8 threaded CPU at level 18 while having one of the best compression ratios compared to other compression algorithms.

How compressing works: NSP => extracted NSP => decrypted NCAs => trim fragments => compressing to NSZ => verify correctness => repacking NSPZ file
How decompressing works: NSPZ file => extracted NSPZ => decompress NSZ => untrim fragments => encrypt NCAs => verify correctness => repack as NSP

Check out my GitHub page to report bugs, follow this project, post your suggestions or contribute.
GitHub link: https://github.com/nicoboss/nsZip/releases

For the new homebrew compatible NSZ format please see https://github.com/nicoboss/nsz
NSZ thread: https://gbatemp.net/threads/nsz-homebrew-compatible-nsp-xci-compressor-decompressor.550556/
Until NSZ support is implemented in nsZip I don't see why you want to use it instead of NSZ

Block compressed NSZ is very similar to the beautiful NSPZ format just without all the unnecessarily complexity which made NSPZ unfeasible for other software to implement. I might remove NSPZ/XCIZ support in the future if I see that there's no point in keeping it to not confuse people.

Differences between NSZ and NSPZ:

NSPZ/XCIZ:
- GitHub Project: https://github.com/nicoboss/nsZip
- Always uses Block compression allowing random read access to play compressed games in the future
- Decrypts the whole NCA
- Trims NDV0 fragments to their header and reconstructs them
- Only supported by nsZip and unfortunately doesn't really have a future

NSZ/XCZ:
- GitHub Project: https://github.com/nicoboss/nsz
- Uses solid compression by default. Block compression can be enabled using the -B option. Block compression will be the default for XCZ
- Decrypts all sections while keeping the first 0x4000 bytes encrypted. Puts informations needed to encrypt inside the header.
- Deleted NDV0 fragments as they have no use for end users as they only exist to save CDN bandwidth
- Already widely used. Supported by Tinfoil, SX Installer v3.0.0 and probably a lot of other software in the future


nsZip v2.0.0 preview 2:

• Fixed XCI to XCIZ compression

nsZip v2.0.0 preview 1:

• New UI made using WPF instead of WinForms
  
• Huge performance improvements
  • Directly decrypt the NSP without extracting it first
  • Directly decompressing NSPZ files without extracting them first
  • Huge encrypting speed increase
  • Huge SHA-256 verification speed increase
  • Enhanced compressing multithreading
  • Support for up to 400 CPU Cores
• Improved game compatibility
• Command Line support
• More settings

Version 1.1.0 Changelog:

• Added support for over multiple NCAs splitted NDV0 fragment trimming

Version 1.0.1 Changelog:

• Let the user continue when detecting a yet unimplemented multifile NDV0 fragment. I'll add proper support for that unimplemented multifile NDV0 fragment format in the following days.
• The compression level can now be changed (tradeoff between speed and compression ratio)
• The block size can now be changed (tradeoff between random access time and compression ratio)

Future planes:

• NSZ implementation

FAQ:
Q: It prod.keys not found!
A: Dump them using Lockpick and copy them to %userprofile%/.switch/prod.keys
Q: How much storage does this save?
A: It depends on the game. The compressed size is usually around 40-80% of the original size.
Q: What about XCI?
A: Compression to XCIZ is supported, but XCIZ to XCI recreation is still in development.
Q: The program throws an error or seems to behaves not as intended!
A: Open an issue on my GitHub page where you give me exact steps how to reproduce.
Q: Does this compress installed games?
A: Not now but it's planned for the far future.

Screenshot:

 

[nicoboss]

Is this Windows only? Any plans on making this available for Mac?

nsZip itself is currently windows only but the NSZ python script from https://github.com/nicoboss/nsz should be platform independent. Because NSZ will probably be the successor on NSPZ there's no reason not to use the python script.

 

[skinnyBIGGS]

Great work guys . I.ll wait until its ironed out abit more a d works with goldleaf i dont like tinfoil

 

[Mat37]

@nicoboss Will the new version have retro-compatibility with the NSPZ format ?

 

[Deathscreton]

I can't seem to get nsZIP to work anymore. It just opens up CMD, then closes immediately. I'm planning on switching to NCZ, but all of my old NSPs are compressed in the old format.

 

[awankelam]

I got JIT error message just right after I click RUN.

It says, "couldn't load file or assembly 'netstandard. Version = 2.0.0.0. Culture=neutral" Bla 3x

What is wrong? What should I do?

 

[awankelam]

I got JIT error message just right after I click RUN.

It says, "couldn't load file or assembly 'netstandard. Version = 2.0.0.0. Culture=neutral" Bla 3x

What is wrong? What should I do?

 

[awankelam]

I got JIT error message just right after I click RUN.

It says, "couldn't load file or assembly 'netstandard. Version = 2.0.0.0. Culture=neutral" Bla 3x

What is wrong? What should I do?

--------------------- MERGED ---------------------------

I got JIT error message just right after I click RUN.

It says, "couldn't load file or assembly 'netstandard. Version = 2.0.0.0. Culture=neutral" Bla 3x

What is wrong? What should I do?

Sorry for multipost. I got signal trouble here. Please delete the rest comments

 

[nicoboss]

@nicoboss Will the new version have retro-compatibility with the NSPZ format ?

The next stable release will still have NSPZ support. In the end block compressed NSZ is very similar to the beautiful NSPZ format just without all the unnecessarily complexity which made NSPZ unfeasible for other software to implement. I might remove NSPZ/XCIZ support in the future if I see that there's no point in keeping it to not confuse people.

Differences between NSZ and NSPZ:

NSPZ/XCIZ:
- GitHub Project: https://github.com/nicoboss/nsZip
- Always uses Block compression allowing random read access to play compressed games in the future
- Decrypts the whole NCA
- Trims NDV0 fragments to their header and reconstructs them
- Only supported by nsZip and unfortunately doesn't really have a future

NSZ/XCZ:
- GitHub Project: https://github.com/nicoboss/nsz
- Uses solid compression by default. Block compression can be enabled using the -B option. Block compression will be the default for XCZ
- Decrypts all sections while keeping the first 0x4000 bytes encrypted. Puts informations needed to encrypt inside the header.
- Deleted NDV0 fragments as they have no use for end users as they only exist to save CDN bandwidth
- Already widely used. Supported by Tinfoil, SX Installer v3.0.0 and probably a lot of other software in the future

Great work guys . I.ll wait until its ironed out abit more a d works with goldleaf i dont like tinfoil

You don't have to use tinfoil. SX Installer v3.0.0 supports NSZ too. Solid compressed NSZ files are already ironed out very well and so widely used that it's unlikely a compatibility breaking change will ever be done. Goldfleaf support won't happen because of https://github.com/XorTroll/Goldleaf/issues/417 - obviously somebody could just fork Goldleafe and implement NSZ as it's open source but don't expect the official version to ever support NSZ.

I can't seem to get nsZIP to work anymore. It just opens up CMD, then closes immediately. I'm planning on switching to NCZ, but all of my old NSPs are compressed in the old format.

I got JIT error message just right after I click RUN.

It says, "couldn't load file or assembly 'netstandard. Version = 2.0.0.0. Culture=neutral" Bla 3x

What is wrong? What should I do?

The portable preview versions come without any installer so missing requirements like the .NET framework will prevent nsZip from working. Try https://github.com/nicoboss/nsZip/files/3459201/nsZip_v2.0.0_debug_v1_Issue3.zip from built at 2nd August 2019 but keep in mind that this is NOT an official version and made to diagnose an issue. Even though this version comes with an installer it WON'T receive any updates and will contain bugs.

 

[Deathscreton]

The portable preview versions come without any installer so missing requirements like the .NET framework will prevent nsZip from working. Try https://github.com/nicoboss/nsZip/files/3459201/nsZip_v2.0.0_debug_v1_Issue3.zip from built at 2nd August 2019 but keep in mind that this is NOT an official version and made to diagnose an issue. Even though this version comes with an installer it WON'T receive any updates and will contain bugs.

Nico, when I try the linked application, I get the same error I got form the other versions. The only version that actually works for me from your github is your v1 files. I've attached a screenshot of what happens when I load up the application. That CMD Prompt screen comes up, stays for a few seconds, then closes and nothing happens at all. Task Manager shows it opening, then closing suddenly. I've tried launching it through PS and CMD. Also just tried launching it via File Explorer as an Admin as well. No difference.

 

[mrdude]

A word of warning for anyone that uses this.

I had a folder called 'compressed' on my desktop - it contained many files and folders from stuff I had downloaded and programs I had been working on. This app deleted that folder and all of it's contents without any warning......so use carefully - probably best on a drive or sandbox/virtual machine - until it has these bugs sorted out.

 

[nicoboss]

A word of warning for anyone that uses this.

I had a folder called 'compressed' on my desktop - it contained many files and folders from stuff I had downloaded and programs I had been working on. This app deleted that folder and all of it's contents without any warning......so use carefully - probably best on a drive or sandbox/virtual machine - until it has these bugs sorted out.

Sorry! Never expect anybody to ever set the temp folder to a location where other folders with the name "decrypted" "encrypted" or "compressed" exists. nsZip needs these folders to store its temporary files and deletes them to clean them up afterwards. The behavior isn't really a bug but Desktop shouldn’t be the default location for temporary files (no idea why it’s desktop on latest preview) and there be a warning when selecting a temporary folder that already contains subfolders with such names. nsZip uses standard delete so files deleted that way can be recovered with any file recovery program as long the clusters where this file were stored aren’t already overwritten with a new file. Other than that, nsZip isn't dangerous in any way - take a look at its source code it you don't believe me.

 

[awankelam]

I use this debug version. But when in the progress I encounter issue that says the file is in use.. I used the stable version too, though .. But its still not going well.

 

[awankelam]

I use this debug version. But when in the progress I encounter issue that says the file is in use.. I used the stable version too, though .. But its still not going well.

This for the detail... How can I fix it?

 

[nicoboss]

I use this debug version. But when in the progress I encounter issue that says the file is in use.. I used the stable version too, though .. But its still not going well.

This for the detail... How can I fix it?

Does this happen to every game you try to compress or only for a specific one? Very interesting. Never seen that one before. Very wired how it crashes in the middle of compressing blocks as there is nothing that really could go wrong there. Are you sure you don't run out of HDD space or RAM? Is your OS x64 or x86? What game are you compressing when this issue occurs?

 

[awankelam]

Does this happen to every game you try to compress or only for a specific one? Very interesting. Never seen that one before. Very wired how it crashes in the middle of compressing blocks as there is nothing that really could go wrong there. Are you sure you don't run out of HDD space or RAM? Is your OS x64 or x86? What game are you compressing when this issue occurs?

Its happened almost every nsp that I've tried. From 6 files only succed 1 file with very small size, about 70mb. The others that failed is about 100mb to 10gb. Is in process of conversion need so much space? I read on explanation said that need space about 5 times of file that we want to convert. But I tried the 100mb file still failed with such kind of error, "cannot access file .. Because used by another process" . FYI, I alocate the output & temp file on non C: disk that still have 20gb space left. Because on C: its only about 2gb left.

 

[nicoboss]

Nico, when I try the linked application, I get the same error I got form the other versions. The only version that actually works for me from your github is your v1 files. I've attached a screenshot of what happens when I load up the application. That CMD Prompt screen comes up, stays for a few seconds, then closes and nothing happens at all. Task Manager shows it opening, then closing suddenly. I've tried launching it through PS and CMD. Also just tried launching it via File Explorer as an Admin as well. No difference.

Its happened almost every nsp that I've tried. From 6 files only succed 1 file with very small size, about 70mb. The others that failed is about 100mb to 10gb. Is in process of conversion need so much space? I read on explanation said that need space about 5 times of file that we want to convert. But I tried the 100mb file still failed with such kind of error, "cannot access file .. Because used by another process" . FYI, I alocate the output & temp file on non C: disk that still have 20gb space left. Because on C: its only about 2gb left.

I tried and can't reproduce any of your issues. I will continue investigate the reported issues and will fix them as soon I can reproduce them.
Why don't you just use https://github.com/nicoboss/nsz instead? NSZ is the successor of the NSPZ File format and until NSZ is implemented into nsZip I don't really see any reason to use it.

 

[RisingHopperZeroOne]

Is there a decompress script for nsz or is it compress only?

EDIT: Saw how to decompress, question answered.

 

[razinx]

Not having any luck with this tool. Any XCI I attempt seems to give all "Good" as its checking it, but then at the end I get,

at LibHac.Nca.DecryptHeader()
at LibHac.Nca..ctor(Keyset keyset, IStorage storage, Boolean leaveOpen)
at nsZip.LibHacControl.ProcessNca.Process(IFile inFile, IFile outFile, Boolean verifyBeforeDecrypting, Keyset keyset, Output Out) in C:\Users\nico\Documents\nsZip\LibHacControl\ProcessNca.cs:line 15
at nsZip.LibHacControl.ProcessXci.Decrypt(String inputFilePath, String outDirPath, Boolean verifyBeforeDecrypting, Keyset keyset, Output Out) in C:\Users\nico\Documents\nsZip\LibHacControl\ProcessXci.cs:line 76
at nsZip.TaskLogic.CompressXCI(String xciFile) in C:\Users\nico\Documents\nsZip\TaskLogic.cs:line 138
at nsZip.MainWindow.RunTask() in C:\Users\nico\Documents\nsZip\MainWindow.xaml.cs:line 182
Unable to decrypt NCA header.
Key Type: Common
Key Name: header_key


What am I missing? I have the prod.key in the .switch directory, so not sure what else I have to do to get this running.

TIA

 

[nicoboss]

Not having any luck with this tool. Any XCI I attempt seems to give all "Good" as its checking it, but then at the end I get,

at LibHac.Nca.DecryptHeader()
at LibHac.Nca..ctor(Keyset keyset, IStorage storage, Boolean leaveOpen)
at nsZip.LibHacControl.ProcessNca.Process(IFile inFile, IFile outFile, Boolean verifyBeforeDecrypting, Keyset keyset, Output Out) in C:\Users\nico\Documents\nsZip\LibHacControl\ProcessNca.cs:line 15
at nsZip.LibHacControl.ProcessXci.Decrypt(String inputFilePath, String outDirPath, Boolean verifyBeforeDecrypting, Keyset keyset, Output Out) in C:\Users\nico\Documents\nsZip\LibHacControl\ProcessXci.cs:line 76
at nsZip.TaskLogic.CompressXCI(String xciFile) in C:\Users\nico\Documents\nsZip\TaskLogic.cs:line 138
at nsZip.MainWindow.RunTask() in C:\Users\nico\Documents\nsZip\MainWindow.xaml.cs:line 182
Unable to decrypt NCA header.
Key Type: Common
Key Name: header_key


What am I missing? I have the prod.key in the .switch directory, so not sure what else I have to do to get this running.

TIA

As far I remember https://github.com/shchmue/Lockpick_RCM/releases should dump it. If not you can calculate it like this:
header_key = header_key_source + kek_source (pkg2->ini1->FS) + aes_kek(pkg1->TZ)/key_gen_source(pkg2->ini1->spl) + master_key_00
master_key_00 = master_key_source (pkg1ldr) + [decrypted keyblob = keyblob_key (pkg1ldr key source + sbk + tsec) + enc keyblob (boot0)]

Anyways only use nsZip for NSPs and try to avoid using it for XCIs. It's only there as proof of concept and its implementation has a mistake leading in some metadata being lost. It was planned to be properly implemented in nsZip v2.0 an its work nearly completed but then NSZ appeared. The new NSZ/XCZ will be the successor of NSPZ/XCIZ and supported by homebrew so all focus is currently at this new file format. nsZip 2.0 will get NSZ/XCI support but the NSZ reference project has way higher priority right now. The new file format also still hasn’t XCI support yes but will probably get it very soon. See https://gbatemp.net/threads/nsz-homebrew-compatible-nsp-xci-compressor-decompressor.550556/

 

[razinx]

As far I remember https://github.com/shchmue/Lockpick_RCM/releases should dump it. If not you can calculate it like this:
header_key = header_key_source + kek_source (pkg2->ini1->FS) + aes_kek(pkg1->TZ)/key_gen_source(pkg2->ini1->spl) + master_key_00
master_key_00 = master_key_source (pkg1ldr) + [decrypted keyblob = keyblob_key (pkg1ldr key source + sbk + tsec) + enc keyblob (boot0)]

Anyways only use nsZip for NSPs and try to avoid using it for XCIs. It's only there as proof of concept and its implementation has a mistake leading in some metadata being lost. It was planned to be properly implemented in nsZip v2.0 an its work nearly completed but then NSZ appeared. The new NSZ/XCZ will be the successor of NSPZ/XCIZ and supported by homebrew so all focus is currently at this new file format. nsZip 2.0 will get NSZ/XCI support but the NSZ reference project has way higher priority right now. The new file format also still hasn’t XCI support yes but will probably get it very soon. See https://gbatemp.net/threads/nsz-homebrew-compatible-nsp-xci-compressor-decompressor.550556/

Got it, thanks for the reply!

 

[Mat37]

When trying to decompress NSZ files (level 22 block compression) i run out of RAM for big games and can't decompress them. Is there a way around this ?

 

[nicoboss]

When trying to decompress NSZ files (level 22 block compression) i run out of RAM for big games and can't decompress them. Is there a way around this ?

nsZip doesn't even support the new NSZ file format yet. Do you mean https://github.com/nicoboss/nsz? If yes please create an issue under https://github.com/nicoboss/nsz/issues/new and I will look into it. Decompression definately shouldn't use much RAM and there's probably a memory leak somewhere.
Please post questions related to NSZ inside https://gbatemp.net/threads/nsz-homebrew-compatible-nsp-xci-compressor-decompressor.550556/