nsZip - NSP compressor/decompressor to reduce storage

nsZip is an open source PC tool to lossless compress/decompress NSP files in order to save a lot of storage while all NCA files keep their exact same hash. [prebreak]1[/prebreak]
But it’s more than just a compression. A lot of data like NDV0 partitions or fragment NCA3 files will be removed while compressing and exactly recreated when decompressed which saves even more space especially on updates higher than v65536.
In addition, the NSZ format was designed with emulators in mind so adding NSZ support to Yuzu will be possible in the future and because NSZ contains decrypted NCAs no keys would be needed to only extract game files. As compression algorithm Zstandard is used to multithreaded compress 256 KB chunks while uncompressible chunks are stored uncompressed. That way NSPZ/XCIZ allows random read access. Zstandard has a 43MB/s compression and 7032MB/s decompression speed on an 8 threaded CPU at level 18 while having one of the best compression ratios compared to other compression algorithms.

How compressing works: NSP => extracted NSP => decrypted NCAs => trim fragments => compressing to NSZ => verify correctness => repacking NSPZ file
How decompressing works: NSPZ file => extracted NSPZ => decompress NSZ => untrim fragments => encrypt NCAs => verify correctness => repack as NSP

Check out my GitHub page to report bugs, follow this project, post your suggestions or contribute.
GitHub link: https://github.com/nicoboss/nsZip/releases

For the new homebrew compatible NSZ format please see https://github.com/nicoboss/nsz
NSZ thread: https://gbatemp.net/threads/nsz-homebrew-compatible-nsp-xci-compressor-decompressor.550556/
Until NSZ support is implemented in nsZip I don't see why you want to use it instead of NSZ

Block compressed NSZ is very similar to the beautiful NSPZ format just without all the unnecessarily complexity which made NSPZ unfeasible for other software to implement. I might remove NSPZ/XCIZ support in the future if I see that there's no point in keeping it to not confuse people.

Differences between NSZ and NSPZ:

NSPZ/XCIZ:
- GitHub Project: https://github.com/nicoboss/nsZip
- Always uses Block compression allowing random read access to play compressed games in the future
- Decrypts the whole NCA
- Trims NDV0 fragments to their header and reconstructs them
- Only supported by nsZip and unfortunately doesn't really have a future

NSZ/XCZ:
- GitHub Project: https://github.com/nicoboss/nsz
- Uses solid compression by default. Block compression can be enabled using the -B option. Block compression will be the default for XCZ
- Decrypts all sections while keeping the first 0x4000 bytes encrypted. Puts informations needed to encrypt inside the header.
- Deleted NDV0 fragments as they have no use for end users as they only exist to save CDN bandwidth
- Already widely used. Supported by Tinfoil, SX Installer v3.0.0 and probably a lot of other software in the future


nsZip v2.0.0 preview 2:
  • Fixed XCI to XCIZ compression
nsZip v2.0.0 preview 1:
  • New UI made using WPF instead of WinForms
  • Huge performance improvements
    • Directly decrypt the NSP without extracting it first
    • Directly decompressing NSPZ files without extracting them first
    • Huge encrypting speed increase
    • Huge SHA-256 verification speed increase
    • Enhanced compressing multithreading
    • Support for up to 400 CPU Cores
  • Improved game compatibility
  • Command Line support
  • More settings
Version 1.1.0 Changelog:
  • Added support for over multiple NCAs splitted NDV0 fragment trimming
Version 1.0.1 Changelog:
  • Let the user continue when detecting a yet unimplemented multifile NDV0 fragment. I'll add proper support for that unimplemented multifile NDV0 fragment format in the following days.
  • The compression level can now be changed (tradeoff between speed and compression ratio)
  • The block size can now be changed (tradeoff between random access time and compression ratio)
Future planes:
  • NSZ implementation
FAQ:
Q: It prod.keys not found!
A: Dump them using Lockpick and copy them to %userprofile%/.switch/prod.keys
Q: How much storage does this save?
A: It depends on the game. The compressed size is usually around 40-80% of the original size.
Q: What about XCI?
A: Compression to XCIZ is supported, but XCIZ to XCI recreation is still in development.
Q: The program throws an error or seems to behaves not as intended!
A: Open an issue on my GitHub page where you give me exact steps how to reproduce.
Q: Does this compress installed games?
A: Not now but it's planned for the far future.

Screenshot:

nsZip_Screenshot.png
 
Last edited by nicoboss,

Mat37

Well-Known Member
Member
Joined
Dec 3, 2016
Messages
790
Trophies
0
Age
21
Location
Does it matter ?
XP
1,368
Country
France
nsZip doesn't even support the new NSZ file format yet. Do you mean https://github.com/nicoboss/nsz? If yes please create an issue under https://github.com/nicoboss/nsz/issues/new and I will look into it. Decompression definately shouldn't use much RAM and there's probably a memory leak somewhere.
Please post questions related to NSZ inside https://gbatemp.net/threads/nsz-homebrew-compatible-nsp-xci-compressor-decompressor.550556/
I just posted an issue with a log file, and yes i'm talking about nsz, my bad
 

lettuce

Well-Known Member
Member
Joined
Jul 27, 2006
Messages
811
Trophies
0
XP
491
Country
So im confused as to which program i should use?, NSPZ or NSZ?? and how to actually use it?, is there not a windows build?
 
Last edited by lettuce,

nicoboss

Well-Known Member
OP
Member
Joined
Feb 1, 2019
Messages
132
Trophies
0
Age
26
XP
1,195
Country
Switzerland
So im confused as to which program i should use?, NSPZ or NSZ?? and how to actually use it?, is there not a windows build?
NSZP is depreciated so use NSZ instead: https://github.com/nicoboss/nsz - it's Python and so works on every platform once you install https://www.python.org/downloads/.
If you can't get it to work here an unofficial Windows executable I made last Sunday: http://www.nicobosshard.ch/nsz/nsz_nuitka_17f153d_portable.zip
The next NSZ release will most likely contain windows binaries.
 

lettuce

Well-Known Member
Member
Joined
Jul 27, 2006
Messages
811
Trophies
0
XP
491
Country

nicoboss

Well-Known Member
OP
Member
Joined
Feb 1, 2019
Messages
132
Trophies
0
Age
26
XP
1,195
Country
Switzerland
hmm the windows executable does seem to load, or is it command line only?
Currently it's command line only. GUI will be implemented soon. Until then just use command line arguments. Use '.\nsz.exe --help' too see how to use it. You probably want to use it like '.\nsz.exe -C --level 22 --threads 8 "game.nsp"' to compress and '.\nsz.exe -D "game.nsz"' to decompress. Instead of "game.nsp" and "game.nsz" you can also specify a folder path containing multiple games.
 
Last edited by nicoboss,
  • Like
Reactions: lettuce

awankelam

Member
Newcomer
Joined
Oct 19, 2019
Messages
14
Trophies
0
Age
35
XP
113
Country
Indonesia
I tried and can't reproduce any of your issues. I will continue investigate the reported issues and will fix them as soon I can reproduce them.
Why don't you just use https://github.com/nicoboss/nsz instead? NSZ is the successor of the NSPZ File format and until NSZ is implemented into nsZip I don't really see any reason to use it.

I've used it. Yet still encounter another issue... With this, I even can't pass the requirements installation sessions. I hope someday, you or anyone can make such kind of program than just need click and run so we can use it for much easier, especially for know nothing about programming just like me. I'll appreciate it so much. Hehehe.
 

Francesca

Well-Known Member
Member
Joined
Mar 17, 2015
Messages
403
Trophies
0
XP
964
Country
Armenia
Will this reduce the file size to the point I can move it to my SD Card with a USB connection (under 4 GB)? How much space does this method save?
 

GnK23

Well-Known Member
Member
Joined
Apr 6, 2016
Messages
170
Trophies
0
Age
35
XP
671
Country
United States
I've used it. Yet still encounter another issue... With this, I even can't pass the requirements installation sessions. I hope someday, you or anyone can make such kind of program than just need click and run so we can use it for much easier, especially for know nothing about programming just like me. I'll appreciate it so much. Hehehe.

Use ryjin https://gbatemp.net/threads/ryjin-a-nsz-converter-mod.550174/page-7 look for the portable version. It uses the same nsz.exe of nicoboss
 

nicoboss

Well-Known Member
OP
Member
Joined
Feb 1, 2019
Messages
132
Trophies
0
Age
26
XP
1,195
Country
Switzerland
I've used it. Yet still encounter another issue... With this, I even can't pass the requirements installation sessions. I hope someday, you or anyone can make such kind of program than just need click and run so we can use it for much easier, especially for know nothing about programming just like me. I'll appreciate it so much. Hehehe.
Latest NSZ commit as python independently portable exe created using "nuitka nsz.py --standalone --plugin-enable=multiprocessing": http://www.nicobosshard.ch/nsz/nsz_nuitka_ab76c8e_portable.zip
 
  • Like
Reactions: awankelam

nicoboss

Well-Known Member
OP
Member
Joined
Feb 1, 2019
Messages
132
Trophies
0
Age
26
XP
1,195
Country
Switzerland
I tried installing nszgui from pip, but it doesn't seem to exist on PyPi. Has it been removed?
Sorry that's a mistake in the readme. It's "pip install nsz[gui]". Untill the next release "pip install nsz" will install the GUI too as since the change of making GUI optional no new version of nsz got released.
 
  • Like
Reactions: kidcharlemagne

lordelan

Well-Known Member
Member
Joined
Jan 4, 2015
Messages
5,769
Trophies
1
Age
44
XP
6,474
Country
Germany
@nicoboss @blawar
Just for clarification: NSZ/XCZ are not really compressing things so if SX OS would ever support mounting XCZ files it wouldn't necessarily lead to longer loading times, right?
 

_Mizuto_

Active Member
Newcomer
Joined
Sep 22, 2019
Messages
25
Trophies
0
Age
22
XP
419
Country
Madagascar
Hi,
What would be the best settings for a high compression ratio ?
And what are the differences between solid and block compression ?
 

Site & Scene News

Popular threads in this forum

General chit-chat
Help Users
    Sicklyboy @ Sicklyboy: https://www.youtube.com/watch?v=zufnIPGYQx8