nsZip - NSP compressor/decompressor to reduce storage

nsZip is an open source PC tool to lossless compress/decompress NSP files in order to save a lot of storage while all NCA files keep their exact same hash. [prebreak]1[/prebreak]
But it’s more than just a compression. A lot of data like NDV0 partitions or fragment NCA3 files will be removed while compressing and exactly recreated when decompressed which saves even more space especially on updates higher than v65536.
In addition, the NSZ format was designed with emulators in mind so adding NSZ support to Yuzu will be possible in the future and because NSZ contains decrypted NCAs no keys would be needed to only extract game files. As compression algorithm Zstandard is used to multithreaded compress 256 KB chunks while uncompressible chunks are stored uncompressed. That way NSPZ/XCIZ allows random read access. Zstandard has a 43MB/s compression and 7032MB/s decompression speed on an 8 threaded CPU at level 18 while having one of the best compression ratios compared to other compression algorithms.

How compressing works: NSP => extracted NSP => decrypted NCAs => trim fragments => compressing to NSZ => verify correctness => repacking NSPZ file
How decompressing works: NSPZ file => extracted NSPZ => decompress NSZ => untrim fragments => encrypt NCAs => verify correctness => repack as NSP

Check out my GitHub page to report bugs, follow this project, post your suggestions or contribute.
GitHub link: https://github.com/nicoboss/nsZip/releases

For the new homebrew compatible NSZ format please see https://github.com/nicoboss/nsz
NSZ thread: https://gbatemp.net/threads/nsz-homebrew-compatible-nsp-xci-compressor-decompressor.550556/
Until NSZ support is implemented in nsZip I don't see why you want to use it instead of NSZ

Block compressed NSZ is very similar to the beautiful NSPZ format just without all the unnecessarily complexity which made NSPZ unfeasible for other software to implement. I might remove NSPZ/XCIZ support in the future if I see that there's no point in keeping it to not confuse people.

Differences between NSZ and NSPZ:

NSPZ/XCIZ:
- GitHub Project: https://github.com/nicoboss/nsZip
- Always uses Block compression allowing random read access to play compressed games in the future
- Decrypts the whole NCA
- Trims NDV0 fragments to their header and reconstructs them
- Only supported by nsZip and unfortunately doesn't really have a future

NSZ/XCZ:
- GitHub Project: https://github.com/nicoboss/nsz
- Uses solid compression by default. Block compression can be enabled using the -B option. Block compression will be the default for XCZ
- Decrypts all sections while keeping the first 0x4000 bytes encrypted. Puts informations needed to encrypt inside the header.
- Deleted NDV0 fragments as they have no use for end users as they only exist to save CDN bandwidth
- Already widely used. Supported by Tinfoil, SX Installer v3.0.0 and probably a lot of other software in the future


nsZip v2.0.0 preview 2:
  • Fixed XCI to XCIZ compression
nsZip v2.0.0 preview 1:
  • New UI made using WPF instead of WinForms
  • Huge performance improvements
    • Directly decrypt the NSP without extracting it first
    • Directly decompressing NSPZ files without extracting them first
    • Huge encrypting speed increase
    • Huge SHA-256 verification speed increase
    • Enhanced compressing multithreading
    • Support for up to 400 CPU Cores
  • Improved game compatibility
  • Command Line support
  • More settings
Version 1.1.0 Changelog:
  • Added support for over multiple NCAs splitted NDV0 fragment trimming
Version 1.0.1 Changelog:
  • Let the user continue when detecting a yet unimplemented multifile NDV0 fragment. I'll add proper support for that unimplemented multifile NDV0 fragment format in the following days.
  • The compression level can now be changed (tradeoff between speed and compression ratio)
  • The block size can now be changed (tradeoff between random access time and compression ratio)
Future planes:
  • NSZ implementation
FAQ:
Q: It prod.keys not found!
A: Dump them using Lockpick and copy them to %userprofile%/.switch/prod.keys
Q: How much storage does this save?
A: It depends on the game. The compressed size is usually around 40-80% of the original size.
Q: What about XCI?
A: Compression to XCIZ is supported, but XCIZ to XCI recreation is still in development.
Q: The program throws an error or seems to behaves not as intended!
A: Open an issue on my GitHub page where you give me exact steps how to reproduce.
Q: Does this compress installed games?
A: Not now but it's planned for the far future.

Screenshot:

nsZip_Screenshot.png
 
Last edited by nicoboss,

nicoboss

Well-Known Member
OP
Member
Joined
Feb 1, 2019
Messages
121
Trophies
0
Age
24
XP
1,003
Country
Switzerland
so it's like winzip or winrar but made specifically for Switch files ?
Yes, NSZ is a compression format specialized to compress the normally encrypted and incompressible NCA files (the content of NSP/XCI files). Unlike winzip/winrar NSZ allows random read access and so would be able to be directly played on Yuzu and even the Switch if somebody bothers writing a custom sysmodule for this.
 

tigersaman

Well-Known Member
Newcomer
Joined
Jul 10, 2018
Messages
78
Trophies
0
Age
29
XP
1,039
Country
Iran
Yes, NSZ is a compression format specialized to compress the normally encrypted and incompressible NCA files (the content of NSP/XCI files). Unlike winzip/winrar NSZ allows random read access and so would be able to be directly played on Yuzu and even the Switch if somebody bothers writing a custom sys module for this.

So for the sake of argument and since i myself have no nsp at hand to test the program can you give a comparison between a normal nsp and nspz in size of some games ?
and also thanks alot for creating this format/program.
 

nicoboss

Well-Known Member
OP
Member
Joined
Feb 1, 2019
Messages
121
Trophies
0
Age
24
XP
1,003
Country
Switzerland
So for the sake of argument and since i myself have no nsp at hand to test the program can you give a comparison between a normal nsp and nspz in size of some games ?
and also thanks alot for creating this format/program.

VVVVVV Base Game:
NSP: 80.6 MB
NSPZ: 55.5 MB

Qbics Paint v131072:
NSP: 142 MB
NSPZ: 46.9 MB

Qbics Paint v65536:
NSP: 69.2 MB
NSPZ: 43.8 MB

Qbics Paint Base Game:
NSP: 65.4 MB
NSPZ: 40.1 MB
 

Essasetic

General Spectator
Member
Joined
Jun 16, 2018
Messages
1,573
Trophies
0
XP
3,256
Country
United Kingdom
Can this further reduce the size of xci as well? That would be neat, if SX then implemented support for xciz as well.
You can already trim XCIs though.

--------------------- MERGED ---------------------------

And also I don't think XCIs have NCAs in them so the program wouldn't work with it.
 

nicoboss

Well-Known Member
OP
Member
Joined
Feb 1, 2019
Messages
121
Trophies
0
Age
24
XP
1,003
Country
Switzerland
What if you have NCA2 NSPs /s
The only way to really obtain NCA2 files is by restoring improperly deleted factory NCA files on a 1.0.0 console and all of them together are only a few hundred MB. There is barely any person that would even be able to obtain them as only a small amount even owns an untouched 1.0.0 switch. I have my factory NCA2 files dumped but it definitely not worth the efford supporting them with nsZip.

--------------------- MERGED ---------------------------

Oh. So XCI trimming but with NSPs. That's cool! I might give it a try if I ever need to get an update or a game.
No this isn't trimming but compressing. When trimming an XCI only useless zeros at the end gets removed.

Can this further reduce the size of xci as well? That would be neat, if SX then implemented support for xciz as well.
Yes XCIZ will further reduce the size of a trimmed XCI and TX could implement the format if they like.

nsZip can already compress XCI to XCIZ but not the opposite way so feels free to try it out if you like. Trimmed XCI compression is as good as NSP compression as they booth are file containers containing NCA files.
 
Last edited by nicoboss,
  • Like
Reactions: PT333

ZachyCatGames

Well-Known Member
Member
Joined
Jun 19, 2018
Messages
3,390
Trophies
1
Location
Hell
XP
3,859
Country
United States
The only way to really obtain NCA2 files is by restoring improperly deleted factory NCA files on a 1.0.0 console and all of them together are only a few hundred MB. There is barely any person that would even be able to obtain them as only a small amount even owns an untouched 1.0.0 switch. I have my factory NCA2 files dumped but it definitely not worth the efford supporting them with nsZip.
Yea, I know it probably isn’t worth the time to support them, I was just joking :P. And 1.0.0 factory titles aren’t the only way to get NCA2s, some pre-1.0.0 dev firmwares and titles are NCA2s
 
Last edited by ZachyCatGames,

thealgorithm

Well-Known Member
Member
Joined
Oct 27, 2015
Messages
393
Trophies
0
Age
46
XP
766
Country
Swaying from the topic a bit, but related in some way. Would there be a possibility at some stage to convert encrypted nsp's to unencrypted ones and to install the unencrypted nsps (similar to how encrypted or decrypted CIA's can be installed on the 3ds)?
 

Mat37

Well-Known Member
Member
Joined
Dec 3, 2016
Messages
790
Trophies
0
Age
20
Location
Does it matter ?
XP
1,163
Country
France
You can already trim XCIs though.

--------------------- MERGED ---------------------------

And also I don't think XCIs have NCAs in them so the program wouldn't work with it.
XCIs have NCAs, but they're not encrypted with the titlekey (this doesn't mean they aren't encrypted at all)
 

nicoboss

Well-Known Member
OP
Member
Joined
Feb 1, 2019
Messages
121
Trophies
0
Age
24
XP
1,003
Country
Switzerland
Swaying from the topic a bit, but related in some way. Would there be a possibility at some stage to convert encrypted nsp's to unencrypted ones and to install the unencrypted nsps (similar to how encrypted or decrypted CIA's can be installed on the 3ds)?
As proven in nsZip its possible to reencrypt an nca and get its exact same hash as before decrypting. So, you could just reencrypt it while installing. You can build something like an encryption stream and pass the so encrypted data directly what needs encrypted NCA data. So it really won’t be that hard to make a homebrew supporting NSPZ installing based on the tinfoil/Goldleaf.

--------------------- MERGED ---------------------------

XCIs have NCAs, but they're not encrypted with the titlekey (this doesn't mean they aren't encrypted at all)
The NCAs inside and XCI are AES-CTR encrypted with decrypted key area keys like nearly all non-RomFS NCAs inside an NSP too and are completely supported by nsZip.
 
Last edited by nicoboss,
  • Like
Reactions: JJTapia19
General chit-chat
Help Users
  • KenniesNewName @ KenniesNewName:
    EDOSTORY Adjustable Dumbbell Set, 44LB/66LB/88LB Free Weights Dumbbells ,4 in 1 Set Barbell, Dumbbell, Kettlebell and Push-ups, Non-slip Handles, Push up, Fitness Home Gym for Men Women https://a.co/d/bE2hsOR Neat might order
    Gift
  • x65943 @ x65943:
    certainly looks interesting
    Gift
  • x65943 @ x65943:
    only 22lbs total tho - how much can you really exercise with that?
    Gift
  • x65943 @ x65943:
    I'm a pretty weak guy and even I use 50lb on each side for dumbbell bench and 30lb for each arm with dumbbell curl
    Gift
  • x65943 @ x65943:
    imo without at least a 30lbs dumbbell you won't get a lot of use out of em - and these max out at like 10lbs for each arm
    Gift
  • x65943 @ x65943:
    oh I see you actually get to pick which set you want - so you could buy the 88 lb one
    Gift
  • KenniesNewName @ KenniesNewName:
    Didn't notice in the title it says 44LB/66LB/88LB
    Gift
  • KenniesNewName @ KenniesNewName:
    I really want to start lifting again
    Gift
  • KenniesNewName @ KenniesNewName:
    Gym memberships here are like $40 a month so in like years worth of time I'd have enough just to buy a nice in home set
    Gift
  • x65943 @ x65943:
    that's true, gyms are expensive.
    Gift
  • KenniesNewName @ KenniesNewName:
    I'd rather just have my own equipment instead of not knowing whose sweaty balls were on it
    Gift
  • x65943 @ x65943:
    kenny I'm very concerned now that you may not be working out properly
    Gift
  • KenniesNewName @ KenniesNewName:
    My life isn't proper
    Gift
  • x65943 @ x65943:
    first of all, there should be exactly zero reasons to place your sweaty balls on any of the equipment - generally you are supposed to wear clothing at the gym
    Gift
  • KenniesNewName @ KenniesNewName:
    You aren't going to the right gym if you need to wear clothes
    Gift
  • KenniesNewName @ KenniesNewName:
    Ball sweat drops down like normal sweat if your balls are always dry I'd recommend a doctor
    Gift
  • KenniesNewName @ KenniesNewName:
    We're talking about exercise balls of course
    Gift
  • x65943 @ x65943:
    what to do if I am a doctor
    Gift
  • KenniesNewName @ KenniesNewName:
    Re-read WebMD articles
    Gift
  • KenniesNewName @ KenniesNewName:
    I got myself a cheapo steering wheel cover here's to hoping the sun doesn't burn my finger tips over hot steering wheels
    Gift
  • x65943 @ x65943:
    I have never experienced that
    Gift
  • x65943 @ x65943:
    just where do you live mate
    Gift
  • KenniesNewName @ KenniesNewName:
    Midwest weathers been like 90 all week
    Gift
  • KenniesNewName @ KenniesNewName:
    We don't have garage parking areas at the apartment either so my car bakes in the sun
    Gift
  • KenniesNewName @ KenniesNewName:
    Midwest weather is so bipolar in the summer
    Gift
    KenniesNewName @ KenniesNewName: Midwest weather is so bipolar in the summer