nsZip - NSP compressor/decompressor to reduce storage

nsZip is an open source PC tool to lossless compress/decompress NSP files in order to save a lot of storage while all NCA files keep their exact same hash. [prebreak]1[/prebreak]
But it’s more than just a compression. A lot of data like NDV0 partitions or fragment NCA3 files will be removed while compressing and exactly recreated when decompressed which saves even more space especially on updates higher than v65536.
In addition, the NSZ format was designed with emulators in mind so adding NSZ support to Yuzu will be possible in the future and because NSZ contains decrypted NCAs no keys would be needed to only extract game files. As compression algorithm Zstandard is used to multithreaded compress 256 KB chunks while uncompressible chunks are stored uncompressed. That way NSPZ/XCIZ allows random read access. Zstandard has a 43MB/s compression and 7032MB/s decompression speed on an 8 threaded CPU at level 18 while having one of the best compression ratios compared to other compression algorithms.

How compressing works: NSP => extracted NSP => decrypted NCAs => trim fragments => compressing to NSZ => verify correctness => repacking NSPZ file
How decompressing works: NSPZ file => extracted NSPZ => decompress NSZ => untrim fragments => encrypt NCAs => verify correctness => repack as NSP

Check out my GitHub page to report bugs, follow this project, post your suggestions or contribute.
GitHub link: https://github.com/nicoboss/nsZip/releases

For the new homebrew compatible NSZ format please see https://github.com/nicoboss/nsz
NSZ thread: https://gbatemp.net/threads/nsz-homebrew-compatible-nsp-xci-compressor-decompressor.550556/
Until NSZ support is implemented in nsZip I don't see why you want to use it instead of NSZ

Block compressed NSZ is very similar to the beautiful NSPZ format just without all the unnecessarily complexity which made NSPZ unfeasible for other software to implement. I might remove NSPZ/XCIZ support in the future if I see that there's no point in keeping it to not confuse people.

Differences between NSZ and NSPZ:

NSPZ/XCIZ:
- GitHub Project: https://github.com/nicoboss/nsZip
- Always uses Block compression allowing random read access to play compressed games in the future
- Decrypts the whole NCA
- Trims NDV0 fragments to their header and reconstructs them
- Only supported by nsZip and unfortunately doesn't really have a future

NSZ/XCZ:
- GitHub Project: https://github.com/nicoboss/nsz
- Uses solid compression by default. Block compression can be enabled using the -B option. Block compression will be the default for XCZ
- Decrypts all sections while keeping the first 0x4000 bytes encrypted. Puts informations needed to encrypt inside the header.
- Deleted NDV0 fragments as they have no use for end users as they only exist to save CDN bandwidth
- Already widely used. Supported by Tinfoil, SX Installer v3.0.0 and probably a lot of other software in the future


nsZip v2.0.0 preview 2:
  • Fixed XCI to XCIZ compression
nsZip v2.0.0 preview 1:
  • New UI made using WPF instead of WinForms
  • Huge performance improvements
    • Directly decrypt the NSP without extracting it first
    • Directly decompressing NSPZ files without extracting them first
    • Huge encrypting speed increase
    • Huge SHA-256 verification speed increase
    • Enhanced compressing multithreading
    • Support for up to 400 CPU Cores
  • Improved game compatibility
  • Command Line support
  • More settings
Version 1.1.0 Changelog:
  • Added support for over multiple NCAs splitted NDV0 fragment trimming
Version 1.0.1 Changelog:
  • Let the user continue when detecting a yet unimplemented multifile NDV0 fragment. I'll add proper support for that unimplemented multifile NDV0 fragment format in the following days.
  • The compression level can now be changed (tradeoff between speed and compression ratio)
  • The block size can now be changed (tradeoff between random access time and compression ratio)
Future planes:
  • NSZ implementation
FAQ:
Q: It prod.keys not found!
A: Dump them using Lockpick and copy them to %userprofile%/.switch/prod.keys
Q: How much storage does this save?
A: It depends on the game. The compressed size is usually around 40-80% of the original size.
Q: What about XCI?
A: Compression to XCIZ is supported, but XCIZ to XCI recreation is still in development.
Q: The program throws an error or seems to behaves not as intended!
A: Open an issue on my GitHub page where you give me exact steps how to reproduce.
Q: Does this compress installed games?
A: Not now but it's planned for the far future.

Screenshot:

nsZip_Screenshot.png
 
Last edited by nicoboss,

Nazosan

Well-Known Member
Member
Joined
May 12, 2009
Messages
576
Trophies
1
XP
1,085
Country
United States
Looks really promising so far. I've been afraid to really go through and compress all my backups with it so far until it's more fully adopted, but at this point it's probably safe to do so and sure would help my poor backup drive (too many big things on the Switch!) I'm really grateful to you for making it portable btw.
 
  • Like
Reactions: nicoboss

LanzoThunder

Member
Newcomer
Joined
Apr 26, 2019
Messages
12
Trophies
0
Age
29
XP
79
Country
Italy
There is a bug inside nsZip v2.0.0 preview 1 that breaks XCI to XCIZ compression. Thanks a lot for reporting it. I fixed it in the latest version!

nsZip v2.0.0 preview 2 Changelog:
- Fixed XCI to XCIZ compression

Okay, no more error about the xciMeta.xml, but the error about the NCA3 file is still there.
Maybe this output could help?
 

Attachments

  • Immagine.png
    Immagine.png
    41 KB · Views: 166

nicoboss

Well-Known Member
OP
Member
Joined
Feb 1, 2019
Messages
132
Trophies
0
Age
26
XP
1,195
Country
Switzerland
Okay, no more error about the xciMeta.xml, but the error about the NCA3 file is still there.
Maybe this output could help?
Could you please let me know on which game this error occurs? It seems to be something game specific as it doesn't happen on the XCI's I'm using for testing. I'll try to fix that error as soon I'm able to reproduce it. Sorry for any bugs in XCI compression - I focused on NSP files so far.

Looks really promising so far. I've been afraid to really go through and compress all my backups with it so far until it's more fully adopted, but at this point it's probably safe to do so and sure would help my poor backup drive (too many big things on the Switch!) I'm really grateful to you for making it portable btw.
You don't really have to be afraid. As long verification is enabled nsZip verifies that you will get the exact same files after decompressing by actually decompressing, untrimming, encrypting, SHA256 hashing and comparing the resulted hashes with the NCAs file names while compressing and so makes nsZip impossible to ever produce a corrupted file even if something goes wrong. If you process all of them at once it will skip the ones that errors so just make sure not to accidentally delete the original file of them and report any errors that occurs so they can be fixed but there won’t be many incompatible NSP files left. I recommend to only compress NSP files for now as XCIZ to XCI decompression still not implemented yet and so compressing XCI files kind of pointless.
 

LanzoThunder

Member
Newcomer
Joined
Apr 26, 2019
Messages
12
Trophies
0
Age
29
XP
79
Country
Italy
Could you please let me know on which game this error occurs? It seems to be something game specific as it doesn't happen on the XCI's I'm using for testing. I'll try to fix that error as soon I'm able to reproduce it. Sorry for any bugs in XCI compression - I focused on NSP files so far.

Just for istance I tried 4 different games (3 .nsp and 1 .xci) and all gave me the same error. In the .zip file there are all the logs.

Btw I just noticed that when it tells me about the error (i.e.: "in nsZip.MainWindow.RunTask() in C:\Users\nico\Documents\nsZip\MainWindow.xaml.cs:riga 182") it says about a path that on my pc is not defined (C:\Users\nico\Documents\...) as my username in not "nico" lol idk if that could be related in any way
 

Attachments

  • LOGs.zip
    9.7 KB · Views: 160

SaulFemm

Well-Known Member
Newcomer
Joined
Oct 25, 2018
Messages
65
Trophies
0
Age
48
XP
595
Country
United States
Actually found a few files which consistently fail to compress with the same error. It's attached. I think this particular example was thrown when trying to compress Diablo III Update v262144.

2019-05-19 07_36_52-Greenshot image editor.png
 
  • Like
Reactions: nicoboss

nicoboss

Well-Known Member
OP
Member
Joined
Feb 1, 2019
Messages
132
Trophies
0
Age
26
XP
1,195
Country
Switzerland
Actually found a few files which consistently fail to compress with the same error. It's attached. I think this particular example was thrown when trying to compress Diablo III Update v262144.

View attachment 167459
Thanks for reporting. I was able to recreate that bug and should be fixed in the next versions.

Is this still under active development? This so far has been the only attempt at reducing NSP size that I've seen so far.
nsZip is still under active development and will probably stay under development for a long time. Due to Linux4Tegra and university development currently slowed down but expect huge progress in July when I have more time again. I’m currently the only developer in this project and only have limited time so feel free to contribute if you want faster progress or write an open source NSP/XCI file launcher homebrew and I will add NSPZ/XCIZ support to it. nsZip is currently the only lossless NSP compression tool and it's better that way as I designed it's underlaying NSZ file format quite well and with random access in mind and it’s more beneficial for the community for developers to work together on one project.
 
Last edited by nicoboss,

zxr750j

Well-Known Member
Member
Joined
Sep 29, 2003
Messages
927
Trophies
2
Location
Utrecht
XP
2,902
Country
Netherlands
Still compressing (takes a while)...
Currently: Bytes before: 1.512.723.338.034, bytes after 1.130.554.058.783
Saving something like 25% !
 

popy

Well-Known Member
Member
Joined
Jul 31, 2018
Messages
255
Trophies
0
Age
39
XP
1,278
Country
Austria
Dear @nicoboss , thx a lot for this nice tool.

Don't get me wrong on my question :-) Dont want to ask an ETA.
Just the following question:

How is development with the feature "XCIZ to XCI recreation" going?
Or (what i do not hope) is development stopped for this great tool?

Would be so nice if i could compress my xci's for backup reasons and also decompress it later.

thx
pOpY
 
Last edited by popy,

popy

Well-Known Member
Member
Joined
Jul 31, 2018
Messages
255
Trophies
0
Age
39
XP
1,278
Country
Austria
Another question:

Have tried to compress xzi to xciz -> works.
After decompressing xciz an nsp was created.

Is this currently the wanted behaviour?
Does those NSPs work as expected?

thx
pOpY
 

popy

Well-Known Member
Member
Joined
Jul 31, 2018
Messages
255
Trophies
0
Age
39
XP
1,278
Country
Austria
DONT DO THIS!
I have tried this: XCI -> XCIZ -> NSP -> install nsp
Game doesnt work and i had hundreds of "unkown" titles (spinning circle) on the home screen.
Had to remove every single from dthe settings menu.
Now its clean again.

pOpY
 

nicoboss

Well-Known Member
OP
Member
Joined
Feb 1, 2019
Messages
132
Trophies
0
Age
26
XP
1,195
Country
Switzerland
Dear @nicoboss , thx a lot for this nice tool.

Don't get me wrong on my question :-) Dont want to ask an ETA.
Just the following question:

How is development with the feature "XCIZ to XCI recreation" going?
Or (what i do not hope) is development stopped for this great tool?

Would be so nice if i could compress my xci's for backup reasons and also decompress it later.

thx
pOpY
I'm glad you like nsZip. XCIZ to XCI will be in the next release - I already have it working locally and most of its code was committed to https://github.com/nicoboss/nsZip/commits/master around 2 weeks ago. There was A LOT done since the last preview and even more since the last stable release and it definitely is time for a stable release very soon. I hope being able to release v2.0 of nsZip in the next few weeks but when exactly I can’t say as a critical bug delaying everything could be found at any time. Things left to do before the v2.0 release: Finish XCIZ to XCI conversion, fix a bug throwing an exception for a very low percentage of games, implement nsZip portable update notifications and a lot of testing. Until then XCI to XCIZ is just proof of concept to show how much storage can be saved - it even says "XCIZ to non-installable NSP" in the file select dialog. Version 2.0 won't be backwards compatible with the proof of concept XCIZ files created in previous versions to make it as easy as possible for an XCI launcher homebrew to support the XCIZ file format and to get XCI hash equality instead of just NCA hash equality. NSPZ will also get NSP hash equality – obviously without affecting compatibility. NSPZ files generated in any version should always be compatible with any future version. Same will apply to XCIZ after the release of nsZip 2.0.

Just for perspective how much work was done since the last release to understand why things take so long: We are currently 70 commits ahead of the latest stable version and 42 commits ahead of the latest preview. Now consider the amount of total commits of nsZip are 97 so more work was done since the last stable release then was done for all versions together until the latest stable release.

Futures that are planned to be in nsZip 2.0:
- XCIZ to XCI
- Near 100% game compatibility
=> All known compatibility related bugs where fixed
- Backend updated to new LibHac library
- Update Notification for nsZip portable
- Tools for NSPs/XCZs/NSPZs/XCIZs:
=> Extract Titlekeys
=> Extract Tickets
=> Extract PFS/HFS
=> Extract RomFS
- Performance improvements
- Everything that already was in v2.0.0p2
 
Last edited by nicoboss,

popy

Well-Known Member
Member
Joined
Jul 31, 2018
Messages
255
Trophies
0
Age
39
XP
1,278
Country
Austria
I'm glad you like nsZip. XCIZ to XCI will be in the next release - I already have it working locally and most of its code was committed to https://github.com/nicoboss/nsZip/commits/master around 2 weeks ago. There was A LOT done since the last preview and even more since the last stable release and it definitely is time for a stable release very soon. I hope being able to release v2.0 of nsZip in the next few weeks but when exactly I can’t say as a critical bug delaying everything could be found at any time. Things left to do before the v2.0 release: Finish XCIZ to XCI conversion, fix a bug throwing an exception for a very low percentage of games, implement nsZip portable update notifications and a lot of testing. Until then XCI to XCIZ is just proof of concept to show how much storage can be saved - it even says "XCIZ to non-installable NSP in the file select dialog. Version 2.0 won't be backwards compatible with the proof of concept XCIZ files created in previous versions to make it as easy as possible for an XCI launcher homebrew to support the XCIZ file format and to get XCI hash equality instead of just NCA hash equality. NSPZ will also get NSP hash equality – obviously without affecting compatibility. NSPZ files generated in any version should always be compatible with any future version. Same will apply to XCIZ after the release of nsZip 2.0.

Just for perspective how much work was done since the last release to understand why things take so long: We are currently 70 commits ahead of the latest stable version and 42 commits ahead of the latest preview. Now consider the amount of total commits of nsZip are 97 so more work was done since the last stable release then was done for all versions together until the latest stable release.

Futures that are planned to be in nsZip 2.0:
- XCIZ to XCI
- Near 100% game compatibility
=> All known compatibility related bugs where fixed
- Backend updated to new LibHac library
- Update Notification for nsZip portable
- Tools for NSPs/XCZs/NSPZs/XCIZs:
=> Extract Titlekeys
=> Extract Tickets
=> Extract PFS/HFS
=> Extract RomFS
- Performance improvements
- Everything that already was in v2.0.0p2

Wow, thats really exciting News!!!!!
Thanks a lot for your time and efforts you put into this.

I think i have missed the point "XCIZ to non-installable NSP", sorry for that.
Is it possible to provide an preview 3 build when XCIZ to XCI conversation is ready? :bow:
I am here to test this on an few games. Can also binary compare (or hash, crc, ...) the XCI before compression and after decompression to check the results.
Just want to say, i am here to test :)

Keep up the good work.
thx
pOpY
 
  • Like
Reactions: nicoboss

gabest

Member
Newcomer
Joined
Sep 19, 2013
Messages
10
Trophies
0
Age
124
XP
107
Country
Hungary
I was testing nsZip in the last couple of days. The block based compression is nice, but honestly, if you want to save the most space, you have to keep the plain, decompressed ncas (forked the project on github) and use winrar or 7zip with the highest compression settings. Went through a terabyte of my collection (A-H) and currently at 60% compression ratio. The source is a mix of zipped xci and nsp.
 

popy

Well-Known Member
Member
Joined
Jul 31, 2018
Messages
255
Trophies
0
Age
39
XP
1,278
Country
Austria
Wow, 60% compression ratio, that's insane.
@nicoboss: would this be an alternative to use 7zip library/algorithm and increase compression ratio?

Anyway, would be nice to get an preview version to test XCI compression/decompression with the current implementation.
I am here to test :-)

thx
pOpY
 

nicoboss

Well-Known Member
OP
Member
Joined
Feb 1, 2019
Messages
132
Trophies
0
Age
26
XP
1,195
Country
Switzerland
I was testing nsZip in the last couple of days. The block based compression is nice, but honestly, if you want to save the most space, you have to keep the plain, decompressed ncas (forked the project on github) and use winrar or 7zip with the highest compression settings. Went through a terabyte of my collection (A-H) and currently at 60% compression ratio. The source is a mix of zipped xci and nsp.
Solid compression (NSZ type 0) was planned since the very beginning. There are 3 types of NSZ files specified in the nsZip readme page. I decided to implemented type 1 first as type 0 doesn't offer random read access and so can't be used for anything then storing while save the most space. Unlike NSZ type 0, NSZ type 1 technically supports running compressed games on Yuzu and even real hardware once somebody implements that.
Thanks a lot for working on a long planned but never implemented feature. I really appreciate your work and am willing to help you as much as I can. If you like you can send me your Discord username and I’ll add you - I should always be available for any questions there. I really like the work you've done on https://github.com/gabest11/nsZip so far. Please open a pull request once you think it's ready to merge. If you like I can even add you as collaborator to the main repository. I had an extremely busy week and will look into the issues you created tomorrow.

Wow, 60% compression ratio, that's insane.
@nicoboss: would this be an alternative to use 7zip library/algorithm and increase compression ratio?

Anyway, would be nice to get an preview version to test XCI compression/decompression with the current implementation.
I am here to test :-)

thx
pOpY
It's always a tradeoff between better compression ratio, compression speed and with what you can do with the compressed file. I obviously want to let everybody decide by their own how to compress their games and the only reason why NSZ type 0 and other compression algorithms weren’t implemented yet is because there are still more important features left to implement first like your highly anticipated compression/decompression.
I'm sorry I still wasn't able to put everything together and build a preview for you so far. University started again last Monday and my PCs water loop leaked after installing a new SSD destroying my GPU resulting in me having to wait for a new one, dealing with driver issues and reinstalling everything on the new SSD. Anyways everything is now ready again and I hope I find time for it this weekend but I can't promise anything as I have a lot of university work to do first.
 
  • Like
Reactions: popy

Site & Scene News

Popular threads in this forum

General chit-chat
Help Users
  • No one is chatting at the moment.
    LeoTCK @ LeoTCK: yes for nearly a month i was officially a wanted fugitive, until yesterday when it ended