RELEASE NSZ - Homebrew compatible NSP/XCI compressor/decompressor

NSZ is an open source MIT licensed python script to lossless compress/decompress NSP files in order to save a lot of storage while all NCA files keep their exact same hash. The NSZ file format for compressed Nintendo Switch games is already widely adopted and supported by popular homebrews like Awoo-Installer, GoldBricks, OG Tinfoil, Tinfoil, Lithium and SX Installer 3.00.

Currently it improves size, speed and bandwidth used for storing and transferring NSZ files while still leading to the same size when installed but integration into Yuzu and CFW to directly play compressed games is planned. NSZ's developments started by blawar based on the idea of nsZip and its NSPZ/XCIZ file format which unfortunately was too complex to be implemented by homebrew developers. I very soon took over his project and am closely working together with all homebrew developers interested in the NSZ format.

How to install:
Put dumped prod.keys to %userprofile%/.switch, install python, execute "pip install nsz" and use "nsz" like every other cmd command.

You can get the latest release of NSZ under https://github.com/nicoboss/nsz/releases
To read the readme and take a look at its source code visit https://github.com/nicoboss/nsz

Results how well NSZ performs: https://gbatemp.net/threads/nsz-title-compression-results.549831/
An easy to use tool for NSZ compression: https://github.com/julesontheroad/NSC_BUILDER
Anoter tool for NSZ compression: https://gbatemp.net/threads/ryjin-a-nsz-converter-mod.550174/

Here some screenshots of the GUI:
http://www.nicobosshard.ch/nsz/NSZ_GUI/Mainscreen.png
http://www.nicobosshard.ch/nsz/NSZ_GUI/Settings.png
http://www.nicobosshard.ch/nsz/NSZ_GUI/Advanced.png
http://www.nicobosshard.ch/nsz/NSZ_GUI/FolderBrowser.png
http://www.nicobosshard.ch/nsz/NSZ_GUI/About.png

Changelog:

NSZ 4.0:

• Implemented Drag & Drop support as requested in #51
• Implemented CRC32 key validation and added support for future masterkeys
  • Fixed the issue of master_key_0a not being recognized
  • Added CRC32 for master_key_0a
• Windows 7 support for Windows builds
• Improved GUI font size scaling
• Set the NSZ GUI window to be TopMost (always on top) on Windows so Drag & Drop gets much more convenient
  • Added a setting to specify if the Kivy window should be always on top or not
• XCI/XCZ finally extracts to folders containing the NCA/NCZ files instead of HFS0 partition dumps
• Fixed a major XCI compatibility bug by implementing compression/decompression support for NCA files with the first section having a smaller or larger offset then 0x4000. This fixes #49
• Added NSPZ (nsZip legacy file format) extraction support
• Make GUI an optional install
• Fixed #59 ncz decompression is not working
• Cleaned up imports for nsz package
• Stop bar manager to avoid broken shells
• Added pywin32 as GUI dependency for Windows. This fixes #56
• Fixed BlockDecompressorReader.seek with whence = 2 (seek relative to the file's end). This fixes #64
• Starting nsz.py from within a different working directory finally works
  • Fixes the current Azure Pipeline issue
• Added solid decompression, block decompression, solid compression and block compression tests to azure-pipelines.yml
• Made NSZ new returning with error code 1 if there are any exceptions
• Fixed deadlock in BlockCompressor.py
• Highly improved block decompression speed by caching the current block
• Added Visual Studio 2019 Python Project
• Added titleId and version to the file list and highly improved its design
• The SelectableLabel items inside the game list now properly scales its height according to the available width and text length of the file path. This fixes #50
• Added a multi-language supporting open source font for #61
• Implemented input folder as output folder by default for #61
• Waiting for Enter before exit when started over GUI so errors and console output can be seen before it closes for #74
• Fixed install failing on Kubuntu 20.04 and a lot of other modern Linux distributions by improving install_linux.sh for #75
• Removed code that manipulated the XCI header size for absolutely no reason which fixes #77
• Improved decompression speed by 400% by heavily reducing the amount of performance intensive status bar refresh calls
• The decision if the last block should be decompressed or copied now matches the file format specifications by comparing the decompressed block size of that specific block with its compressed size. This issue was caused by missing the (unlikely) edge case that the last block can be larger when compressed without exceeding the general block size. This fixes #79
• Improved BlockCompression performance and overall CPU usage by reducing the amount of performance intensive status bar refresh calls.
• Ensure that the line right to the curser is clean when the application terminates
• Implemented undupe, undupe-dryrun, undupe-prioritylist, undupe-whitelist and undupe-old-versions to remove duplicate games
• Highly improved the missing prod.keys/keys.txt error message by not showing the stack trace and waiting for a user input before exiting
• Highly improved README.md
• General system stability improvements to enhance the user's experience.

NSZ 3.1:

• Fixed broken decompression in v3.0.0 due to an optional argument not marked as such and an outdated file extension comparison
• Allow the selection of individual files inside the OpenFileDialog
• Made it visible which drive is currently selected inside the OpenFileDialog and SaveFileDialog
• Replaced the background_down image with the background_normal one as it looks ugly inside the FileDialogs
• Removed SaveFileDialog as its unused and a pain to maintain
• Highly improved the OpenFileDialog by allowing switching between the icon and list layout
• Added filter to the OpenFileDialog so it only displays nsp, nsz, xci, xcz and ncz files
• No longer showing the empty placeholder for the device selection on non-Windows platforms an made per OpenFileDialog argument specifiable file filters possible
• Finally enabled the selection of folders and selecting multiple items at the same time
• Automatically creating the empty gui folder required for settings to be saved in nsz portable which is exactly the fix manually applied to nsz_v3.0.0_hotfix1_win64_portable.zip
• Fixed the warnings that appeared in the console on every GUI launch
• Made the file browser view mode buttons in the same design as the Windows device selection buttons
• Implemented deletion of selected GameList items using the delete or backspace key
• Highly improved the GameList item selection and deletion
• Vixed verification only mode not executing when called from GUI
• Set the default amount of threads for solid compression to 3 while keeping the number of logical cores the default for block compression because the majority is now using task parallelization for solid compression
• Fixed: AttributeError: 'RootWidget' object has no attribute 'verify'
• Dirty fixed Kivy throwing "Error in sys.excepthook" during shutdown if the amount of RecycleView data was ever decreasing
  • I tried multiple hours fixing this the proper way without success. I also tried removing all children what makes it to throw the same exception without even decreasing the amount of data and tried filling it with dummy data on shutdown in self.rootWidget.gameList.recycleView.shutdown() which didn’t worked because it’s already too late to spawn everything needed to avoid this exception.
• Improved the wording of the error summary
• Updated testing and deployment scripts
• Fixed orphan processes remaining after the main process terminated after block compressing a very short task resulting in the orphan processes spamming "AttributeError: 'ForkAwareLocal' object has no attribute 'connection'" exceptions
• Fixed NSZ GUI icon not showing
• General system stability improvements to enhance the user's experience.

NSZ 3.0:

• GUI:
  • Contains all functions available using command line arguments
• XCZ support
  • Block compressed (default)
  • Solid compressed
  • XCZ to XCI decompression
• Implemented task parallel solid compression. See --multi
• Implemented titlekey extraction with titlekeys.txt and titledb support. See --titlekeys
• Added regex support to the extract option to allow the user to specify exactly which files should be extracted from the container. See --extractregex
• NCZ decompression directly in nsz #38
• Updated IndependentNczDecompressor to the latest version inside nsz
• Fully replace tqdm with enlighten
  • Implemented multithreaded multiple process bars communication system for solid compression and verification while avoiding stdout race conditions
• Endless decompression/verification on a few block compressed games #25
• Fixed a major bug causing the space between block compressed NCZ files to be filled up with 0x00 to fit their uncompressed size
• Major Scripts Cleanup
• Fixed NSZ Decompressor TQDM Progress Bar
• Fixed --rm-old-version
• fixed path bug with decompression
• Removed pycryptodome v3.9.0 restriction as its latest v3.9.3 works fine
• Improved installation guide
• Fixed an exception that could occur when trying to receive keys under special circumstances during debugging
• Fixed a major bug inside the enhanced file existing check leading to files with the same name as a file already existing inside the output directory being overwritten without specifying this behavior using the –overwrite command line argument. This bug was cause by comparing the output file path instead of the output filename with existing filenames.
• Made nsz pip package building Kivy compatible
• Switched from Nuitka to PyInstaller due to Kivy compatibility
• Changed how --extract and --verify arguments are handled internally
• Set up CI with Azure Pipelines using self-hosted server
• Fully switched to pathlib. This fixes #41 and a lot of other file path related issues
• Fixed Enhanced File Existing Check. Adapting to pathlib and finally fixing --overwrite and --rm-old-version
• Improved exception handling related to outdated keys.txt which fixes issue #29 and #40
• General system stability improvements to enhance the user's experience.

NSZ 2.1.1:

• Fixed block compression for pip and Nuitka (nsz_win64_portable) by using sys.argv[0] instead of __main__.__file__ so threads no longer need to be prevented from initializing their own nut environment
• Made installation instructions easier to see and understand
• Scripts to automate testing and publishing

NSZ 2.1:

• 98 commits worth of changes since v2.0
• Added pip support
• Added Nuitka standalone windows build support
• Enhanced File Existing Check #20
  • Skip already compressed/decompressed files by default
  • --overwrite
  • --rm-old-version
  • Extracting TitleIDs and Versions from filename if possible (#17 and #19)
    • The titleID checking is now immensely faster than when extracting from Cnmt
  • --parseCnmt to get TitleID/version from Cnmt if not extractable from filename
    • Otherwise it falls back to simple filename checking which is much faster
• Batch error handling with tracebacks (#16)
  • Some debugging codes to find out erroneous files
  • Prevent batch process raising errors
  • Batch error handling with tracebacks
• The --thread option now works even for solid compression however the progress bar still has some visual glitches
• NSP/NSZ file hash verification now uses the hashes inside Cnmt instead of the nca filename (#22)
• Fixed decompression memory leak (#21)
  • The memory leak only occurs for dctx.stream_reader and was fixed by switching to the simple decompressing API which makes more sense for block decompression anyways
• Improved pageReadSize calculation speed by using math instead of a while loop
• Added option --remove-source that deletes the source file after compression or decompression (#24)
  • For this we finally properly closed file containers too
• Fix huge compression memory leak (#13)
• Fixed wrong working directory when starting nut.py from a different directory (#18)
• Improved exception handling during TitleID/Version extraction
  • Added support for prod.keys
• Reorganized file structure
• Fixed keys.txt path to always be the folder containing nsz.py
• Fixed default thread amount to be cpu_count()
• Implemented python version checking to prevent python from showing the users confusing compatibility related exceptions
  • Compatible and tested with Python 3.6 and later
• General system stability improvements to enhance the user's experience.

NSZ 2.0:

• Fully implemented block compression which can be enabled using the --block option
  • Supports for random read access on compressed files
  • Highly multithreaded compression when block compressing
  • Technically supports playing compressed games in the future
  • Current title installers do not support this yet
  • Comes with a low compression ratio cost
• Implemented NSP/NSZ file hash verification
• Overwrite/Duplicate protection
• Reorganized the project's folder structure and enhanced code readability
• Improved user feedback in form of better understandable messages and errors
• Fixed a lot of bugs and non-working features
• Added MIT License so all code inside this project can be used for whatever you like
• General system stability improvements to enhance the user's experience.

NSZ 1.0:

• Scripts to compress and decompress NSZ files.

Differences between NSZ and NSPZ:

NSZ/XCZ:
- GitHub Project: https://github.com/nicoboss/nsz
- Uses solid compression by default. Block compression can be enabled using the -B option. Block compression will be the default for XCZ
- Decrypts all sections while keeping the first 0x4000 bytes encrypted. Puts information needed to encrypt inside the header.
- Deleted NDV0 fragments as they have no use for end users as they only exist to save CDN bandwidth
- Already widely used. Supported by Tinfoil, SX Installer v3.0.0 and probably a lot of other software in the future

NSPZ/XCIZ:
- GitHub Project: https://github.com/nicoboss/nsZip
- Always uses Block compression allowing random read access to play compressed games in the future
- Decrypts the whole NCA
- Trims NDV0 fragments to their header and reconstructs them
- Only supported by nsZip and unfortunately doesn't really have a future

 

NSZ is a compressed NSP and XCZ a compressed XCI. NSZ is a lossless compression format so you can convert back and forth between NSP and NSZ without losing any data. To decompress an NSZ file back to an NSP just use https://github.com/nicoboss/nsz with "python nsz.py -D YourCompressedGame.nsz"

 

Today NSZ 2.1 got finally released!

NSZ 2.1.1:

• Fixed block compression for pip and Nuitka (nsz_win64_portable) by using sys.argv[0] instead of __main__.__file__ so threads no longer need to be prevented from initializing their own nut environment
• Made installation instructions easier to see and understand
• Scripts to automate testing and publishing

NSZ 2.1:

• 98 commits worth of changes since v2.0
• Added pip support
• Added Nuitka standalone windows build support
• Enhanced File Existing Check #20
  • Skip already compressed/decompressed files by default
  • --overwrite
  • --rm-old-version
  • Extracting TitleIDs and Versions from filename if possible (#17 and #19)
    • The titleID checking is now immensely faster than when extracting from Cnmt
  • --parseCnmt to get TitleID/version from Cnmt if not extractable from filename
    • Otherwise it falls back to simple filename checking which is much faster
• Batch error handling with tracebacks (#16)
  • Some debugging codes to find out erroneous files
  • Prevent batch process raising errors
  • Batch error handling with tracebacks
• The --thread option now works even for solid compression however the progress bar still has some visual glitches
• NSP/NSZ file hash verification now uses the hashes inside Cnmt instead of the nca filename (#22)
• Fixed decompression memory leak (#21)
  • The memory leak only occurs for dctx.stream_reader and was fixed by switching to the simple decompressing API which makes more sense for block decompression anyways
• Improved pageReadSize calculation speed by using math instead of a while loop
• Added option --remove-source that deletes the source file after compression or decompression (#24)
  • For this we finally properly closed file containers too
• Fix huge compression memory leak (#13)
• Fixed wrong working directory when starting nut.py from a different directory (#18)
• Improved exception handling during TitleID/Version extraction
  • Added support for prod.keys
• Reorganized file structure
• Fixed keys.txt path to always be the folder containing nsz.py
• Fixed default thread amount to be cpu_count()
• Implemented python version checking to prevent python from showing the users confusing compatibility related exceptions
  • Compatible and tested with Python 3.6 and later
• General system stability improvements to enhance the user's experience.

 

Block compressed NSZ/XCZ are compressed in chunks what makes them mountable. XCI files are known to be the mountable format so XCZ will use block compression by default while NSZ uses solid compression by default.

Block compressed NSZ/XCI mounting for PC will probably be released in the next few weeks. This will allow playing compressed games directly on emulator and use them with any tools not supporting NSZ/XCZ without decompression them first. When it will come to real hardware depends when and if Blawar manages to find somebody helping him to write a game mounting system model or if SX will implement XCZ support but that somebody will do it in the future seems quite likely.

Tinfoil currently already supports installing block compressed NSZ files but when doing so there's no benefit over solid NSZ. In the long-term block compressed NSZ seems to be better than solid compressed ones but if you never plan to mount your games there's no reason to use it as it comes with around 3% of compression ratio cost.

 

The NCAs are bit-identical. NSP is just a container similar to tar files and there are no clean NSPs as you can only dump/CDN download clean NCAs and repack them with unofficial tools. However nobody cares as it's just a container storing the actual files and the only real difference is the order in which the files are stored inside this container. Hash the NCA files instead and you are fine. Hashing NCAs other than for verification like NSZ does using the -V option is quite pointless as they are all already hashed. You find the official NCA hashes inside the CNMT file which the Nintendo Switch and NSZ uses to verify correctness. If you don't want to phrase CNMT: The NCA filename always corresponds to the first half of its SHA256 hash. If you really want NSPs to always have the same hash just repack them with the same tool you packed them in the first place but that's like the most pointless thing to do.

As you might already know NSZ removes completely useless delta fragments NCAs also often called NDV0 files. If you want to keep them, I can add a command storing them uncompressed for you. Or better just use nsZip and trim them to TCA as you can untrimmed them anyway what would save you multiple Terabytes. Please keep in mind that delta fragments have absolutely no use and are only made to save CDN band weight by only containing the modified data fragments between the current and a previous update. Keeping them is useless because you already have the updated NCAs with the applied patches.

In the end NSZ uses a completely lossless compression format and you will keep the exact same NCA files independent of how often you compress/decompress. Use -V to verify correctness while compressing all your games. I definitely would consider NSZ offering archival quality standards.

 

Yes use https://gbatemp.net/threads/ryjin-a-nsz-converter-mod.550174/
ryjin is NSZ just with a batch script packed as an exe containing a menu what to do. The next version of NSZ will most likely come with a GUI so I hope you are already looking forward to it. Alternatively https://github.com/julesontheroad/NSC_BUILDER whould probably be an option too. But keep in mind that command line really isn't that hard. If you download NSZ portable it's just "nsz -C inputFolder --level 22" to compress a full folder with the maximal compression level of level 22.

 

Your issue could be related to https://github.com/nicoboss/nsz/issues/39 which was fixed last week and it’s fix will be in the NSZ 3.0 which will be released later today.
Only strange that it eventually finishes for you while the on skygunner's seem to be stuck forever like in the last week fixed https://github.com/nicoboss/nsz/issues/25 but he also mentioned it occurring for solid compression which clearly contradicts Issue #25 so you probably describe the same issue but just were more patient then skygunner. Please provide as much information as you can like for which game this issue occurs. Because of https://github.com/nicoboss/nsz/issues/31 the process bar is currently bugged for solid multithreaded compression however this won't occur on block compression but still make sure it's not what causes you to think it's stack even though NSZ is working hard just without showing it on the process bar. Monitor disk and CPU activities using the preinstalled Windows Resource Monitor while it's stuck. It could be something disk caching related where Windows is caching everything in RAM and then is forced to write it on your slow HDD wile verifying. Please just try gathering as much information as possible and report back with everything you found out.