RELEASE NSZ - Homebrew compatible NSP/XCI compressor/decompressor

Discussion in 'Switch - Emulation, Homebrew & Software Projects' started by nicoboss, Oct 20, 2019.

  1. nicoboss
    OP

    nicoboss Advanced Member

    Newcomer
    5
    Feb 1, 2019
    Switzerland
    NSZ is an open source MIT licensed python script to lossless compress/decompress NSP files in order to save a lot of storage while all NCA files keep their exact same hash. The NSZ file format for compressed Nintendo Switch games is already widely adopted and supported by popular homebrews like Tinfoil and SX Installer v3.0.0.

    Currently it improves size, speed and bandwidth used for storing and transferring NSZ files while still leading to the same size when installed but integration into Yuzu and CFW to directly play compressed games is planned. NSZ's developments started by blawar based on the idea of nsZip and its NSPZ/XCIZ file format which unfortunately was too complex to be implemented by homebrew developers. I very soon took over his project and am closely working together with all homebrew developers interested in the NSZ format.

    How to install:
    Put dumped prod.keys to %userprofile%/.switch, install python, execute "pip install nsz" and use "nsz" like every other cmd command.

    You can get the latest release of NSZ under https://github.com/nicoboss/nsz/releases
    To read the readme and take a look at its source code visit https://github.com/nicoboss/nsz

    Results how well NSZ performs: https://gbatemp.net/threads/nsz-title-compression-results.549831/
    An easy to use tool for NSZ compression: https://github.com/julesontheroad/NSC_BUILDER
    Anoter tool for NSZ compression: https://gbatemp.net/threads/ryjin-a-nsz-converter-mod.550174/


    Changelog:

    NSZ 2.1.1:
    • Fixed block compression for pip and Nuitka (nsz_win64_portable) by using sys.argv[0] instead of __main__.__file__ so threads no longer need to be prevented from initializing their own nut environment
    • Made installation instructions easier to see and understand
    • Scripts to automate testing and publishing

    NSZ 2.1:
    • 98 commits worth of changes since v2.0
    • Added pip support
    • Added Nuitka standalone windows build support
    • Enhanced File Existing Check #20
      • Skip already compressed/decompressed files by default
      • --overwrite
      • --rm-old-version
      • Extracting TitleIDs and Versions from filename if possible (#17 and #19)
        • The titleID checking is now immensely faster than when extracting from Cnmt
      • --parseCnmt to get TitleID/version from Cnmt if not extractable from filename
        • Otherwise it falls back to simple filename checking which is much faster
    • Batch error handling with tracebacks (#16)
      • Some debugging codes to find out erroneous files
      • Prevent batch process raising errors
      • Batch error handling with tracebacks
    • The --thread option now works even for solid compression however the progress bar still has some visual glitches
    • NSP/NSZ file hash verification now uses the hashes inside Cnmt instead of the nca filename (#22)
    • Fixed decompression memory leak (#21)
      • The memory leak only occurs for dctx.stream_reader and was fixed by switching to the simple decompressing API which makes more sense for block decompression anyways
    • Improved pageReadSize calculation speed by using math instead of a while loop
    • Added option --remove-source that deletes the source file after compression or decompression (#24)
      • For this we finally properly closed file containers too
    • Fix huge compression memory leak (#13)
    • Fixed wrong working directory when starting nut.py from a different directory (#18)
    • Improved exception handling during TitleID/Version extraction
      • Added support for prod.keys
    • Reorganized file structure
    • Fixed keys.txt path to always be the folder containing nsz.py
    • Fixed default thread amount to be cpu_count()
    • Implemented python version checking to prevent python from showing the users confusing compatibility related exceptions
      • Compatible and tested with Python 3.6 and later
    • General system stability improvements to enhance the user's experience.

    NSZ 2.0:
    • Fully implemented block compression which can be enabled using the --block option
      • Supports for random read access on compressed files
      • Highly multithreaded compression when block compressing
      • Technically supports playing compressed games in the future
      • Current title installers do not support this yet
      • Comes with a low compression ratio cost
    • Implemented NSP/NSZ file hash verification
    • Overwrite/Duplicate protection
    • Reorganized the project's folder structure and enhanced code readability
    • Improved user feedback in form of better understandable messages and errors
    • Fixed a lot of bugs and non-working features
    • Added MIT License so all code inside this project can be used for whatever you like
    • General system stability improvements to enhance the user's experience.
    NSZ 1.0:
    • Scripts to compress and decompress NSZ files.

    Differences between NSZ and NSPZ:

    NSZ/XCZ:
    - GitHub Project: https://github.com/nicoboss/nsz
    - Uses solid compression by default. Block compression can be enabled using the -B option. Block compression will be the default for XCZ
    - Decrypts all sections while keeping the first 0x4000 bytes encrypted. Puts information needed to encrypt inside the header.
    - Deleted NDV0 fragments as they have no use for end users as they only exist to save CDN bandwidth
    - Already widely used. Supported by Tinfoil, SX Installer v3.0.0 and probably a lot of other software in the future

    NSPZ/XCIZ:
    - GitHub Project: https://github.com/nicoboss/nsZip
    - Always uses Block compression allowing random read access to play compressed games in the future
    - Decrypts the whole NCA
    - Trims NDV0 fragments to their header and reconstructs them
    - Only supported by nsZip and unfortunately doesn't really have a future
     
    Last edited by nicoboss, Nov 14, 2019
  2. eman_not_ava

    eman_not_ava Member

    Newcomer
    2
    Nov 7, 2015
    Netherlands
    probably the billionth person to ask, but:
    can you also convert a nsz back to xci, cause I'm trying to extract game contents for which I need an xci
     
  3. nicoboss
    OP

    nicoboss Advanced Member

    Newcomer
    5
    Feb 1, 2019
    Switzerland
    NSZ is a compressed NSP and XCZ a compressed XCI. NSZ is a lossless compression format so you can convert back and forth between NSP and NSZ without losing any data. To decompress an NSZ file back to an NSP just use https://github.com/nicoboss/nsz with "python nsz.py -D YourCompressedGame.nsz"
     
    Last edited by nicoboss, Nov 13, 2019
  4. eman_not_ava

    eman_not_ava Member

    Newcomer
    2
    Nov 7, 2015
    Netherlands
    I'll have to look into that as I do have python installed but I haven't the fuggiest how to use it.

    But now that I think about it, can I use this tool to unpack a NSZ file directly? Or do I have to convert it to NSP then to XCI just to unpack it?
     
  5. duckbill007

    duckbill007 Advanced Member

    Newcomer
    5
    May 5, 2011
    Russia
    No, You can not use this tool to unpack nsz. Only nsz->nsp
     
  6. eman_not_ava

    eman_not_ava Member

    Newcomer
    2
    Nov 7, 2015
    Netherlands
    Would be a nice feature, although not necessary it would be nice nonetheless.

    point is that I have a game in NSZ format installed on my switch, and I want to extract the RomFS contents from it, so either I have to convert that NSZ to NSP and unpack that or I have to re-dump the game on my switch to a XCI format and unpack that.

    Both are a bit lengthy processes, so that's why I hoped I could do it with this directly.

    Well, no matter, I guess I'll go the long way around
     
  7. nicoboss
    OP

    nicoboss Advanced Member

    Newcomer
    5
    Feb 1, 2019
    Switzerland
    Today NSZ 2.1 got finally released!

    NSZ 2.1.1:
    • Fixed block compression for pip and Nuitka (nsz_win64_portable) by using sys.argv[0] instead of __main__.__file__ so threads no longer need to be prevented from initializing their own nut environment
    • Made installation instructions easier to see and understand
    • Scripts to automate testing and publishing

    NSZ 2.1:
    • 98 commits worth of changes since v2.0
    • Added pip support
    • Added Nuitka standalone windows build support
    • Enhanced File Existing Check #20
      • Skip already compressed/decompressed files by default
      • --overwrite
      • --rm-old-version
      • Extracting TitleIDs and Versions from filename if possible (#17 and #19)
        • The titleID checking is now immensely faster than when extracting from Cnmt
      • --parseCnmt to get TitleID/version from Cnmt if not extractable from filename
        • Otherwise it falls back to simple filename checking which is much faster
    • Batch error handling with tracebacks (#16)
      • Some debugging codes to find out erroneous files
      • Prevent batch process raising errors
      • Batch error handling with tracebacks
    • The --thread option now works even for solid compression however the progress bar still has some visual glitches
    • NSP/NSZ file hash verification now uses the hashes inside Cnmt instead of the nca filename (#22)
    • Fixed decompression memory leak (#21)
      • The memory leak only occurs for dctx.stream_reader and was fixed by switching to the simple decompressing API which makes more sense for block decompression anyways
    • Improved pageReadSize calculation speed by using math instead of a while loop
    • Added option --remove-source that deletes the source file after compression or decompression (#24)
      • For this we finally properly closed file containers too
    • Fix huge compression memory leak (#13)
    • Fixed wrong working directory when starting nut.py from a different directory (#18)
    • Improved exception handling during TitleID/Version extraction
      • Added support for prod.keys
    • Reorganized file structure
    • Fixed keys.txt path to always be the folder containing nsz.py
    • Fixed default thread amount to be cpu_count()
    • Implemented python version checking to prevent python from showing the users confusing compatibility related exceptions
      • Compatible and tested with Python 3.6 and later
    • General system stability improvements to enhance the user's experience.
     
  8. Rahkeesh

    Rahkeesh GBAtemp Maniac

    Member
    7
    Apr 3, 2018
    United States
    I'm confused on the current status of block compression. Does this actually decrease the installed size of NSZs? Or is it more for XCZ users?
     
    Last edited by Rahkeesh, Nov 16, 2019
  9. nicoboss
    OP

    nicoboss Advanced Member

    Newcomer
    5
    Feb 1, 2019
    Switzerland
    Block compressed NSZ/XCZ are compressed in chunks what makes them mountable. XCI files are known to be the mountable format so XCZ will use block compression by default while NSZ uses solid compression by default.

    Block compressed NSZ/XCI mounting for PC will probably be released in the next few weeks. This will allow playing compressed games directly on emulator and use them with any tools not supporting NSZ/XCZ without decompression them first. When it will come to real hardware depends when and if Blawar manages to find somebody helping him to write a game mounting system model or if SX will implement XCZ support but that somebody will do it in the future seems quite likely.

    Tinfoil currently already supports installing block compressed NSZ files but when doing so there's no benefit over solid NSZ. In the long-term block compressed NSZ seems to be better than solid compressed ones but if you never plan to mount your games there's no reason to use it as it comes with around 3% of compression ratio cost.
     
    Last edited by nicoboss, Nov 16, 2019
    Rahkeesh likes this.
  10. wherehere

    wherehere Newbie

    Newcomer
    1
    Nov 19, 2019
    Canada
    Is it possible to get a bit-identical NSP when compressing/decompressing? I'm using a dat file to keep track of my NSPs, and it'd be great to use NSZ to save space, but the hashes change when going from NSP -> NSZ -> recreated NSP. I'm hoping for some ability to make it an archival quality compression like CHD where you can get exactly the original data back and preserve hashes.
     
    Last edited by wherehere, Nov 19, 2019
  11. nicoboss
    OP

    nicoboss Advanced Member

    Newcomer
    5
    Feb 1, 2019
    Switzerland
    The NCAs are bit-identical. NSP is just a container similar to tar files and there are no clean NSPs as you can only dump/CDN download clean NCAs and repack them with unofficial tools. However nobody cares as it's just a container storing the actual files and the only real difference is the order in which the files are stored inside this container. Hash the NCA files instead and you are fine. Hashing NCAs other than for verification like NSZ does using the -V option is quite pointless as they are all already hashed. You find the official NCA hashes inside the CNMT file which the Nintendo Switch and NSZ uses to verify correctness. If you don't want to phrase CNMT: The NCA filename always corresponds to the first half of its SHA256 hash. If you really want NSPs to always have the same hash just repack them with the same tool you packed them in the first place but that's like the most pointless thing to do.

    As you might already know NSZ removes completely useless delta fragments NCAs also often called NDV0 files. If you want to keep them, I can add a command storing them uncompressed for you. Or better just use nsZip and trim them to TCA as you can untrimmed them anyway what would save you multiple Terabytes. Please keep in mind that delta fragments have absolutely no use and are only made to save CDN band weight by only containing the modified data fragments between the current and a previous update. Keeping them is useless because you already have the updated NCAs with the applied patches.

    In the end NSZ uses a completely lossless compression format and you will keep the exact same NCA files independent of how often you compress/decompress. Use -V to verify correctness while compressing all your games. I definitely would consider NSZ offering archival quality standards.
     
    Last edited by nicoboss, Nov 19, 2019
  12. Clydefrosch

    Clydefrosch GBAtemp Guru

    Member
    12
    Jan 2, 2009
    Germany
    being a noob that never quite understood commandlines, has anyone made like a batchfile or something for easy compression or decrompression?
     
  13. nicoboss
    OP

    nicoboss Advanced Member

    Newcomer
    5
    Feb 1, 2019
    Switzerland
    Yes use https://gbatemp.net/threads/ryjin-a-nsz-converter-mod.550174/
    ryjin is NSZ just with a batch script packed as an exe containing a menu what to do. The next version of NSZ will most likely come with a GUI so I hope you are already looking forward to it. Alternatively https://github.com/julesontheroad/NSC_BUILDER whould probably be an option too. But keep in mind that command line really isn't that hard. If you download NSZ portable it's just "nsz -C inputFolder --level 22" to compress a full folder with the maximal compression level of level 22.
     
  14. Clydefrosch

    Clydefrosch GBAtemp Guru

    Member
    12
    Jan 2, 2009
    Germany
    not sure why but the ryjin download doesn't seem to really start for me.
    but I'll just wait for a gui to come along, thanks :)
     
  15. dujdujduj

    dujdujduj Newbie

    Newcomer
    1
    Wednesday
    United States
    Code:
    81% 30075486208 -> 24518742738  - 049dd08d818a01355283840a031a5922.nca
    I tested the level 22 block compression on Witcher 3. It compressed the 28.0 GB nsp to a 22.8 GB nsz. That's not bad at all. It leaves more than enough space to install the high-quality audio mod, for example. I look forward to being able to save a lot of space on my SD card. Even Nintendo should be jealous; with compression several of the games they printed on 16 GB cartridges could have fit on 8 GB carts instead, saving them money.
     
    Last edited by dujdujduj, Dec 6, 2019 at 3:44 PM
Quick Reply
Draft saved Draft deleted
Loading...