RELEASE nsZip - NSP compressor/decompressor to reduce storage

Discussion in 'Switch - Emulation, Homebrew & Software Projects' started by nicoboss, Feb 1, 2019.


    11,597

    80
    Front-page RELEASE
    nsZip is an open source PC tool to lossless compress/decompress NSP files in order to save a lot of storage while all NCA files keep their exact same hash.
    But it’s more than just a compression. A lot of data like NDV0 partitions or fragment NCA3 files will be removed while compressing and exactly recreated when decompressed which saves even more space especially on updates higher than v65536.
    In addition, the NSZ format was designed with emulators in mind so adding NSZ support to Yuzu will be possible in the future and because NSZ contains decrypted NCAs no keys would be needed to only extract game files. As compression algorithm Zstandard is used to multithreaded compress 256 KB chunks while uncompressible chunks are stored uncompressed. That way NSPZ/XCIZ allows random read access. Zstandard has a 43MB/s compression and 7032MB/s decompression speed on an 8 threaded CPU at level 18 while having one of the best compression ratios compared to other compression algorithms.

    How compressing works: NSP => extracted NSP => decrypted NCAs => trim fragments => compressing to NSZ => verify correctness => repacking NSPZ file
    How decompressing works: NSPZ file => extracted NSPZ => decompress NSZ => untrim fragments => encrypt NCAs => verify correctness => repack as NSP

    Check out my GitHub page to report bugs, follow this project, post your suggestions or contribute.
    GitHub link: https://github.com/nicoboss/nsZip


    Version 1.1.0 Changelog:
    - Added support for over multiple NCAs splitted NDV0 fragment trimming

    Version 1.0.1 Changelog:
    - Let the user continue when detecting a yet unimplemented multifile NDV0 fragment. I'll add proper support for that unimplemented multifile NDV0 fragment format in the following days.
    - The compression level can now be changed (tradeoff between speed and compression ratio)
    - The block size can now be changed (tradeoff between random access time and compression ratio)

    Future planes:
    - XCIZ to XCI recreation (Currently working on)
    - Improve performance a lot by not saving every step to HDD
    - Mono builds for cross platform support
    - Yuzu integration
    - Homebrew integration

    FAQ:
    Q: It prod.keys not found!
    A: Dump them using Lockpick and copy them to %userprofile%/.switch/prod.keys
    Q: How much storage does this save?
    A: It depends on the game. The compressed size is usually around 40-80% of the original size.
    Q: What about XCI?
    A: Compression to XCIZ is supported, but XCIZ to XCI recreation is still in development.
    Q: The program throws an error or seems to behaves not as intended!
    A: Open an issue on my GitHub page where you give me exact steps how to reproduce.

    Screenshot:

    nsZip_Screenshot.
     
    Sciguy429, survive9, D3fau4 and 36 others like this.
    Discussion (80 replies)
  1. ScarletDreamz

    ScarletDreamz [Debug Mode]

    Member
    11
    Feb 16, 2015
    United States
    California
    LadyHaywood and nicoboss like this.
  2. MorningBlunt

    MorningBlunt Member

    Newcomer
    2
    Jun 11, 2017
    Can this be done directly from the SD/External HDD?
     
  3. tigersaman

    tigersaman Member

    Newcomer
    2
    Jul 10, 2018
    Iran
    so it's like winzip or winrar but made specifically for Switch files ?
     
  4. nicoboss
    OP

    nicoboss Member

    Newcomer
    2
    Feb 1, 2019
    Switzerland
    Yes, NSZ is a compression format specialized to compress the normally encrypted and incompressible NCA files (the content of NSP/XCI files). Unlike winzip/winrar NSZ allows random read access and so would be able to be directly played on Yuzu and even the Switch if somebody bothers writing a custom sysmodule for this.
     
  5. Oleboy555

    Oleboy555 Wie dit leest is een zemmel

    Member
    5
    Feb 8, 2017
    Netherlands
    Amsterdam
    thats awesome work! thanks for making this :yayswitch:
     
    nicoboss likes this.
  6. tigersaman

    tigersaman Member

    Newcomer
    2
    Jul 10, 2018
    Iran
    So for the sake of argument and since i myself have no nsp at hand to test the program can you give a comparison between a normal nsp and nspz in size of some games ?
    and also thanks alot for creating this format/program.
     
  7. raxadian

    raxadian GBAtemp Regular

    Member
    2
    Nov 10, 2018
    Argentina
    This looks interesting.
     
  8. Clydefrosch

    Clydefrosch GBAtemp Psycho!

    Member
    10
    Jan 2, 2009
    Germany
    so the main use of this will be slimming down romsets and... making it easier to share them online, eh
     
  9. ZachyCatGames

    ZachyCatGames GBAtemp Advanced Fan

    Member
    5
    Jun 19, 2018
    United States
    Hell
    What if someone has NCA2 NSPs /s
     
    Last edited by ZachyCatGames, Feb 1, 2019
  10. nicoboss
    OP

    nicoboss Member

    Newcomer
    2
    Feb 1, 2019
    Switzerland
    VVVVVV Base Game:
    NSP: 80.6 MB
    NSPZ: 55.5 MB

    Qbics Paint v131072:
    NSP: 142 MB
    NSPZ: 46.9 MB

    Qbics Paint v65536:
    NSP: 69.2 MB
    NSPZ: 43.8 MB

    Qbics Paint Base Game:
    NSP: 65.4 MB
    NSPZ: 40.1 MB
     
    Laroon, eyeliner, Girtana1 and 2 others like this.
  11. Essasetic

    Essasetic Your local TX Fanboy

    Member
    7
    Jun 16, 2018
    United Kingdom
    Oh. So XCI trimming but with NSPs. That's cool! I might give it a try if I ever need to get an update or a game.
     
  12. wurstpistole

    wurstpistole Proud TX supporter

    Member
    11
    Nov 19, 2015
    Gambia, The
    Can this further reduce the size of xci as well? That would be neat, if SX then implemented support for xciz as well.
     
  13. Essasetic

    Essasetic Your local TX Fanboy

    Member
    7
    Jun 16, 2018
    United Kingdom
    You can already trim XCIs though.

    — Posts automatically merged - Please don't double post! —

    And also I don't think XCIs have NCAs in them so the program wouldn't work with it.
     
  14. wurstpistole

    wurstpistole Proud TX supporter

    Member
    11
    Nov 19, 2015
    Gambia, The
    I just wondered because in the coming soon section there's "Xciz" mentioned.
     
  15. nicoboss
    OP

    nicoboss Member

    Newcomer
    2
    Feb 1, 2019
    Switzerland
    The only way to really obtain NCA2 files is by restoring improperly deleted factory NCA files on a 1.0.0 console and all of them together are only a few hundred MB. There is barely any person that would even be able to obtain them as only a small amount even owns an untouched 1.0.0 switch. I have my factory NCA2 files dumped but it definitely not worth the efford supporting them with nsZip.

    — Posts automatically merged - Please don't double post! —

    No this isn't trimming but compressing. When trimming an XCI only useless zeros at the end gets removed.

    Yes XCIZ will further reduce the size of a trimmed XCI and TX could implement the format if they like.

    nsZip can already compress XCI to XCIZ but not the opposite way so feels free to try it out if you like. Trimmed XCI compression is as good as NSP compression as they booth are file containers containing NCA files.
     
    Last edited by nicoboss, Feb 1, 2019
  16. ZachyCatGames

    ZachyCatGames GBAtemp Advanced Fan

    Member
    5
    Jun 19, 2018
    United States
    Hell
    Yea, I know it probably isn’t worth the time to support them, I was just joking :P. And 1.0.0 factory titles aren’t the only way to get NCA2s, some pre-1.0.0 dev firmwares and titles are NCA2s
     
    Last edited by ZachyCatGames, Feb 1, 2019
  17. thealgorithm

    thealgorithm GBAtemp Fan

    Member
    4
    Oct 27, 2015
    Swaying from the topic a bit, but related in some way. Would there be a possibility at some stage to convert encrypted nsp's to unencrypted ones and to install the unencrypted nsps (similar to how encrypted or decrypted CIA's can be installed on the 3ds)?
     
  18. Mat37

    Mat37 GBAtemp Advanced Fan

    Member
    4
    Dec 3, 2016
    France
    Does it matter ?
    XCIs have NCAs, but they're not encrypted with the titlekey (this doesn't mean they aren't encrypted at all)
     
  19. nicoboss
    OP

    nicoboss Member

    Newcomer
    2
    Feb 1, 2019
    Switzerland
    As proven in nsZip its possible to reencrypt an nca and get its exact same hash as before decrypting. So, you could just reencrypt it while installing. You can build something like an encryption stream and pass the so encrypted data directly what needs encrypted NCA data. So it really won’t be that hard to make a homebrew supporting NSPZ installing based on the tinfoil/Goldleaf.

    — Posts automatically merged - Please don't double post! —

    The NCAs inside and XCI are AES-CTR encrypted with decrypted key area keys like nearly all non-RomFS NCAs inside an NSP too and are completely supported by nsZip.
     
    Last edited by nicoboss, Feb 1, 2019
    JJTapia19 likes this.
Loading...