Homebrew RELEASE NSZ - Homebrew compatible NSP/XCI compressor/decompressor

[SonikBR]

Thanks! This will save me a lot of time!

 

[nicoboss]

Today NSZ 4.0 got finally released!
https://github.com/nicoboss/nsz/releases/tag/4.0.0

After being in development for over half a year here finally NSZ v4.0 which brings a lot of new features. Enjoy that huge update. I will try to make more but smaller releases in the future.

NSZ 4.0 Changelog:

• Implemented Drag & Drop support as requested in #51
• Implemented CRC32 key validation and added support for future masterkeys
  • Fixed the issue of master_key_0a not being recognized
  • Added CRC32 for master_key_0a
• Windows 7 support for Windows builds
• Improved GUI font size scaling
• Set the NSZ GUI window to be TopMost (always on top) on Windows so Drag & Drop gets much more convenient
  • Added a setting to specify if the Kivy window should be always on top or not
• XCI/XCZ finally extracts to folders containing the NCA/NCZ files instead of HFS0 partition dumps
• Fixed a major XCI compatibility bug by implementing compression/decompression support for NCA files with the first section having a smaller or larger offset then 0x4000. This fixes #49
• Added NSPZ (nsZip legacy file format) extraction support
• Make GUI an optional install
• Fixed #59 ncz decompression is not working
• Cleaned up imports for nsz package
• Stop bar manager to avoid broken shells
• Added pywin32 as GUI dependency for Windows. This fixes #56
• Fixed BlockDecompressorReader.seek with whence = 2 (seek relative to the file's end). This fixes #64
• Starting nsz.py from within a different working directory finally works
  • Fixes the current Azure Pipeline issue
• Added solid decompression, block decompression, solid compression and block compression tests to azure-pipelines.yml
• Made NSZ new returning with error code 1 if there are any exceptions
• Fixed deadlock in BlockCompressor.py
• Highly improved block decompression speed by caching the current block
• Added Visual Studio 2019 Python Project
• Added titleId and version to the file list and highly improved its design
• The SelectableLabel items inside the game list now properly scales its height according to the available width and text length of the file path. This fixes #50
• Added a multi-language supporting open source font for #61
• Implemented input folder as output folder by default for #61
• Waiting for Enter before exit when started over GUI so errors and console output can be seen before it closes for #74
• Fixed install failing on Kubuntu 20.04 and a lot of other modern Linux distributions by improving install_linux.sh for #75
• Removed code that manipulated the XCI header size for absolutely no reason which fixes #77
• Improved decompression speed by 400% by heavily reducing the amount of performance intensive status bar refresh calls
• The decision if the last block should be decompressed or copied now matches the file format specifications by comparing the decompressed block size of that specific block with its compressed size. This issue was caused by missing the (unlikely) edge case that the last block can be larger when compressed without exceeding the general block size. This fixes #79
• Improved BlockCompression performance and overall CPU usage by reducing the amount of performance intensive status bar refresh calls.
• Ensure that the line right to the curser is clean when the application terminates
• Implemented undupe, undupe-dryrun, undupe-prioritylist, undupe-whitelist and undupe-old-versions to remove duplicate games
• Highly improved the missing prod.keys/keys.txt error message by not showing the stack trace and waiting for a user input before exiting
• Highly improved README.md
• General system stability improvements to enhance the user's experience.

 

[Reploid]

Good to see such a project being alive and well.

BTW, do you think there is a real chance that sometime in the future we'll be able to compressed games on switch, not just install them? Kinda like cso format on PSP

 

[nicoboss]

Good to see such a project being alive and well.

BTW, do you think there is a real chance that sometime in the future we'll be able to compressed games on switch, not just install them? Kinda like cso format on PSP

The block compressed NSZ format already supports random read access. In the next major update, a function to mount block compressed NSZ files as NSP on PC is planned. This will allow NSZ to be used where only NSP is supported like for the Yuzu Emulator. Atmosphere plans to reimplement fs in late 2021 or 2022 according to https://github.com/Atmosphere-NX/At...c1a260dcf/docs/roadmap.md#fs-reimplementation. Once that's done modifying it to support launching block compressed NSZ games shouldn't be hard. Ways to achieve this before fs reimplementation seem to be too difficult and require firmware specific patches. If somebody knows how to easy man in the middle every fs call or is interested in working with me on this please let me know. I might look into emuEMMC/LayeredFS but I don't think there will be anything useful there.
My personal interest in launching compressed games directly on the Nintendo Switch is huge so I think it will happen someday especially if Atmosphere ever reimplements fs.

 

[Reploid]

The block compressed NSZ format already supports random read access. In the next major update, a function to mount block compressed NSZ files as NSP on PC is planned. This will allow NSZ to be used where only NSP is supported like for the Yuzu Emulator. Atmosphere plans to reimplement fs in late 2021 or 2022 according to https://github.com/Atmosphere-NX/At...c1a260dcf/docs/roadmap.md#fs-reimplementation. Once that's done modifying it to support launching block compressed NSZ games shouldn't be hard. Ways to achieve this before fs reimplementation seem to be too difficult and require firmware specific patches. If somebody knows how to easy man in the middle every fs call or is interested in working with me on this please let me know. I might look into emuEMMC/LayeredFS but I don't think there will be anything useful there.
My personal interest in launching compressed games directly on the Nintendo Switch is huge so I think it will happen someday especially if Atmosphere ever reimplements fs.

Thanks for the definitive answer. It might take awhile, but better late, than never.

 

[nicoboss]

NSZ 4.0.1 release:
https://github.com/nicoboss/nsz/releases/tag/4.0.1

For the users that had issues with my latest release being wrongly flagged by Windows Defender here another version that completely replaced PyInstaller with WinPython. This version only affects the portable Windows release. The PIP release wasn't updated as only the deployment of the Portable Windows build is affected by this change.

NSZ 4.0.1 Changelog:

• Replaced PyInstaller with WinPython for the portable Windows release.
  • I spent a lot of time deleting every piece of the python standard library not required to run nsz in order to save storage and speed up extraction time by having less files to extract.
  • This was done because PyInstaller triggered false positive detections on Windows Defender as reported in #82
  • Tested on Windows 7, Windows 10 and Windows Server 2019

 

[mathew77]

The block compressed NSZ format

Hi, and thanks for that!.. Could you please tell which homebrew software can install block compressed (not solid ones) .NSZ files for now into the Switch?..

 

[Badablek]

Hi, and thanks for that!.. Could you please tell which homebrew software can install block compressed (not solid ones) .NSZ files for now into the Switch?..

Tinfoil 6+



Awoo Installer only supports solid NSZ
Goldleaf does not support NSZ at all

 

[mathew77]

XCZ

Just tried to compress XCI to XCZ via the latest NSZ 4.0.1, but have some errors and the corrupted file:

Spoiler

Unconfirmed: crc32(master_key_10) = 2012895168
[INFO   ] [Logger      ] Record log in C:\home\.kivy\logs\kivy_20-10-05_3.txt
[INFO   ] [deps        ] Successfully imported "kivy_deps.glew" 0.1.12
[INFO   ] [deps        ] Successfully imported "kivy_deps.sdl2" 0.1.22
[INFO   ] [Kivy        ] v1.11.1
[INFO   ] [Kivy        ] Installed at "C:\Program Files\nsz_v4.0.1_win64_portabl
e\lib\site-packages\kivy\__init__.py"
[INFO   ] [Python      ] v3.7.7 (tags/v3.7.7:d7c567b08f, Mar 10 2020, 10:41:24)
[MSC v.1900 64 bit (AMD64)]
[INFO   ] [Python      ] Interpreter at "C:\Program Files\nsz_v4.0.1_win64_porta
ble\python.exe"
[INFO   ] [Factory     ] 184 symbols loaded
[INFO   ] [Image       ] Providers: img_tex, img_dds, img_sdl2, img_gif (img_pil
, img_ffpyplayer ignored)
[INFO   ] [Window      ] Provider: sdl2
[INFO   ] [GL          ] Using the "OpenGL" graphics system
[INFO   ] [GL          ] GLEW initialization succeeded
[INFO   ] [GL          ] Backend used <glew>
[INFO   ] [GL          ] OpenGL version <b'4.0.0 - Build 10.18.10.5129'>
[INFO   ] [GL          ] OpenGL vendor <b'Intel'>
[INFO   ] [GL          ] OpenGL renderer <b'Intel(R) HD Graphics 4000'>
[INFO   ] [GL          ] OpenGL parsed version: 4, 0
[INFO   ] [GL          ] Shading version <b'4.00 - Build 10.18.10.5129'>
[INFO   ] [GL          ] Texture max size <16384>
[INFO   ] [GL          ] Texture max units <16>
[INFO   ] [Window      ] auto add sdl2 input provider
[INFO   ] [Window      ] virtual keyboard not allowed, single mode, not docked
[INFO   ] [Text        ] Provider: sdl2
[INFO   ] [Base        ] Start application main loop
[INFO   ] [GL          ] NPOT texture support is available
[INFO   ] [Base        ] Leaving application in progress...
[INFO   ] [WindowSDL   ] exiting mainloop and closing.

             NSZ v4.0   ,;:;;,
                       ;;;;;
               .=',    ;:;;:,
              /_', "=. ';:;:;
              @=:__,  \,;:;:'
                _(\.=  ;:;;'
               `"_(  _/="`
                `"'

Block compressing (level 22) \(xci uncompr
essed) Crocs World Run [0100EC400D712000][v0] (0.59 GB).xci -> (xci uncompressed) Crocs World Run [0100EC400D712000][v0]
(0.59 GB).xcz
[ADDING]     secure 0 bytes to NSP
[ADDING]     9b51a207e581c96a9c3dbe13dc2f6b49.ncz 616382464 bytes to NSP
Compressing   0%|                            |   0/587 MiB [00:00<?, 0.00 MiB/s]
Compressing   2%|▌                      |  10/587 MiB [00:00<00:19, 31.05 MiB/s]
Compressing   4%|█                      |  21/587 MiB [00:01<00:16, 35.00 MiB/s]
Compressing   5%|█▌                     |  32/587 MiB [00:02<00:31, 17.98 MiB/s]
Compressing   7%|█▌                     |  43/587 MiB [00:03<00:33, 16.26 MiB/s]
Compressing   9%|██                      |  54/587 MiB [00:06<00:55, 9.63 MiB/s]
Compressing  11%|██▌                     |  65/587 MiB [00:07<00:54, 9.75 MiB/s]
Compressing  13%|███                    |  76/587 MiB [00:07<00:50, 10.25 MiB/s]
Compressing  15%|███▌                    |  87/587 MiB [00:09<00:51, 9.88 MiB/s]
Compressing  17%|████                   |  98/587 MiB [00:09<00:47, 10.33 MiB/s]
Compressing  19%|████▌                  | 109/587 MiB [00:10<00:44, 10.85 MiB/s]
Compressing  20%|████▌                  | 120/587 MiB [00:11<00:44, 10.68 MiB/s]
Compressing  22%|█████                  | 131/587 MiB [00:12<00:40, 11.39 MiB/s]
Compressing  24%|█████▌                 | 142/587 MiB [00:12<00:37, 11.87 MiB/s]
Compressing  26%|██████                 | 153/587 MiB [00:13<00:37, 11.68 MiB/s]
Compressing  28%|██████▌                | 164/587 MiB [00:15<00:39, 10.97 MiB/s]
Compressing  30%|███████                | 175/587 MiB [00:16<00:39, 10.62 MiB/s]
Compressing  32%|███████▌               | 186/587 MiB [00:18<00:39, 10.18 MiB/s]
Compressing  34%|████████                | 197/587 MiB [00:20<00:39, 9.92 MiB/s]
Compressing  35%|████████▌               | 208/587 MiB [00:22<00:39, 9.65 MiB/s]
Compressing  37%|████████▌              | 219/587 MiB [00:22<00:36, 10.12 MiB/s]
Compressing  39%|█████████              | 230/587 MiB [00:22<00:35, 10.32 MiB/s]
Compressing  41%|██████████              | 241/587 MiB [00:25<00:36, 9.68 MiB/s]
Compressing  43%|██████████▌             | 252/587 MiB [00:25<00:34, 9.95 MiB/s]
Compressing  45%|███████████             | 263/587 MiB [00:27<00:33, 9.84 MiB/s]
Compressing  47%|███████████             | 274/587 MiB [00:28<00:32, 9.93 MiB/s]
Compressing  49%|███████████▌            | 285/587 MiB [00:31<00:33, 9.22 MiB/s]
Compressing  50%|████████████            | 296/587 MiB [00:33<00:32, 9.06 MiB/s]
Compressing  52%|████████████▌           | 307/587 MiB [00:34<00:31, 9.01 MiB/s]
Compressing  54%|█████████████           | 318/587 MiB [00:35<00:30, 9.05 MiB/s]
Compressing  56%|█████████████▌          | 329/587 MiB [00:37<00:29, 8.83 MiB/s]
Compressing  58%|██████████████          | 340/587 MiB [00:39<00:28, 8.78 MiB/s]
Compressing  60%|██████████████▌         | 351/587 MiB [00:42<00:28, 8.45 MiB/s]
Compressing  62%|███████████████         | 362/587 MiB [00:45<00:28, 8.08 MiB/s]
Compressing  64%|███████████████▌        | 373/587 MiB [00:46<00:26, 8.18 MiB/s]
Compressing  65%|███████████████▌        | 384/587 MiB [00:47<00:25, 8.12 MiB/s]
Compressing  67%|████████████████        | 395/587 MiB [00:49<00:24, 8.12 MiB/s]
Compressing  69%|████████████████▌       | 406/587 MiB [00:53<00:23, 7.72 MiB/s]
Compressing  71%|█████████████████       | 417/587 MiB [00:53<00:22, 7.87 MiB/s]
Compressing  73%|█████████████████▌      | 428/587 MiB [00:55<00:20, 7.85 MiB/s]
Compressing  75%|██████████████████      | 439/587 MiB [00:56<00:19, 7.78 MiB/s]
Compressing  77%|██████████████████▌     | 450/587 MiB [00:57<00:17, 7.90 MiB/s]
Compressing  79%|███████████████████     | 461/587 MiB [00:59<00:16, 7.78 MiB/s]
Compressing  80%|███████████████████▌    | 472/587 MiB [01:01<00:15, 7.68 MiB/s]
Compressing  82%|███████████████████▌    | 483/587 MiB [01:04<00:14, 7.54 MiB/s]
Compressing  84%|████████████████████    | 494/587 MiB [01:06<00:12, 7.48 MiB/s]
Compressing  86%|████████████████████▌   | 505/587 MiB [01:07<00:11, 7.49 MiB/s]
Compressing  88%|█████████████████████   | 516/587 MiB [01:09<00:09, 7.49 MiB/s]
Compressing  90%|█████████████████████▌  | 527/587 MiB [01:10<00:08, 7.54 MiB/s]
Compressing  92%|██████████████████████  | 538/587 MiB [01:11<00:07, 7.53 MiB/s]
Compressing  94%|██████████████████████▌ | 549/587 MiB [01:13<00:05, 7.55 MiB/s]
Compressing  95%|███████████████████████ | 560/587 MiB [01:15<00:04, 7.50 MiB/s]
Compressing  97%|███████████████████████▌| 571/587 MiB [01:18<00:02, 7.34 MiB/s]
Compressing  99%|████████████████████████| 582/587 MiB [01:18<00:01, 7.42 MiB/s]
Compressing 100%|████████████████████████| 587/587 MiB [01:20<00:00, 7.35 MiB/s]
compressed 7% 616382464 -> 44020735  - 9b51a207e581c96a9c3dbe13dc2f6b49.nca
[ADDING]     d954fc7ef08e6141decc628cc6c94ad7.nca 822784 bytes to NSP
[ADDING]     a9a086d259582f66f388fa51a21ff45f.nca 110080 bytes to NSP
[ADDING]     8a0dbc39bc702ae59a5a11e79561f64a.cnmt.nca 3584 bytes to NSP
[VERIFY NSZ] \(xci uncompressed) Crocs Wor
ld Run [0100EC400D712000][v0] (0.59 GB).xcz
[NCZBLOCK]   Using Block decompresion
Decompress   0%|                             |   0/587 MiB [00:00<?, 0.00 MiB/s]
Decompress   9%|██                     |  50/587 MiB [00:00<00:05, 103.73 MiB/s]
Decompress  17%|████                   | 100/587 MiB [00:01<00:05, 103.19 MiB/s]
Decompress  26%|██████                 | 150/587 MiB [00:01<00:04, 104.16 MiB/s]
Decompress  34%|████████               | 200/587 MiB [00:02<00:04, 104.93 MiB/s]
Decompress  43%|██████████             | 250/587 MiB [00:02<00:03, 105.21 MiB/s]
Decompress  51%|████████████           | 300/587 MiB [00:03<00:03, 105.07 MiB/s]
Decompress  60%|█████████████▌         | 350/587 MiB [00:03<00:02, 104.88 MiB/s]
Decompress  68%|███████████████▌       | 400/587 MiB [00:04<00:02, 104.82 MiB/s]
Decompress  77%|█████████████████▌     | 450/587 MiB [00:04<00:01, 104.43 MiB/s]
Decompress  85%|███████████████████▌   | 500/587 MiB [00:05<00:01, 103.79 MiB/s]
Decompress  94%|█████████████████████▌ | 550/587 MiB [00:05<00:00, 103.24 MiB/s]
Decompress 100%|███████████████████████| 587/587 MiB [00:06<00:00, 102.06 MiB/s]
[CORRUPTED]  9b51a207e581c96a9c3dbe13dc2f6b49.ncz
Error while compressing file: \(xci uncomp
_.py", line 54, in compress
    verify(outFile, True)
  File "C:\Program Files\nsz_v4.0.1_win64_portable\lib\site-packages\nsz\__init_
_.py", line 65, in verify
    NszVerify(filePath, raiseVerificationException, statusReportInfo, pleaseNoPr
int)
  File "C:\Program Files\nsz_v4.0.1_win64_portable\lib\site-packages\nsz\NszDeco
mpressor.py", line 46, in verify
    __decompressXcz(filePath, None, False, raiseVerificationException, statusRep
ortInfo, pleaseNoPrint)
  File "C:\Program Files\nsz_v4.0.1_win64_portable\lib\site-packages\nsz\NszDeco
mpressor.py", line 223, in __decompressXcz
    __decompressContainer(secureIn, None, fileHashes, write, raiseVerificationEx
ception, statusReportInfo, pleaseNoPrint)
  File "C:\Program Files\nsz_v4.0.1_win64_portable\lib\site-packages\nsz\NszDeco
mpressor.py", line 88, in __decompressContainer
    raise Exception("Verification detected hash missmatch")
Exception: Verification detected hash missmatch


Done!



Press Enter to exit...

Any advice?

Tinfoil 6+

Thanks!

 

[Presjar]

Hi!
If I drag an NSP intro the UI window and it doesn't show up, does that mean I am missing some keys in my keys.text file?
I'm able to compress older games, like BotW but Crysis Remastered has the following error in the console window. Many other NSP I have that are newer also have the same error.

Error while adding \\192.168.2.10\DownloadTemp\Nintendo Switch Roms\Crysis Remastered\Crysis Remastered.nsp to gamelist: cannot unpack non-iterable NoneType object
Traceback (most recent call last):
File "D:\Switch\Tools\nsz_v4.0.1_win64_portable\lib\site-packages\nsz\gui\GameList.py", line 127, in handledrops
(titleIDExtracted, versionExtracted) = FileExistingChecks.ExtractTitleIDAndVersion(fullPath, True)
TypeError: cannot unpack non-iterable NoneType object

 

[Arthas91]

Hi, Is there a guide anywhere on how to use this on Mac OS. I would really like to use this to compress nsp to nsz. Any help is very appreciated. Thank you.

 

[Stellar]

The GUI does nothing but crash for me. It will not load input files or folders.

 

[Jacoghi]

I really appreciate the time you guys put into this, the compression is awesome

 

[Naguz]

Is there any way to get back something similar to the original nsp file, chhecksum-wise? Like what rescene does for rar files.

 

[MaDTi]

Error with xci files that have an update or dlc. Example - pokemon sword+expansion, super mario odyssey ver. 3.
And request for future releases to add the option to save the update partition and untrim FF after xcz->xci - 1:1 lossless xci-xsz-xci. The purpose of archiving files only.

 

[BLUE_FIRE]

Hi!

@nicoboss I've just registered because I'm using your compression tool but you state thate it's a lossless compressor/decompressor and I'm getting different hashes for my .nsp files before compressing them and after compressing them, how is that possible?

I've just read this response #11 from this thread.
Thanks for writing the tool.

 

[mathew77]

@nicoboss
So now we need a way to compress these Sparse storage games (Last Stop, Lost in Random, Rainbow Billy The Curse of the Leviathan).

 

[nicoboss]

Hi, Is there a guide anywhere on how to use this on Mac OS. I would really like to use this to compress nsp to nsz. Any help is very appreciated. Thank you.

Install Python 3 then enter "pip3 install --upgrade nsz" in the terminal. If this doesn't work download the Repository and execute "pip3 install -r requirements.txt" in the Terminal while being in that folder and then just use nsz like "python nsz.py --help"

Is there any way to get back something similar to the original nsp file, chhecksum-wise? Like what rescene does for rar files.

Hi!

@nicoboss I've just registered because I'm using your compression tool but you state thate it's a lossless compressor/decompressor and I'm getting different hashes for my .nsp files before compressing them and after compressing them, how is that possible?

I've just read this response #11 from this thread.
Thanks for writing the tool.

As described in #11 the NCA files are hash identical. The only thing that makes the NSP files to sometimes have a different hash is the order in which you pack the NCA files into an NSP container. If you want your NSP to be hash identical just make sure to pack the files in the same order. This however is absolutely pointless. Just hash the NCA files instead which are already are sha256 hashed and signed. NCAs not being hash identical would break the signature and file integrity check so obviously they are.

@nicoboss
So now we need a way to compress these Sparse storage games (Last Stop, Lost in Random, Rainbow Billy The Curse of the Leviathan).

Keep an eye on https://github.com/nicoboss/nsz/issues/100. It will likely be implemented soon.

 

[Jacoghi]

Install Python 3 then enter "pip3 install --upgrade nsz" in the terminal. If this doesn't work download the Repository and execute "pip3 install -r requirements.txt" in the Terminal while being in that folder and then just use nsz like "python nsz.py --help"



As described in #11 the NCA files are hash identical. The only thing that makes the NSP files to sometimes have a different hash is the order in which you pack the NCA files into an NSP container. If you want your NSP to be hash identical just make sure to pack the files in the same order. This however is absolutely pointless. Just hash the NCA files instead which are already are sha256 hashed and signed. NCAs not being hash identical would break the signature and file integrity check so obviously they are.


Keep an eye on https://github.com/nicoboss/nsz/issues/100. It will likely be implemented soon.

Looking forward to this!

 

[zxr750j]

I've got a folder of old NSPZ files i'd like to convert back to nsp, but they don't show up in the GUI. I read they should be supported?