Homebrew RELEASE NSZ - Homebrew compatible NSP/XCI compressor/decompressor

[nicoboss]

Whole folder selection doesn't seem to be working in win portable 3.1.1? I'm compressing NSP to NSZ via solid, selected the input folder and specified an output folder, run it and the console window shows arguments have been parsed but it just sits there doing nothing. If I select a few single files then everything works correctly.

I was able to reproduce this bug. Should be fixed in the next release which will be in around 3 weeks. 2 weeks until my university exams are over and around 1 week for preparing another release - currently latest commit doesn't even work.

@nicoboss nice update. Haven't been active for a while. I tested your latest release and I have a few suggestion.
1. Is it possible that after the convertion the menu would return or the program would exit after a couple of seconds?
2. An option where the default input and output folder can be set to speed up the process everytime the app is executed.
Thanks!

1. Yes great idea but maybe even better would be not to kill the GUI at all and instead spawn a new independent process for every task so the user can already prepare the next task while the current one is still running and even start them in parallel.
2. Great idea too. I will probably just set the last chosen folder as default like most applications do. As you might have noticed since the latest release Drag & Drop support was added so I don't really expect many to continue using the open File/Folder dialog anyways.

 

[Tr10]

What's the default output folder for nsz to nsp conversion?

 

[mathew77]

Hbg said: "Latest version of the nicoboss script is borked. Use the latest good one instead".
Can you please comment @nicoboss, is it safe to use their NSZ script, or there's a some Embrace, extend and extinguish strategy from them?

 

[nicoboss]

What's the default output folder for nsz to nsp conversion?

The default folder is the location from where you call nsz. For example, if you open a command line window and your current directory is "C:\Users\Administrator" and you call nsz this will also be the default output folder. Same should apply for booth CLI and GUI. If you use the portable version and open nsz.exe the default folder will be the same where nsz.exe is located.

Hbg said: "Latest version of the nicoboss script is borked. Use the latest good one instead".
Can you please comment @nicoboss, is it safe to use their NSZ script, or there's a some Embrace, extend and extinguish strategy from them?

Yes as long you use the -V option using nsz is always safe no matter what build you use as this guarantees that decompressing the compressed .nsz file will produce bit equivalent NCA files.

Just use latest release as latest commit indeed is broken but that's kind of expected if you use work in progress code. I'm committing directly to master so never trust latest master even though it works almost always perfectly fine. There is a reason why there are releases. If I feel my code is stable enough for real world, I will test it and if everything seems fine create a new release. As mentioned before: Even if you use a broken version or encounter or experience any bug the -V option will throw a verification exception in that case so as long you are enabling verification you are perfectly safe even if you use latest commit but as most/all games would just error while compressing if something is broken.

For NSP using nsz is perfectly fine and very well tested. For XCI please wait until next release as most NCA files inside an XCI can't be compressed with latest release as they use different offsets for the first section which I fixed in master but currently isn't considered stable for real world use. This change is also what made master containing a lot of bugs as I rushed that change just before my university exams started and since then wasn't able to properly finish it. Now that my exams are finally over, I will finish it during the next week and make another release with XCI compression fully working.

I don't recommend using any unofficial forks and don't have time to review their safety. Take a look at them by your own if you like. Blawar is a contributor of nsz and also in /hbg/ so I assume using their fork should be safe but never reviewed or tested it. Honestly just use latest release with verification instead or wait another week for NSZ 3.2.

 

[Tuxie]

Is it theoretically possible to completely restore a .nsz to the original .nsp file, or .xcz to the original .xci file, so that hashes match? I only tried two random games and neither nsp->nsz->nsp nor xci->xcz->xci produced the same original files.

 

[nicoboss]

Is it theoretically possible to completely restore a .nsz to the original .nsp file, or .xcz to the original .xci file, so that hashes match? I only tried two random games and neither nsp->nsz->nsp nor xci->xcz->xci produced the same original files.

The NCA hashes are always exactly the same even after compressing and decompressing. NSP/NSZ and XCI/XCZ is just a Filesystem like fat32 where the NCA/NCZ files are stored. You could copy the exact same files on your external HDD multiple times and still the files system would never hash match as it's undefined behavior where inside the physical storage which files is stored. Same applies for PFS (Partitioned file system) (NSP/NSZ) and HFS (Hashed file system) (XCI/XCZ). There is no such thing like clean NSP files as all NSPs we can obtain are repacked from NCAs even if downloaded from official store and so pseudo random. It doesn't matter how files are stored inside a file system as long all files inside that file system are hash equivalent. You don't care how your files are physically stored on your USB device as long all files contain the exact same hash they had before copying them. NSP files aren’t even signed in any ways as they don't really exist outside the official SDK where they also include a lot of additional files, we would never include in our recreated NSPs as they are redundant. All NSP files out there will never match with the original one submitted to Nintendo and trying to recreate the pseudo random way the real SDK stores them is useless and even if I do so I still wouldn't be bit equivalent with any other NSP files out there. I just store the files inside the NSP as the operating system lists them when requesting every entry in a dictionary which I guess is the same way the official SDK does. So even with the official SDK you would likely get different results on different operating systems and system settings. You should never compare hashes of NSP files but always extract them and compare the files inside. You will notice that all files are hash equivalent maybe with some useless NDV0 fragment containing patch files missing. In the end NSP files created by nsz have the same quality as any other NSP. There are no "official NSPs" only “official NCAs”. The only reason we even use NSP containers is for convenience as otherwise we would have a folder containing multiple NCA files for every game.

 

[Tuxie]

I understand that, but I am told that the (currently private) No-Intro Switch DATs have hashes of specific XCI and NSP files, not their NCAs, so for the purpose of collecting and DAT-matching NSZ is not lossless.

I'm not familiar with the NSP/XCI formats yet but would it be feasible/simple to make a special version that just decrypts everything so it can be compressed well by 7zip but keeps all non-file data intact, so it can be reversed? Kind of like .nkit.iso for GameCube and Wii in purpose?

 

[nicoboss]

I understand that, but I am told that the (currently private) No-Intro Switch DATs have hashes of specific XCI and NSP files, not their NCAs, so for the purpose of collecting and DAT-matching NSZ is not lossless.

I'm not familiar with the NSP/XCI formats yet but would it be feasible/simple to make a special version that just decrypts everything so it can be compressed well by 7zip but keeps all non-file data intact, so it can be reversed? Kind of like .nkit.iso for GameCube and Wii in purpose?

The compression itself gets hash equivalently reversed. Only NCA files are getting compressed to NCZ files and decompressed back to NCA files. The whole compression is fully lossless and bit/hash equivalent and based on zStandard. Compressing the decrypted NCA files with 7zip wouldn't help you at all and lead to the exact same problem as you still have to unpack and repack the NSP container in order to decrypt/encrypt NCA files. There obviously also where reasons why zStandard was chosen LZMA2 used in 7Zip like Nintendo Switch hardware support and performance.

Technically It would be possible to in place decrypt/encrypt NCAs like done in original nsZip but without any headers that file format got way too complex and died because nobody will ever adopt it and the reason why the original nsZip format was replaced by this one. See https://github.com/nicoboss/nsZip for the depreciated NSPZ/XCIZ legacy format.

The main problem is that idiot who though it would be intelligent hashing NSP files instead of the underlaying NCA files which by the way are already sha265 hashed and signed by Nintendo. The problem is that always if you unpack and repack an NSP you get something different because for example the of the physical order of the files inside the file system changes. If they made any reasonable specification in what physical order NCA files should be stored inside an NSP container let me know and I might adopt it. Please keep in mind that currently every tool packs NSP files differently and they all won't hash match so maybe better just let them know that they are idiots hashing an arbitrary generated files system. I mean I could just like keep the physical file order when converting between NSP and NSZ but still keeping an arbitrary order is a quite dumb thing to do but maybe I will just because it would be so easy to do but the order is not the only arbitrary thing inside a file system so it probably still won't hash match.

 

[8BitWonder]

Getting some errors thrown when trying to solid compress the day-1 ACNH updat
(With NSZ v3.1.1 and latest keys dumped from 9.2.0 placed at %userprofile%/.switch/prod.keys)

Anyone else experiencing this issue or is there something wrong on my end?

nsz -C -l 22 -S -t 3 -o "%cd%" "Animal Crossing New Horizons [01006F8002326000][US][v65536].nsp"

Solid compressing (level 22) D:\...\Animal Crossing New Horizons [01006F8002326000][US][v65536].nsp -> D:\...\Animal Crossing New Horizons [01006F8002326000][US][v65536].nsz
[ADDING]     01006f8002326800000000000000000b.cert 1792 bytes to NSP
[ADDING]     01006f8002326800000000000000000b.tik 704 bytes to NSP
[ADDING]     05f8f1822cf9e2e12e95c21291a50c66.nca 264704 bytes to NSP
[ADDING]     8f2308a98857bc40fe8dbaa3150c6176.cnmt.nca 4096 bytes to NSP
[ADDING]     8f2308a98857bc40fe8dbaa3150c6176.cnmt.xml 1413 bytes to NSP
[ADDING]     9774fb91618bb9913c6c4999bd5e8b23.nca 1512448 bytes to NSP
[ADDING]     cb177504c5224ad95ab8502251dda7a3.ncz 264371712 bytes to NSP
Traceback (most recent call last):
  File "nsz\SolidCompressor.py", line 128, in solidCompressNsp
  File "nsz\SolidCompressor.py", line 42, in processContainer
Exception: NCA can't be decrypted. Outdated keys.txt?

 

[mathew77]

solid compress the day-1 ACNH update

Tried to compress it, the results were good I think, here's some logs:

Spoiler

*******************************
COMPRESS A NSP\XCI
*******************************

 Compressing with level 22 and 2 threads
Animal Crossing New Horizons [01006
F8002326800][v65536].nsp -> x64\NSCB_outp
ut\Animal Crossing New Horizons [01006F8002326800][v65536].nsz

- Appending: cb177504c5224ad95ab8502251dda7a3.ncz (252.12MB)

  * Compressed at 81% from 252.12MB to 206.66MB  - cb177504c5224ad95ab8502251dda
7a3.nca
  * Compressed in 01:55 at speed: 2.3 MB/s

- Appending: 9774fb91618bb9913c6c4999bd5e8b23.nca (1.44MB)
- Appending: 05f8f1822cf9e2e12e95c21291a50c66.nca (258.5KB)
- Appending: 8f2308a98857bc40fe8dbaa3150c6176.cnmt.nca (4.0KB)
- Appending: 8f2308a98857bc40fe8dbaa3150c6176.cnmt.xml (1.48KB)
- Appending: cb177504c5224ad95ab8502251dda7a3.programinfo.xml (134.88KB)
- Appending: 9774fb91618bb9913c6c4999bd5e8b23.nacp.xml (8.23KB)
- Appending: 05f8f1822cf9e2e12e95c21291a50c66.legalinfo.xml (1.71KB)
- Appending: 01006f8002326800000000000000000b.tik (704B)
- Appending: 01006f8002326800000000000000000b.cert (1.75KB)
...................................................
STILL 0 FILES TO PROCESS
...................................................
---------------------------------------------------
*********** ALL FILES WERE PROCESSED! *************
---------------------------------------------------

Did that with the latest NSCB https://github.com/julesontheroad/NSC_BUILDER/releases

How can I help you more?

 

[8BitWonder]

Tried to compress it, the results were good I think, here's some logs:

Spoiler

*******************************
COMPRESS A NSP\XCI
*******************************

 Compressing with level 22 and 2 threads
Animal Crossing New Horizons [01006
F8002326800][v65536].nsp -> x64\NSCB_outp
ut\Animal Crossing New Horizons [01006F8002326800][v65536].nsz

- Appending: cb177504c5224ad95ab8502251dda7a3.ncz (252.12MB)

  * Compressed at 81% from 252.12MB to 206.66MB  - cb177504c5224ad95ab8502251dda
7a3.nca
  * Compressed in 01:55 at speed: 2.3 MB/s

- Appending: 9774fb91618bb9913c6c4999bd5e8b23.nca (1.44MB)
- Appending: 05f8f1822cf9e2e12e95c21291a50c66.nca (258.5KB)
- Appending: 8f2308a98857bc40fe8dbaa3150c6176.cnmt.nca (4.0KB)
- Appending: 8f2308a98857bc40fe8dbaa3150c6176.cnmt.xml (1.48KB)
- Appending: cb177504c5224ad95ab8502251dda7a3.programinfo.xml (134.88KB)
- Appending: 9774fb91618bb9913c6c4999bd5e8b23.nacp.xml (8.23KB)
- Appending: 05f8f1822cf9e2e12e95c21291a50c66.legalinfo.xml (1.71KB)
- Appending: 01006f8002326800000000000000000b.tik (704B)
- Appending: 01006f8002326800000000000000000b.cert (1.75KB)
...................................................
STILL 0 FILES TO PROCESS
...................................................
---------------------------------------------------
*********** ALL FILES WERE PROCESSED! *************
---------------------------------------------------

Did that with the latest NSCB https://github.com/julesontheroad/NSC_BUILDER/releases

How can I help you more?

NSCB worked on my end, thanks.
Not sure why nsz was giving me trouble.

 

[nraudigy2]

How to run the gui? Not sure what to do after I ran pip install nsz[gui] from command. It seems to be installed, but don't know how to start the gui.

 

[nicoboss]

How to run the gui? Not sure what to do after I ran pip install nsz[gui] from command. It seems to be installed, but don't know how to start the gui.

Just open the cmd and enter "nsz" without any command line arguments.

If the Scripts folder isn't inside your path environement variables for some reason just manually execute C:\Python37\Scripts\nsz.exe where you have to replace C:\Python37 with your Python installation location.

 

[logandzwon]

I have all these files called something like "package update 1.4.nsp" and I would like to rename them correctly.

Is there way to get nsz to output the package's version?
nsz -I package.nsp give me the TitleID, but I do not see the version.

 

[nicoboss]

I have all these files called something like "package update 1.4.nsp" and I would like to rename them correctly.

There already is a tool called nut to do so (often referred as NUT OG as latest NUT can't do that) You can obtain NUT OG from https://github.com/plato79/nut and latest titledb you can option form: https://github.com/blawar/titledb

Using nsz's --titlekeys command line argument you can update that titledb with your games titlekeys if you like however just to properly auto rename your games this shouldn't be required. NUT OG doesn't support the .nsz file extension by default so either mass rename them to nsp using "ren *.nsz *.nsp" or just patch it to support them if you even have any .nsz files. Be careful not to mix that fake .nsp files with your real .nsp files and make sure to rename them back using "ren *.nsp *.nsz" after autorenaming them.

I will probably add the ability to see the version to –-info soon as there isn’t really any reason why it’s not shown except speed which doesn’t really matter for --info.

 

[Selaht]

Getting some errors thrown when trying to solid compress the day-1 ACNH updat
(With NSZ v3.1.1 and latest keys dumped from 9.2.0 placed at %userprofile%/.switch/prod.keys)

Anyone else experiencing this issue or is there something wrong on my end?

nsz -C -l 22 -S -t 3 -o "%cd%" "Animal Crossing New Horizons [01006F8002326000][US][v65536].nsp"

Solid compressing (level 22) D:\...\Animal Crossing New Horizons [01006F8002326000][US][v65536].nsp -> D:\...\Animal Crossing New Horizons [01006F8002326000][US][v65536].nsz
[ADDING]     01006f8002326800000000000000000b.cert 1792 bytes to NSP
[ADDING]     01006f8002326800000000000000000b.tik 704 bytes to NSP
[ADDING]     05f8f1822cf9e2e12e95c21291a50c66.nca 264704 bytes to NSP
[ADDING]     8f2308a98857bc40fe8dbaa3150c6176.cnmt.nca 4096 bytes to NSP
[ADDING]     8f2308a98857bc40fe8dbaa3150c6176.cnmt.xml 1413 bytes to NSP
[ADDING]     9774fb91618bb9913c6c4999bd5e8b23.nca 1512448 bytes to NSP
[ADDING]     cb177504c5224ad95ab8502251dda7a3.ncz 264371712 bytes to NSP
Traceback (most recent call last):
  File "nsz\SolidCompressor.py", line 128, in solidCompressNsp
  File "nsz\SolidCompressor.py", line 42, in processContainer
Exception: NCA can't be decrypted. Outdated keys.txt?

Same problem here.
But, at least for me, it only occurs when trying to compress newer games (released after march-2020 or so) even with updated keys (10.0.3).

It would be great, as others have already suggested, to return do gui after the operation and an option to set default folders.

Anyway, thanks for this awesome tool!

 

[nicoboss]

Same problem here.
But, at least for me, it only occurs when trying to compress newer games (released after march-2020 or so) even with updated keys (10.0.3).

It would be great, as others have already suggested, to return do gui after the operation and an option to set default folders.

Anyway, thanks for this awesome tool!

Latest release doesn't support masterkey 0x10 and above. Just use latest master and it will work. This will be fixed with a lot of other things in the next release which I will publish as soon as possible. Latest master also has some GUI improvements like Drag & Drop. Even more GUI features and a lot of work on nsz will be done after university exam period ends in early July.

 

[duckbill007]

@nicoboss, back to hasing. Even keeping file order is not enough: files in the "original" nsp can be aligned for example by megabytes and can has "holes" between them.

 

[SonikBR]

Has anyone compiled a list of games and sizes after compression?
I'm running this over a collection of 1TB of data... Would be good to skip files known to have poor compression ratio.

 

[mathew77]

Has anyone compiled a list of games and sizes after compression?

Some kind of a games list (but it's pretty old though):