Homebrew NSP to NSZ - C++

[snuffbot1]

Here you go, a very small c++ version for converting nsp/xci files to nsz/xcz and back again. Source code and exe are in the zip.

Example usage:
Read the included readme file for usage.

This is a command line tool, at some point I might make a gui. I just made it for my own personal use but feel free to mod/share or whatever you want. It's modified rust code version on the net that I converted to c++ for my own use.

 

[mathew77]

@snuffbot1
Thank you for your lightweight C++ realization, I like the idea but unfortunately for right now it's very slow.

I tried to test it versus the nszgui by nicoboss, the results with 200 MB NSP file (level 22 blocks and long-distance) are:

1. His version took 9 seconds to compress with maximum CPU utilization across all cores:
   
   
   
2. Your version took 31 seconds, three and a half times long and the CPU load average was pretty low:
   
   
   

   nszrs.exe -C -L -B -l 22 -k prod.keys test.nsp

Please optimize it if you could.

 

[snuffbot1]

@snuffbot1
Thank you for your lightweight C++ realization, I like the idea but unfortunately for right now it's very slow.
I tried to test it versus the nszgui by nicoboss, the results with 200 MB NSP file (level 22 blocks and long-distance) are:

1) His version took 9 seconds to compress with maximum CPU utilization across all cores:

View attachment 592261

2) Your version took 31 seconds, three and a half times long and the CPU load average was pretty low:

View attachment 592262

nszrs.exe -C -L -B -l 22 -k prod.keys test.nsp

Please optimize it if you could.

I'm adding xci support just now, I didn't look at nicboss version, but I assume he is starting a few threads so it's using more cpu usage rather that taking longer. I could add that I guess. I've got xcz decompress done and am about 90% though adding xci to xcz, so I'll have a look at multithreading/cpu core use once I'm done with that.

 

[mathew77]

@snuffbot1
..and add the ability to verify for hash mismatches (NCA & PFS0, SHA256) right after the compression, too, please!..

 

[snuffbot1]

@snuffbot1
..and add the ability to verify for hash mismatches (NCA & PFS0, SHA256) right after the compression, too, please!..

I'll think about it, I've just finished xci stuff just now so block and solid compress works. I'll add flags so you can remove update/logo partitions as it's pointless packing an xci with firmware update files included (defeats the point really).

Note to self:
Build windows fancy shmancy GUI with drag and drop support... dropdowns/switches and shit!

Post automatically merged: Yesterday at 3:05 PM

@mathew77

Try this command:
nszrs -C -S -l 20 -k prod.keys -o ./ game.nsp # solid

That maxed out my cpu and should be faster, I've also added xci > xcz < xci stuff and fixed a few bugs. Report issues and I'll fix them before making a gui. For verification you can use NXfile viewer for now, when I make a gui I'll probably add that as a separate feature from the exe I posted.

 

[mathew77]

solid

It's the deal breaker, I need the block compression, only. nicoboss maxed out the CPU even with block compression method... It's also has an option to auto-verify the results afterwards.

The problem is, it's python-based, so heavy and cumbersome, I'd like C++ solution more.

 

[snuffbot1]

It's the deal breaker, I need the block compression, only. nicoboss maxed out the CPU even with block compression method... It's also has an option to auto-verify the results afterwards.

The problem is, it's python-based, so heavy and cumbersome, I'd like C++ solution more.

Solid produces much smaller files, why do you need block?

EDIT: Verification of nca's flag added. Will post when I finish checking stuff, most stuff is completed now, just some tweaks/bugs to fix...

Post automatically merged: Yesterday at 9:52 PM

@mathew77

Have a look here at ZSTD_c_nbWorkers section:
https://facebook.github.io/zstd/zstd_manual.html

To speed things up, how many workers are set in the python code you are currently using?

 

[mathew77]

verification of nca's flag added.

Thank you!..

why do you need block?

The main reason is that when installing large and high-compressed .NSZ files, like 15 GB with Level 22 in long-distance mode, they do not need the Application mode on a Switch, standard Applet mode works well. It's crucial for inexperience users = so my needs.

My ZSTD_c_nbWorkers is 8 though.

 

[snuffbot1]

Thank you!..


The main reason is that when installing large and high-compressed .NSZ files, like 15 GB with Level 22 in long-distance mode, they do not need the Application mode on a Switch, standard Applet mode works well. It's crucial for inexperience users = so my needs.

My ZSTD_c_nbWorkers is 8 though.

OK no problem, what I'll do is look at ZSTD_c_nbWorkers code in nsp/xci packing but I think it already automatically sets workers by itself, maybe I can add an override flag. I've been fixing some minor bugs and stuff but I am nearly completed now with the small exe (only started making this from scratch yesterday), I've still got a gui to make but I'll release the update command line exe later today. At the minute I am working something else. Verfication now works though I didn't add it to check automatically after compression/decompression. I just put it with it's own flag, like this : nszrs -V -k prod.keys game.xci
I suppose you could make a little batch script to run that automatically if you wanted. In the gui when I do that I want it as a seperate thing so that's why I've done it like that.

Post automatically merged: Today at 9:40 AM

@mathew77

I've added a flag for nsp solid compression so you can set your threads - please can you do some tests and report back.

nszrs -C -S -k prod.keys -o out/ game.nsp # auto threads
nszrs -C -S -t 1 -k prod.keys -o out/ game.nsp # single-threaded
nszrs -C -S -t 4 -k prod.keys -o out/ game.nsp # 4 workers

What to do is set the -t value to 8 or 20 or whatever and let me know what kind of speed you get without your CPU going mental or running out of memory, once Done I'll add code for block mode as currently this is single thread for now which is why it's slower than the python version you are using.

Other example commands:
Example commands
nszrs -C -B -k prod.keys -s 20 -o ./ game.xci # block
nszrs -C -S -k prod.keys -o ./ game.xci # solid
nszrs -C -k prod.keys -o ./ game.xci # default block
nszrs -C -B -k prod.keys -s 20 -o ./ game.nsp # block
nszrs -C -S -l 20 -K -P -k prod.keys -o ./ game.nsp # solid (better than block mode for compression)
nszrs -C -k prod.keys -o ./ game.nsp # default solid
nszrs -C -L -o ./ game.nsp

nszrs -C -k prod.keys -S -l 22 -x --keep -o ./solid game.xci
nszrs -C -B -k prod.keys -s 20 -o ./ game.xci # default (drops logo, empties update/normal)
nszrs -C -B -k prod.keys -s 18 --keep -o ./ game.xci # keeps everything
nszrs -C -S -l 20 -k prod.keys -o ./ game.xci

nszrs -D -F -k prod.keys -o ./ ./game.nsz
nszrs -D -k prod.keys -o ./ ./game.xcz
nszrs -D -k prod.keys -o ./ ./game.nsz # unpacks to identicle as source nsp

nszrs -C -S -k prod.keys -o out/ game.nsp # auto threads
nszrs -C -S -t 1 -k prod.keys -o out/ game.nsp # single-threaded
nszrs -C -S -t 4 -k prod.keys -o out/ game.nsp # 4 workers

nszrs -V -k prod.keys game.xci # verify file

 

[mathew77]

It's 1 min 3 sec for a 200 MB file, for now:

nszrs.exe -C -L -S -l 22 -k prod.keys test.nsp

No crashes at all, will wait for my beloved block compression mode!..

 

[snuffbot1]

It's 1 min 3 sec for a 200 MB file, for now:

nszrs.exe -C -L -S -l 22 -k prod.keys test.nsp

View attachment 592418
No crashes at all, will wait for my beloved block compression mode!..

I've done it for block now, it works good. I just need to fix the percentage output, what kind of time were you getting before?

 

[mathew77]

@snuffbot1
Apprx. ~9 seconds with block compressing mode 22 level with long-distant mode per 200 MB, I guess.

 

[snuffbot1]

Here you go, this works with block compression now:

nszrs -C -B -t 20 -k prod.keys -o ./ game.xci

Test this and let me know how you get on, what speeds you expected and what you get.

 

[mathew77]

Test this and let me know how you get on, what speeds you expected and what you get.

Perfect!.. It's 9 seconds now!..



There's still a little headroom to improvements, CPU load average is ~85-90% but for now it's no less then python version, and I love it, my gratitudes.

 

[snuffbot1]

Perfect!.. It's 9 seconds now!..

View attachment 592427

There's still a little headroom to improvements, CPU load average is ~85-90% but for now it's no less then python version, and I love it, my gratitudes.

Tell me what improvements you want, I've got my own few tweaks to do but I'll consider adding yours. Once that's done you'll get a nice gui to use it with so it will basically be drag and drop. Also increase threads to 40, it will be even faster.

 

[duckbill007]

Long distance mode is useless in case of block compression because your blocks are smaller than distance.
Also python code still uses native zstd library, so it is the same code. Also nicoboss' is the reference implementation, so I prefer to always use it.

 

[snuffbot1]

OP updated with latest source code and exe file.

@DuCkBiLL - nicboss python version GUI is horrible to use and looks janky and the python version kept crashing/failing to run for me. I'd also like to add my own features so meh! So please don't use this version and stick to using the version you currently have. I've made this version in about a day and it does pretty much eveything I want for now, but I can change it whenever I want and so can everyone else that hates looking at crappy GUI's and doesn't want to download python and have lots of python files filling up their hard drive when they can just have 1 single small exe file to do the same thing.