Homebrew RELEASE NSZ - Homebrew compatible NSP/XCI compressor/decompressor

LangDuTienTu

Member
Newcomer
Joined
Oct 4, 2023
Messages
5
Trophies
0
Age
36
XP
37
Country
Vietnam
Did you use solid compression?

No, I use block compression. As solid compression does not save much more but takes a lot longer.
Post automatically merged:

@LangDuTienTu
Here's a lil old but still functioning chart, take a look:

Thanks. I compare some of games, they are almost as same size as that list. Problem is most of my games are not in the top % list and nsz truely helps only a bit. I guess that newer games around this time improve and already compressed so nsz does not help much.

I have exactly 2846 files, 2.93 TB and nsz compress them to 2.42 TB
 
Last edited by LangDuTienTu,
  • Like
Reactions: mathew77

LangDuTienTu

Member
Newcomer
Joined
Oct 4, 2023
Messages
5
Trophies
0
Age
36
XP
37
Country
Vietnam
I wanna ask a little bit more detail about nsz, that does it consume more TBW of SD Card than normal nsp. IMO exp while we copy an nsz through DBI it's gonna decompress to write the ncz than decompress the ncz to nca, such add write cycle of the ncz. Correct me if Iam wrong.
 

wontell

Member
Newcomer
Joined
Dec 25, 2021
Messages
5
Trophies
0
XP
48
Country
Jamaica
I've compressed a rom and decompressed it, then I checked the original's and the decompressed file's SHA3-516 hashes and they are not identical, is this intended/expected? I assumed the files would be identical. Even in size they are not the same.
 

LangDuTienTu

Member
Newcomer
Joined
Oct 4, 2023
Messages
5
Trophies
0
Age
36
XP
37
Country
Vietnam
I've compressed a rom and decompressed it, then I checked the original's and the decompressed file's SHA3-516 hashes and they are not identical, is this intended/expected? I assumed the files would be identical. Even in size they are not the same.

You need enabling Keep Delta Fragments
 

Joonboi

Member
Newcomer
Joined
Mar 5, 2020
Messages
15
Trophies
0
Age
40
XP
145
Country
United States
i had an issue and it seems to only happen with this one file. I take it the file must be faulty, but the file plays 100%. do i have to re rip it, or is this a nsz related issue?

Code:
    NszVerify(filePath, removePadding, raiseVerificationException, raisePfs0Exception, originalFilePath, statusReportInfo, pleaseNoPrint)
  File "V:\LaunchBox\Games\Nintendo Switch\nsz_v4.5.0_win64_portable\lib\site-packages\nsz\NszDecompressor.py", line 46, in verify
    __decompressNsz(filePath, None, removePadding, False, raiseVerificationException, raisePfs0Exception, originalFilePath, statusReportInfo, pleaseNoPrint)
  File "V:\LaunchBox\Games\Nintendo Switch\nsz_v4.5.0_win64_portable\lib\site-packages\nsz\NszDecompressor.py", line 191, in __decompressNsz
    fileHashes = FileExistingChecks.ExtractHashes(filePath)
  File "V:\LaunchBox\Games\Nintendo Switch\nsz_v4.5.0_win64_portable\lib\site-packages\nsz\FileExistingChecks.py", line 14, in ExtractHashes
    container.open(str(gamePath), 'rb')
  File "V:\LaunchBox\Games\Nintendo Switch\nsz_v4.5.0_win64_portable\lib\site-packages\nsz\Fs\Nsp.py", line 198, in open
    super(Nsp, self).open(path or self.path, mode, cryptoType, cryptoKey, cryptoCounter)
  File "V:\LaunchBox\Games\Nintendo Switch\nsz_v4.5.0_win64_portable\lib\site-packages\nsz\Fs\Pfs0.py", line 204, in open
    r = super(Pfs0, self).open(path, mode, cryptoType, cryptoKey, cryptoCounter)
  File "V:\LaunchBox\Games\Nintendo Switch\nsz_v4.5.0_win64_portable\lib\site-packages\nsz\Fs\BaseFs.py", line 97, in open
    r = super(BaseFs, self).open(path, mode, cryptoType, cryptoKey, cryptoCounter)
  File "V:\LaunchBox\Games\Nintendo Switch\nsz_v4.5.0_win64_portable\lib\site-packages\nsz\Fs\File.py", line 210, in open
    self.f = open(path, mode)
FileNotFoundError: [Errno 2] No such file or directory: 'V:\\LaunchBox\\Games\\Nintendo Switch\\NSP\\Mario Kart 8 Deluxe[0100152000022000][v0].nsz'
 

Attachments

  • Screenshot (21).png
    Screenshot (21).png
    292.7 KB · Views: 79

wontell

Member
Newcomer
Joined
Dec 25, 2021
Messages
5
Trophies
0
XP
48
Country
Jamaica
i had an issue and it seems to only happen with this one file. I take it the file must be faulty, but the file plays 100%. do i have to re rip it, or is this a nsz related issue?

Code:
FileNotFoundError: [Errno 2] No such file or directory: 'V:\\LaunchBox\\Games\\Nintendo Switch\\NSP\\Mario Kart 8 Deluxe[0100152000022000][v0].nsz'

You are trying to access the NSZ file which by the screenshot you took it doesn't exist there, only the NSP file
 

Joonboi

Member
Newcomer
Joined
Mar 5, 2020
Messages
15
Trophies
0
Age
40
XP
145
Country
United States
You are trying to access the NSZ file which by the screenshot you took it doesn't exist there, only the NSP file
yes, because it isn't being created.
when run using CLI command, this is the output:


Code:
Solid compressing (level 18) V:\LaunchBox\Games\Nintendo Switch\updates\Mario Kart 8 Deluxe[0100152000022800][v1114112].nsp -> V:\LaunchBox\Games\Nintendo Switch\updates\Mario Kart 8 Deluxe[0100152000022800][v1114112].nsz
[ADDING]     01001520000228000000000000000004.cert 1792 bytes to NSP
[ADDING]     01001520000228000000000000000004.tik 704 bytes to NSP
[ADDING]     1b74c7e0fbbca563874c81c337a3a4f9.nca 1521152 bytes to NSP
[ADDING]     575977908d8a2697c8a907fd57b819db.nca 244224 bytes to NSP
[ADDING]     628a3b5efa1b39e1d938d276ea10d5b8.cnmt.nca 9216 bytes to NSP
[ADDING]     628a3b5efa1b39e1d938d276ea10d5b8.cnmt.xml 1659 bytes to NSP
[ADDING]     90ce7b97ac68f950a1969e3c1f58c821.ncz 4785980416 bytes to NSP
Traceback (most recent call last):
  File "v:\LaunchBox\Games\Nintendo Switch\nsz_v4.5.0_win64_portable\lib\site-packages\nsz\SolidCompressor.py", line 133, in solidCompressNsp
    processContainer(container, nsp, compressionLevel, keepDelta, useLongDistanceMode, threads, statusReport, id, pleaseNoPrint)
  File "v:\LaunchBox\Games\Nintendo Switch\nsz_v4.5.0_win64_portable\lib\site-packages\nsz\SolidCompressor.py", line 96, in processContainer
    buffer += partitions[partNr].read(CHUNK_SZ - len(buffer))
  File "v:\LaunchBox\Games\Nintendo Switch\nsz_v4.5.0_win64_portable\lib\site-packages\nsz\Fs\File.py", line 327, in read
    self._buffer = super(BufferedFile, self).read(pageReadSize)
  File "v:\LaunchBox\Games\Nintendo Switch\nsz_v4.5.0_win64_portable\lib\site-packages\nsz\Fs\File.py", line 84, in read
    return self.f.read(size)
  File "v:\LaunchBox\Games\Nintendo Switch\nsz_v4.5.0_win64_portable\lib\site-packages\nsz\Fs\File.py", line 330, in read
    raise IOError('read returned empty ' + hex(self.tellAbsolute()))
OSError: read returned empty 0x6d7e44df


Compressing  38%|███████████████████████▊                                      | 1744/4564 MiB [03:38<05:52, 8.01 MiB/s]
Done!
 
Last edited by Joonboi,

nicoboss

Well-Known Member
OP
Member
Joined
Feb 1, 2019
Messages
132
Trophies
0
Age
27
XP
1,207
Country
Switzerland
I just released NSZ 4.6.0:

Major new features:
  • --verify now uses file-level sha256 NSP hash validation. --verify now guarantees that the sha256 hash of the original file and the decompressed file will match and so the file is bit-identical recreated.
  • Major XCI/XCZ rewrite to support multiple XCI partitions: Specify --keep to keep all partitions including their content. By default, the content of all but the secure partition is removed but the empty partition itself are kept for Yuzu Emulator compatibility.
Bugfixes:
  • Added missing null terminator at the end of the string table. This fixes #151
  • Fixed issue with PFS0 header overlapping the first file when the --remove-padding option is used
  • Fixed some logic responsible to handle too short PFS0 header sizes
  • Fixed PFS0 header padding so it follows the PFS0 specification. This fixes #150
  • We don't want to touch the default Kivy font but instead use LabelBase to set DEFAULT_FONT for this specific application. This fixes #154
  • Added the ability to read full xci dumps thanks to @alucryd
  • Added master_key_10 key hash thanks to @seiya-git
  • Better support deleting source files. This fixes #141
Command line argument changes:
  • Renamed --remove-padding to --fix-padding. This was done as this option now makes NSP padding match the No-Intro standard.
  • Renamed --keep-delta to --keep as it's no longer just used to keep NDV0 files but also to keep all XCI partitions
  • --verify and --remove-padding are incompatible with each other's. For compatibility reasons --quick-verify will be used automatically if this situation occurs to match the command line argument behavior prior to NSZ v4.6.0.
 
Last edited by nicoboss,

nicopretto

Member
Newcomer
Joined
Mar 13, 2019
Messages
5
Trophies
0
Age
44
XP
76
Country
Italy
I don't understand why the GUI can't be downloaded normally from github with the program folder.
I honestly couldn't figure out how to put it.
 

nicoboss

Well-Known Member
OP
Member
Joined
Feb 1, 2019
Messages
132
Trophies
0
Age
27
XP
1,207
Country
Switzerland
On Monday I released NSZ 4.6.1
This day one patch fixes the MPLUS1p-Medium.ttf not found issue of NSZ 4.6 GUI

Older NSZ versions could corrupt the global Kivy configurations as reported in #154. This was fixed in NSZ 4.6. Some users where unable to start NSZ 4.6 GUI due to having corrupted Kivy configurations. This release automatically repairs the Kivy configuration. It does so by resetting the default_font property if corrupted. This fixes #164




Thanks. I compare some of games, they are almost as same size as that list. Problem is most of my games are not in the top % list and nsz truely helps only a bit. I guess that newer games around this time improve and already compressed so nsz does not help much.

I have exactly 2846 files, 2.93 TB and nsz compress them to 2.42 TB
@LangDuTienTu Your observation that newer games often already come compressed is right. With the slow SD-Card and slow eMMC read speed it's much faster to trade of some CPU cycles to decompress game assets on the fly. Good game developers slowly started to notice the potential to optimizing loading times that way. There now even is an officially way to compress to content of NCA files as far I'm aware. Still worth using NSZ in my opinion. Especially if you use block compression only blocks that get smaller if compressed will actually be stored compressed.

I wanna ask a little bit more detail about nsz, that does it consume more TBW of SD Card than normal nsp. IMO exp while we copy an nsz through DBI it's gonna decompress to write the ncz than decompress the ncz to nca, such add write cycle of the ncz. Correct me if Iam wrong.
@LangDuTienTu It uses less TBW if copied to the SD card and then installed and the same TBW if installed from a remote location (USB/LAN/WLAN) as it directly reads the compressed file and writes the decompressed files to the SD Card/eMMC. NCZ files are directly read from the PFS0 container of the compressed NSZ/XCZ and never written to your SD Card. As you can see even on PC all those intermediate steps are never written to disk when you convert an NSP to an NSZ or NSZ to NSP. It all happens in small chanks in RAM using virtual file systems.

I've compressed a rom and decompressed it, then I checked the original's and the decompressed file's SHA3-516 hashes and they are not identical, is this intended/expected? I assumed the files would be identical. Even in size they are not the same.
@wontell Are you talking about NSP => NSZ => NSP. If so this definitely should be bit-identical and so hash identical. Please try using NSZ 4.6.1 with the command line options --keep and --verify. --verify now actually computes the SHA265 hash of the entire NSP file and errors if decompression wouldn't be bit-identical.

If we are talking about XCI => XCZ => XCI then this is expected behavior. There is not much to worry about it as all NCA files inside the HFS0 file-system are already bit-identically recreated. With the major XCI/XCZ rewrite in NSP 4.6 things are getting quite close to bit-identical recreation but still a lot of work is required to get the HFS0 recreation bit-identical. For further discussion about this topic take a look at #147: "Full XCI (Key Area + XCI) and reversible XCZ". The No-Intro standard now supports XCI so bit-identical recreation makes a lot of sense and is planned for the next major version.

i had an issue and it seems to only happen with this one file. I take it the file must be faulty, but the file plays 100%. do i have to re rip it, or is this a nsz related issue?
@Joonboi It could be a corrupted dump but seems unlikely. Most likely you are using an outdated prod.keys dump missing the keys required to decrypt this dump. Please also make sure to have the latest prod.keys files in your %userprofile%/.switch folder. Please retest this on the latest NSP 4.6.1 release and let me know if everything is working now.

I don't understand why the GUI can't be downloaded normally from github with the program folder.
I honestly couldn't figure out how to put it.
@nicopretto What are you talking about? It can be downloaded from GitHub without any issues. Just download nsz_v4.6.1_win64_portable.zip from nicoboss/nsz/releases and extract it to any folder you like. It's a portable application so it really doesn't matter to which location you extract it. Alternatively you can also install Python 3.10 and install nsz including its GUI using pip install nsz[gui]. I guess I really need to update the initial post of this thread as there nothing is mentioned about the portable Windows version which already exists for over 4 years. Please let me know if you need any further help installing NSZ GUI. Did you manage to successfully install it?
 
  • Like
Reactions: LangDuTienTu

KurosuDesu

New Member
Newbie
Joined
Feb 12, 2024
Messages
3
Trophies
0
Age
54
XP
25
Country
France
Hello, and first, please don't read any entitlement or judging in the following. Not meant, and sorry if it sounds like that. I'm slowly discovering the NSZ format, initially from nsb builder, and I have a few questions.

I'm not targeting preservation/bit-exactness, but:
1) How are the nsz outputs by SAK or NSB builder related? It doesn't seem they use your binaries, so it does not look as easy as updating these binaries to change what these tools support.

Also it seems there isn't a quick path to having nsz support in any emulator, so the format is, for now, mostly meant for archival. Given this, could it be possible to update the format (hoping there's a version field in case this requires adding new signalling) to support:
2) No block size (and no index for random access) besides the archive being solid, or maybe leave that to the underlying compressor Ran through command-line, nsz default is solid
3) Get LZMA2 support (eg by outputting a ns7 file); in relation to 2), I think 7zip/the 7z format handles its own multiplexing of encoded streams to achieve parallelism

As of now, the games I tested it on don't compress much. I remember you mentioning the start 256KB of each nca are left encrypted.
4) I imagine once the nca are (mostly) unencrypted, that's the most compression one can get (ignoring these 256KB chunks)

I think the nsz format is "ticketless" (ie uses standard crypto?).
5) What is produced on uncompressing? A ticketless/standard-crypto nsp? I think you mentioned your will to not include the title key in the nsz format, as a copyrighted material.

Thanks for your replies.
 
Last edited by KurosuDesu,

sonicsleep

Well-Known Member
Newcomer
Joined
May 23, 2009
Messages
48
Trophies
1
XP
271
Country
The fitgirl release of skyward sword HD includes the emulator as well as the game, and firmware, and it comes to 2.7GB in total, yet the NSP is over 7GB. Idk what the NSZ gets down to, but I doubt it's close to 2.7GB. I haven't checked but do you support outputting a file that has no compression but is just decrypted, like dolphin can? I can lzma 7z those dolphin ones as long as I STORE them instead of compressing. Single-threaded LZMA1 compression with a 3GB dictionary only uses 3GB of RAM to decompress. It may take 32GB of ram to compress, but after that it's done!

Anyway, if fit girl can get that at 2.7GB... how does NSZ compare? Making a decrypted but block uncompressed NSZ would be less size than the NSP, be supposedly playable when unpacked from a solid archive!, and have the best compression since it is just decrypted and available for windows compact or 7zip. those r my thoughts, can u make one without any compression at all, in preparation for your 5.0 nsz mounting feature if i read that correctly, as i'd like those nsz's 7zipped to hell beforehand.
 

KurosuDesu

New Member
Newbie
Joined
Feb 12, 2024
Messages
3
Trophies
0
Age
54
XP
25
Country
France
@sonicsleep NSZ doesn't use LZMA (my suggestion) and if yes, it would use the default (non-fast LZMA2) implementation that is either slow or use a small dictionary. An alternative I've been exploring is simply using hactool/hacpack to convert a base NSP to plaintext, which then compresses well. I suspect people out there do the same. Currently my exploration are stuck on the handling of update NSPs which use a so-call BKTR section using a base NCA (from the base NSP).

My previous test was using -l 0 on xci: it is a special case which stores the blocks decyphered and uncompressed, and the NSZ fomat doesn't do that (solid compression case in the code). But it is cumbersome: you need to convert your NSP (including multiple ones eg packed by NSC_builder/NSCB) to XCI then with nsz to XCZ, then use eg 7zip or an alternative. And then the reverse is uncompressing with 7zip then repacking the XCZ with nsz to XCI.
 
Last edited by KurosuDesu,

blaizX

Active Member
Newcomer
Joined
Nov 24, 2009
Messages
43
Trophies
1
XP
223
Country
i don't want multiple compression tasks in parallel
but the program always does 4 compression-tasks at same time.

my command is:
nsz --threads 1 --level 18 --keep --verify --output L:\output\ -C T:\input\

what am I doing wrong?
how is the "command" to disable "--multi"?
(windows10)

or is there a option/flag to set cpu-prio to low?
 

Jack132

New Member
Newbie
Joined
Sep 21, 2024
Messages
1
Trophies
0
Age
44
XP
13
Country
Kazakhstan
Are there any discord group? I get error about prod keys
More specifically
Could not find title key “alphanumeric code”!


Error while compressing file: “path to the file”
 

Site & Scene News

Popular threads in this forum

General chit-chat
Help Users
  • AncientBoi @ AncientBoi:
    is this a 3ds fight?
  • kijetesantakalu042 @ kijetesantakalu042:
    N4dsXl > is this a 3ds fight?
  • BigOnYa @ BigOnYa:
    N3dsXL takes your Bishop, Checkmate.
    +2
  • kijetesantakalu042 @ kijetesantakalu042:
    N23ds >
    N3dsXL takes your Bishop, Checkmate.
  • The Real Jdbye @ The Real Jdbye:
    wtf is a n23ds
    +3
  • BigOnYa @ BigOnYa:
    School must be out for the kiddies.
  • kijetesantakalu042 @ kijetesantakalu042:
    @The Real Jdbye the prequal to the n24ds
  • Jayro @ Jayro:
    *steals all the skittles*
  • sombrerosonic @ sombrerosonic:
    damn, no skittles for me then, I guess ill be doing M&Ms then
  • kijetesantakalu042 @ kijetesantakalu042:
    Starbursts are a fate worse than death
  • sombrerosonic @ sombrerosonic:
    M&M's Mixed with Recess Peices
  • Jayro @ Jayro:
    I like Starbusts, much better than the rubbery hi-chew crap.
    +2
  • Jayro @ Jayro:
    But I'd have to say that Airheads are one of my favorites, especially when they've been warmed up a little in the hands.
    +1
  • SylverReZ @ SylverReZ:
    @Jayro, Agreed. I like them.
  • SylverReZ @ SylverReZ:
    Skittles get into your teeth, they have the same boring flavour.
  • SylverReZ @ SylverReZ:
    Only the colours tell your brain what flavour it should be, it triggers a sense response.
  • kijetesantakalu042 @ kijetesantakalu042:
    @Jayro Starburst and airheads are good but the way starbursts are wrapped...
  • SylverReZ @ SylverReZ:
    Eat them along with the wrappers. Saves your finger nails lol.
    +1
  • kijetesantakalu042 @ kijetesantakalu042:
    @SylverReZ I used to do that but my doctor said it was bad for my stomach
    +1
  • Jayro @ Jayro:
    Eew, I'm not gonna eat the wrappers... I have in the past as a kid in the 90's, and yuck!
    +1
  • NinStar @ NinStar:
    you are old
  • kijetesantakalu042 @ kijetesantakalu042:
    You know what you did,
    John Smith
  • S @ salazarcosplay:
    hello everyone
    +1
  • CzarVine @ CzarVine:
    At this point I don't eat candy much and I may just cut it out entirely. Although starbursts, airheads, and skittles all taste good to me.
    +1
    CzarVine @ CzarVine: At this point I don't eat candy much and I may just cut it out entirely. Although starbursts... +1