Homebrew RELEASE NSZ - Homebrew compatible NSP/XCI compressor/decompressor

[LangDuTienTu]

Did you use solid compression?

No, I use block compression. As solid compression does not save much more but takes a lot longer.

Post automatically merged: Oct 6, 2023

@LangDuTienTu
Here's a lil old but still functioning chart, take a look:

Thanks. I compare some of games, they are almost as same size as that list. Problem is most of my games are not in the top % list and nsz truely helps only a bit. I guess that newer games around this time improve and already compressed so nsz does not help much.

I have exactly 2846 files, 2.93 TB and nsz compress them to 2.42 TB

 

[LangDuTienTu]

I wanna ask a little bit more detail about nsz, that does it consume more TBW of SD Card than normal nsp. IMO exp while we copy an nsz through DBI it's gonna decompress to write the ncz than decompress the ncz to nca, such add write cycle of the ncz. Correct me if Iam wrong.

 

[wontell]

I've compressed a rom and decompressed it, then I checked the original's and the decompressed file's SHA3-516 hashes and they are not identical, is this intended/expected? I assumed the files would be identical. Even in size they are not the same.

 

[LangDuTienTu]

I've compressed a rom and decompressed it, then I checked the original's and the decompressed file's SHA3-516 hashes and they are not identical, is this intended/expected? I assumed the files would be identical. Even in size they are not the same.

You need enabling Keep Delta Fragments

 

[wontell]

You need enabling Keep Delta Fragments

I used that, but just to be sure I retest it and I'm getting the same output, different byte size.

 

[Joonboi]

i had an issue and it seems to only happen with this one file. I take it the file must be faulty, but the file plays 100%. do i have to re rip it, or is this a nsz related issue?

    NszVerify(filePath, removePadding, raiseVerificationException, raisePfs0Exception, originalFilePath, statusReportInfo, pleaseNoPrint)
  File "V:\LaunchBox\Games\Nintendo Switch\nsz_v4.5.0_win64_portable\lib\site-packages\nsz\NszDecompressor.py", line 46, in verify
    __decompressNsz(filePath, None, removePadding, False, raiseVerificationException, raisePfs0Exception, originalFilePath, statusReportInfo, pleaseNoPrint)
  File "V:\LaunchBox\Games\Nintendo Switch\nsz_v4.5.0_win64_portable\lib\site-packages\nsz\NszDecompressor.py", line 191, in __decompressNsz
    fileHashes = FileExistingChecks.ExtractHashes(filePath)
  File "V:\LaunchBox\Games\Nintendo Switch\nsz_v4.5.0_win64_portable\lib\site-packages\nsz\FileExistingChecks.py", line 14, in ExtractHashes
    container.open(str(gamePath), 'rb')
  File "V:\LaunchBox\Games\Nintendo Switch\nsz_v4.5.0_win64_portable\lib\site-packages\nsz\Fs\Nsp.py", line 198, in open
    super(Nsp, self).open(path or self.path, mode, cryptoType, cryptoKey, cryptoCounter)
  File "V:\LaunchBox\Games\Nintendo Switch\nsz_v4.5.0_win64_portable\lib\site-packages\nsz\Fs\Pfs0.py", line 204, in open
    r = super(Pfs0, self).open(path, mode, cryptoType, cryptoKey, cryptoCounter)
  File "V:\LaunchBox\Games\Nintendo Switch\nsz_v4.5.0_win64_portable\lib\site-packages\nsz\Fs\BaseFs.py", line 97, in open
    r = super(BaseFs, self).open(path, mode, cryptoType, cryptoKey, cryptoCounter)
  File "V:\LaunchBox\Games\Nintendo Switch\nsz_v4.5.0_win64_portable\lib\site-packages\nsz\Fs\File.py", line 210, in open
    self.f = open(path, mode)
FileNotFoundError: [Errno 2] No such file or directory: 'V:\\LaunchBox\\Games\\Nintendo Switch\\NSP\\Mario Kart 8 Deluxe[0100152000022000][v0].nsz'

 

[wontell]

i had an issue and it seems to only happen with this one file. I take it the file must be faulty, but the file plays 100%. do i have to re rip it, or is this a nsz related issue?

FileNotFoundError: [Errno 2] No such file or directory: 'V:\\LaunchBox\\Games\\Nintendo Switch\\NSP\\Mario Kart 8 Deluxe[0100152000022000][v0].nsz'

You are trying to access the NSZ file which by the screenshot you took it doesn't exist there, only the NSP file

 

[Joonboi]

You are trying to access the NSZ file which by the screenshot you took it doesn't exist there, only the NSP file

yes, because it isn't being created.
when run using CLI command, this is the output:

Solid compressing (level 18) V:\LaunchBox\Games\Nintendo Switch\updates\Mario Kart 8 Deluxe[0100152000022800][v1114112].nsp -> V:\LaunchBox\Games\Nintendo Switch\updates\Mario Kart 8 Deluxe[0100152000022800][v1114112].nsz
[ADDING]     01001520000228000000000000000004.cert 1792 bytes to NSP
[ADDING]     01001520000228000000000000000004.tik 704 bytes to NSP
[ADDING]     1b74c7e0fbbca563874c81c337a3a4f9.nca 1521152 bytes to NSP
[ADDING]     575977908d8a2697c8a907fd57b819db.nca 244224 bytes to NSP
[ADDING]     628a3b5efa1b39e1d938d276ea10d5b8.cnmt.nca 9216 bytes to NSP
[ADDING]     628a3b5efa1b39e1d938d276ea10d5b8.cnmt.xml 1659 bytes to NSP
[ADDING]     90ce7b97ac68f950a1969e3c1f58c821.ncz 4785980416 bytes to NSP
Traceback (most recent call last):
  File "v:\LaunchBox\Games\Nintendo Switch\nsz_v4.5.0_win64_portable\lib\site-packages\nsz\SolidCompressor.py", line 133, in solidCompressNsp
    processContainer(container, nsp, compressionLevel, keepDelta, useLongDistanceMode, threads, statusReport, id, pleaseNoPrint)
  File "v:\LaunchBox\Games\Nintendo Switch\nsz_v4.5.0_win64_portable\lib\site-packages\nsz\SolidCompressor.py", line 96, in processContainer
    buffer += partitions[partNr].read(CHUNK_SZ - len(buffer))
  File "v:\LaunchBox\Games\Nintendo Switch\nsz_v4.5.0_win64_portable\lib\site-packages\nsz\Fs\File.py", line 327, in read
    self._buffer = super(BufferedFile, self).read(pageReadSize)
  File "v:\LaunchBox\Games\Nintendo Switch\nsz_v4.5.0_win64_portable\lib\site-packages\nsz\Fs\File.py", line 84, in read
    return self.f.read(size)
  File "v:\LaunchBox\Games\Nintendo Switch\nsz_v4.5.0_win64_portable\lib\site-packages\nsz\Fs\File.py", line 330, in read
    raise IOError('read returned empty ' + hex(self.tellAbsolute()))
OSError: read returned empty 0x6d7e44df


Compressing  38%|███████████████████████▊                                      | 1744/4564 MiB [03:38<05:52, 8.01 MiB/s]
Done!

 

[nicoboss]

I just released NSZ 4.6.0:

Major new features:

• --verify now uses file-level sha256 NSP hash validation. --verify now guarantees that the sha256 hash of the original file and the decompressed file will match and so the file is bit-identical recreated.
• Major XCI/XCZ rewrite to support multiple XCI partitions: Specify --keep to keep all partitions including their content. By default, the content of all but the secure partition is removed but the empty partition itself are kept for Yuzu Emulator compatibility.

Bugfixes:

• Added missing null terminator at the end of the string table. This fixes #151
• Fixed issue with PFS0 header overlapping the first file when the --remove-padding option is used
• Fixed some logic responsible to handle too short PFS0 header sizes
• Fixed PFS0 header padding so it follows the PFS0 specification. This fixes #150
• We don't want to touch the default Kivy font but instead use LabelBase to set DEFAULT_FONT for this specific application. This fixes #154
• Added the ability to read full xci dumps thanks to @alucryd
• Added master_key_10 key hash thanks to @seiya-git
• Better support deleting source files. This fixes #141

Command line argument changes:

• Renamed --remove-padding to --fix-padding. This was done as this option now makes NSP padding match the No-Intro standard.
• Renamed --keep-delta to --keep as it's no longer just used to keep NDV0 files but also to keep all XCI partitions
• --verify and --remove-padding are incompatible with each other's. For compatibility reasons --quick-verify will be used automatically if this situation occurs to match the command line argument behavior prior to NSZ v4.6.0.

 

[nicopretto]

I don't understand why the GUI can't be downloaded normally from github with the program folder.
I honestly couldn't figure out how to put it.

 

[nicoboss]

On Monday I released NSZ 4.6.1
This day one patch fixes the MPLUS1p-Medium.ttf not found issue of NSZ 4.6 GUI

Older NSZ versions could corrupt the global Kivy configurations as reported in #154. This was fixed in NSZ 4.6. Some users where unable to start NSZ 4.6 GUI due to having corrupted Kivy configurations. This release automatically repairs the Kivy configuration. It does so by resetting the default_font property if corrupted. This fixes #164

---

Thanks. I compare some of games, they are almost as same size as that list. Problem is most of my games are not in the top % list and nsz truely helps only a bit. I guess that newer games around this time improve and already compressed so nsz does not help much.

I have exactly 2846 files, 2.93 TB and nsz compress them to 2.42 TB

@LangDuTienTu Your observation that newer games often already come compressed is right. With the slow SD-Card and slow eMMC read speed it's much faster to trade of some CPU cycles to decompress game assets on the fly. Good game developers slowly started to notice the potential to optimizing loading times that way. There now even is an officially way to compress to content of NCA files as far I'm aware. Still worth using NSZ in my opinion. Especially if you use block compression only blocks that get smaller if compressed will actually be stored compressed.

I wanna ask a little bit more detail about nsz, that does it consume more TBW of SD Card than normal nsp. IMO exp while we copy an nsz through DBI it's gonna decompress to write the ncz than decompress the ncz to nca, such add write cycle of the ncz. Correct me if Iam wrong.

@LangDuTienTu It uses less TBW if copied to the SD card and then installed and the same TBW if installed from a remote location (USB/LAN/WLAN) as it directly reads the compressed file and writes the decompressed files to the SD Card/eMMC. NCZ files are directly read from the PFS0 container of the compressed NSZ/XCZ and never written to your SD Card. As you can see even on PC all those intermediate steps are never written to disk when you convert an NSP to an NSZ or NSZ to NSP. It all happens in small chanks in RAM using virtual file systems.

I've compressed a rom and decompressed it, then I checked the original's and the decompressed file's SHA3-516 hashes and they are not identical, is this intended/expected? I assumed the files would be identical. Even in size they are not the same.

@wontell Are you talking about NSP => NSZ => NSP. If so this definitely should be bit-identical and so hash identical. Please try using NSZ 4.6.1 with the command line options --keep and --verify. --verify now actually computes the SHA265 hash of the entire NSP file and errors if decompression wouldn't be bit-identical.

If we are talking about XCI => XCZ => XCI then this is expected behavior. There is not much to worry about it as all NCA files inside the HFS0 file-system are already bit-identically recreated. With the major XCI/XCZ rewrite in NSP 4.6 things are getting quite close to bit-identical recreation but still a lot of work is required to get the HFS0 recreation bit-identical. For further discussion about this topic take a look at #147: "Full XCI (Key Area + XCI) and reversible XCZ". The No-Intro standard now supports XCI so bit-identical recreation makes a lot of sense and is planned for the next major version.

i had an issue and it seems to only happen with this one file. I take it the file must be faulty, but the file plays 100%. do i have to re rip it, or is this a nsz related issue?

@Joonboi It could be a corrupted dump but seems unlikely. Most likely you are using an outdated prod.keys dump missing the keys required to decrypt this dump. Please also make sure to have the latest prod.keys files in your %userprofile%/.switch folder. Please retest this on the latest NSP 4.6.1 release and let me know if everything is working now.

I don't understand why the GUI can't be downloaded normally from github with the program folder.
I honestly couldn't figure out how to put it.

@nicopretto What are you talking about? It can be downloaded from GitHub without any issues. Just download nsz_v4.6.1_win64_portable.zip from nicoboss/nsz/releases and extract it to any folder you like. It's a portable application so it really doesn't matter to which location you extract it. Alternatively you can also install Python 3.10 and install nsz including its GUI using pip install nsz[gui]. I guess I really need to update the initial post of this thread as there nothing is mentioned about the portable Windows version which already exists for over 4 years. Please let me know if you need any further help installing NSZ GUI. Did you manage to successfully install it?

 

[KurosuDesu]

Hello, and first, please don't read any entitlement or judging in the following. Not meant, and sorry if it sounds like that. I'm slowly discovering the NSZ format, initially from nsb builder, and I have a few questions.

I'm not targeting preservation/bit-exactness, but:
1) How are the nsz outputs by SAK or NSB builder related? It doesn't seem they use your binaries, so it does not look as easy as updating these binaries to change what these tools support.

Also it seems there isn't a quick path to having nsz support in any emulator, so the format is, for now, mostly meant for archival. Given this, could it be possible to update the format (hoping there's a version field in case this requires adding new signalling) to support:
2) No block size (and no index for random access) besides the archive being solid, or maybe leave that to the underlying compressor Ran through command-line, nsz default is solid
3) Get LZMA2 support (eg by outputting a ns7 file); in relation to 2), I think 7zip/the 7z format handles its own multiplexing of encoded streams to achieve parallelism

As of now, the games I tested it on don't compress much. I remember you mentioning the start 256KB of each nca are left encrypted.
4) I imagine once the nca are (mostly) unencrypted, that's the most compression one can get (ignoring these 256KB chunks)

I think the nsz format is "ticketless" (ie uses standard crypto?).
5) What is produced on uncompressing? A ticketless/standard-crypto nsp? I think you mentioned your will to not include the title key in the nsz format, as a copyrighted material.

Thanks for your replies.

 

[sonicsleep]

The fitgirl release of skyward sword HD includes the emulator as well as the game, and firmware, and it comes to 2.7GB in total, yet the NSP is over 7GB. Idk what the NSZ gets down to, but I doubt it's close to 2.7GB. I haven't checked but do you support outputting a file that has no compression but is just decrypted, like dolphin can? I can lzma 7z those dolphin ones as long as I STORE them instead of compressing. Single-threaded LZMA1 compression with a 3GB dictionary only uses 3GB of RAM to decompress. It may take 32GB of ram to compress, but after that it's done!

Anyway, if fit girl can get that at 2.7GB... how does NSZ compare? Making a decrypted but block uncompressed NSZ would be less size than the NSP, be supposedly playable when unpacked from a solid archive!, and have the best compression since it is just decrypted and available for windows compact or 7zip. those r my thoughts, can u make one without any compression at all, in preparation for your 5.0 nsz mounting feature if i read that correctly, as i'd like those nsz's 7zipped to hell beforehand.

 

[KurosuDesu]

@sonicsleep NSZ doesn't use LZMA (my suggestion) and if yes, it would use the default (non-fast LZMA2) implementation that is either slow or use a small dictionary. An alternative I've been exploring is simply using hactool/hacpack to convert a base NSP to plaintext, which then compresses well. I suspect people out there do the same. Currently my exploration are stuck on the handling of update NSPs which use a so-call BKTR section using a base NCA (from the base NSP).

My previous test was using -l 0 on xci: it is a special case which stores the blocks decyphered and uncompressed, and the NSZ fomat doesn't do that (solid compression case in the code). But it is cumbersome: you need to convert your NSP (including multiple ones eg packed by NSC_builder/NSCB) to XCI then with nsz to XCZ, then use eg 7zip or an alternative. And then the reverse is uncompressing with 7zip then repacking the XCZ with nsz to XCI.