I see, that's good to know for the NSPs. Now I only need to get this confirmed for the XCI side as well and everything's good to go. Don't know if the notes were being generic here in just mentioning NSP/NSZ files and the same applies to the XCI files.
For XCZ it is guaranteed that every file inside the HFS0 container is recreated bit-identically but there is no guarantee for the HFS0 container itself. This means all NCA files inside the NCZ will be recreated bit-identically and pass their signature checks but the hash of the XCI before compression and after decompression might not match as the empty space between files might not be preserved. To my knowledge there currently doesn't exist any no-intro standard for XCI files so none of this should really matter. Still there seems to be quite a lot of interest for bit-identical HFS0 recreation so I will probably implement it later this year (likely October if not earlier). Doing so would be quite simple and require almost no changes as it's almost the same as bit-identical PFS0 recreation used for NSP files. I wouldn't even be surprised if some XCI files can already be recreated bit-identically thanks to the changes made for NSP recreation but it's not something currently officially supported. Feel free to do some testing by your own and let me know about the results.
Last edited by nicoboss,
Probably a stupid question, but I haven't seen this information mentioned or I glanced too quickly. Does compressing an XCI file get rid of the update partition often present on cartridge? Just curious.
This is not a stupid question at all. Thanks a lot for mentioning this. I wasn't sure myself and had to check the source code. Current XCZ files indeed only contains the "secure" HFS0 partition while the "normal", "logo", and "update" HFS0 partitions are skipped. This really surprised me but it also makes sense not to include redundant data as the file format was originally designed to only keep the game bit identical while saving as much storage as possible and only recently the focus switched towards bit-identical preservation. Now it makes sense why the XCI checksums before compression and after decompression never matched despite HFS0 recreation already being quite good. I will for sure add an option to include all HFS0 partitions in the next update similar to how I did for the NDV0 delta fragments using the --keep-delta for NSZ.
Last edited by nicoboss,
Thanks for the clarification, much appreciated. If you have some more spare time to waste on teaching a peasant, I wouldn't mind also having some brief explanation on why the "sparse" nsps can't be compressed to nsz (spitting out errors). An example of sparse nsp is the game Lost in Random.
Nintendo should officially support NCZ in horizon and completely replace the downloading of NCA files for updates and games. I feel like i'm missing something for how hostile some seem to be towards compression. Why would Nintendo want games people buy to take longer to play and update instead of using a open source library or implementing the standard themselves?
Last edited by shrekexists,
I'm not sure on Nintendo's stance towards speeding up the update process, but I have noticed that various Yuzu and/or Ryujinx developers are kind of hostile towards the NSZ format. I did found one instance where someone explained that there are some technical hickups with supporting the ability to run Switch games in the emulators as NSZ files (read: not install and then play, but decompress on the fly). This much I can understand; this affects performance and resources required. What I don't understand however is - and you hear this one more often - is that this standard shouldn't supported because it's main purpose is piracy (file sharing) and that it could put the emulator in legal jeopardy. That's just ridiculous. Compression is just compression. By that logic they might as well not have bothered with the emulator at all, because it's no different than claiming that any point they make about the legality of the emulator just by claiming "most people use it for piracy".
The emulators can automatically convert incompatible or too demanding nsz files or tell people what to do for it to work. When I backup a Wii or Gamecube game from a disc I don't even want the uncompressed copy. I'm not sure nsz would actually slow anything down yuzu doesn't use much CPU but it would cap loading speeds at the speed of decompression so it would have to be setup to decompress in parallel. pzstd could be used in the emulators and on the console itself for parallel decompression.
Please correct me if I'm wrong but I don't think there is any legal way to obtain sparse games. How I assume spare games work is that the Nintendo eShop serves you a spare base game and the 0-day update at the same time and while installing they get combined. This is very similar to how NDV0 fragments get applied to an existing update by only downloading the difference between the update you have and the latest version. Maybe I'm misunderstanding something but I don't think you can just dump sparse games. I even tried MitM intercepting the Nintendo eShop download using the disable_ca_verification and disable_browser_ca_verification patches from misson20000 but unfortunately the Nintendo eShop seems to not be affected by those patches and still applies the CA check. So the only way I see would be to write CA patches to for the Nintendo eShop or use something like plato79's nut fork to directly download from Nintendo CDN which wouldn't be legal. But maybe I'm all wrong and you can just dump sparse games nowadays as I haven't tried dumping them for a long time. Can somebody try to dump one and let me know if dumping them is now possible?
There are also practical reasons for not supporting sparse games. A sparse game without it's update is pointless. Even if you could convert a sparse game to an NSZ the NSZ could never be played or installed without also providing the update linked to it. I might look into the possibility of adding support for sparse games but it's not on the top of my priority list but seams like a fun project if they are legally obtainable. If there is any documentation about the spare NSP file format please let me know.
Compressing the games would actully save Nintendo a ton of money in CDN cost. Even better they could use block compressed NSZ files to allow players playing games without having to decompress them saving precious eMMC and CD Card storage. On the Nintendo Switch games are only allowed to use 3 out of the 4 cores so the 4th core could be easily used to decompress the data on the fly which would even improving load times as more data can be read using the same SD read speed. I would however expect them to compress things on an NCA level so they can leave the encryption layer unchanged. This is actually exactly what they started to do for some games as far I'm aware but it's extremely rare and I never looked into it. If Nintendo wants to use this file format the MIT license definitely allows them to do so.
With NDV0 Nintendo already has an awesome way to speed up updates in a much better way but definitely would be worth it for base games and updates where the prior update is missing.
It's so ironic. Back when I developed the NSPZ file format which was the precursor to the NSZ file format I did so with the help from the ReSwitched Discord and advice from the Yuzu team. Back in the NSPZ days they where not against game compression in any way. I feel it's really unfair that only because some pirates started to using the file format it's now considered unethical. Without NSZ pirates would just use NSP/XCI which most of them even do. The entire piracy scene on private trackers is using NSP/XCI as far I'm aware. I find it such a stupid argument that the only reason to compress a game is to illegally distribute it. Everyone that owns a huge game collection will understand that without compression it will get absolutely huge in storage size and I find it very understandable that someone buying games digitally wants to have a personal backup for when the Nintendo Switch eShop will inevitable stop existing at some point in the future. This is also why I mostly focus on NSZ and not XCZ as game cartridges seam for me a quite safe and convenient way to store my physical games.
I even went as far to keep the first 0x4000 bytes encrypted in the NSP/NSZ file format to ensure I’m not decrypting anything that could be considered DRM and to make sure real keys are still needed to play the games. I also made sure not to include any copyrighted material like keys inside my tool. It’s very important for me to keep my tool and the NSZ/XCZ file format totally legal. I even put the statement "This project shall only be used for legally purchased games." in the first paragraph of the README.md file to discourage pirates from using my file format.
There is absolutely no technical or legal reason for emulators not to support the NSZ file format. With the block compressed NSZ format it's absolutely no issue to directly play them without decompressing them first and solid compressed NSZ files could easily be decompressed automatically. Because the emulator community doesn't want anything to do with NSZ I already know that with NSZ v5.0 I will implement a way to mount block compressed NSZ files as NSP so users can play them on any emulator without decompressing them first.
What's even more interesting is that the same method to directly play block compressed games could be implemented on the Nintendo Switch if SciresM ever completes his FS reimplementation but with the speed of Atmosphere development having slowed down a lot in the past years I'm not sure if this will ever happen. In the ReSwitched Discord SciresM mentioned: "Even keeping in mind that FS reimpl work is kinda temporarily stalled until eol (besides research)," So indeed quite unfortunate.
Last edited by nicoboss,
One thing I think is worth pointing out there about cartridges nowadays is that while I'm sure they'll last longer than the eShop does, it's not as safe as it used to be. Flash memory is used in cartridges because of bigger game sizes nowadays and this type of memory is less robust over time than ROM memory (and similar technologies) are; the charge in the memory cells won't last forever. It'll probably last beyond the eShop's lifespan, but considering that hardware can and will fail at some point, the ability to make backups is for that reason vital across the board. But that's just my opinion.
Thank you very much for the tool, Nicoboss.
I wonder if it would be possible the following requests:
- An option to convert .XCI files directly to .NSZ.
- An option to check an already existing .NSZ and see what level compression was used, allowing to recompress it with highest level compression if it was converted with a lower one. I know the tool can decompress a .NSZ into a .NSP, but I would like if it could recompress automatically .NSZ files using the higest level possible.
Thanks for your time.
I wonder if it would be possible the following requests:
- An option to convert .XCI files directly to .NSZ.
- An option to check an already existing .NSZ and see what level compression was used, allowing to recompress it with highest level compression if it was converted with a lower one. I know the tool can decompress a .NSZ into a .NSP, but I would like if it could recompress automatically .NSZ files using the higest level possible.
Thanks for your time.
I fully agree but your HDD/SDD will likely corrupt the data much earlier due to bit rot. The best thing you can do about it is to create a ZFS RAID 1, RAID 5 or RAID 6 pool and make sure the scheduled scrub happens every month to fix bit rot. The second-best thing would likely be to burn them on a Blue Ray so your backups will likely exceed your own lifetime before disk rot will eventually destroy them as well. You indeed made a great point why even for game cartages backups are important. I will definitely make sure to backup all my game cartages.
It definitely would be possible to add booth those features.
I assume for XCI to NSP/NSZ conversion one can just dump the content of the secure partition of the XCI into an PFS0 container but I need to do some more research how it really needs to be done. This is not that high on my priority list and so I don't think it will be implemented soon. With most title installers can just install XCZ as if it would be an NSZ so not sure how high the demand there is for such a feature. I might still see me doing it just because it barely requires any effort to implement. If have the ability and time feel free to implement this by your own and create a merge request if you don’t want to wait.
Your second requested feature will be implemented with NSZ v5.0.0 for sure as transcompressing will be absolutely required to convert solid into block compressed NSZ files once mounting block compressed NSZ files are a thing. Regarding checking the compression level, I wasn't sure if this i s possible as the file format never stores it somewhere but luckily it turns out it probably is:
Python:
decompressor = zstd.ZstdDecompressor()
params = decompressor.get_frame_parameters(data.read())
print(params.compression_level)
Last edited by nicoboss,
Hi @nicoboss, I got a couple more questions about some details that weren't clear to me from reading the github page and this thread (or I may have read over it; can't rule that possibility out).
There's frequently mentions of partions and such, like PFS0, NVD0, HFS0, etc. What are these partitions, exactly? What is their purpose?
The other thing I wanted to ask about is solid and block compression. I understand that solid compression is better for storage size and block compression allows for random read access. How big is the difference between solid and block compression? Does it impact install time compared to solid compression? Why is solid compression the default for NSP while block compression is the default for XCI?
I'm hoping to understand these things better so I can make a better choice on how to go about compressing my library (which will likely be soon as I'm starting to run out of space on my HDD, at least for the NSP portion).
There's frequently mentions of partions and such, like PFS0, NVD0, HFS0, etc. What are these partitions, exactly? What is their purpose?
The other thing I wanted to ask about is solid and block compression. I understand that solid compression is better for storage size and block compression allows for random read access. How big is the difference between solid and block compression? Does it impact install time compared to solid compression? Why is solid compression the default for NSP while block compression is the default for XCI?
I'm hoping to understand these things better so I can make a better choice on how to go about compressing my library (which will likely be soon as I'm starting to run out of space on my HDD, at least for the NSP portion).
I just released NSZ 4.5.0:
Implemented bit-identical FileEntryTable padding recreation
@Nynrah Let's try to make things clearer for you. Please don't hesitate to ask if something is still unclear to you.
PFS0 (Partition File System):
The file system used in NSP/NSZ files. See it like NTFS/ExFAT/Fat32 but specially made for NSP files.
HFS0 (SHA-256 File System):
The file system used in XCI/XCZ files. Very simular to PFS0.
NVD0:
Mostly known as delta fragments. They contain a diff between the previous and current update. That way the user doesn't have to download the entire update with all the difference between the current version and the base game but only what changed since the last update. Delta fragments are large and have abslutely no value to be perserved as all update dumps already contain the full update with the applied delta fragments. They are removed unless --keep-delta is specified.
Block compression vs. Solid compression:
As you realized it's a trade-off between compression ratio and the ability to have random read access. Block compression installs around as fast as solid compression with current homebrews. Block compression can however make heavy use of multithreading as every block is compressed/decompressed independently so theoretically it could be much faster but the speed is limited by your SD-Card anyways. If you just want to archive your dumps solid compression is the way to go. If you want to in the future be able to play your dumps without first decompressing them block compression is the way to go. The higher of a block size you specify the slower the random read access speed gets but the better of a compression ratio you get so the difference between solid and block compression depends on the specified block size. But be carefully with setting the block size too high or using block compression gets pointless due to the game having to wait too long for every read access. The reason block compression is the default for XCZ is because players expect to be able to play XCI files without having to install them while NSP always had to be installed first which is why I choose to use solid compression as default for NSZ. Keep in mind that there currently exists no software to play block compressed games but with NSZ 5.0 I will develop a way to mount them on PC so they can be used by every Nintendo Switch emulator.
Implemented bit-identical FileEntryTable padding recreation
- The size of FileEntryTable including any non-standard zero padding at the end is kept
- The FileEntryTable size is kept even if NDV0 files got removed as they are by default if --keep-delta isn't specified.
- The padding at the end of FileEntryTable will increase in this case
- Use --remove-padding if you don't want to keep the FileEntryTable size and instead make it the lowest possible size according to the no-
intro standard. Keep in mind that if your input is already no-intro compliant this option is not needed. - This fixes rare cases of non-standard NSP files not being able to be bit identically recreated due to the removal of non-standard FileEntryTable zero padding as reported by @nitro322
@Nynrah Let's try to make things clearer for you. Please don't hesitate to ask if something is still unclear to you.
PFS0 (Partition File System):
The file system used in NSP/NSZ files. See it like NTFS/ExFAT/Fat32 but specially made for NSP files.
HFS0 (SHA-256 File System):
The file system used in XCI/XCZ files. Very simular to PFS0.
NVD0:
Mostly known as delta fragments. They contain a diff between the previous and current update. That way the user doesn't have to download the entire update with all the difference between the current version and the base game but only what changed since the last update. Delta fragments are large and have abslutely no value to be perserved as all update dumps already contain the full update with the applied delta fragments. They are removed unless --keep-delta is specified.
Block compression vs. Solid compression:
As you realized it's a trade-off between compression ratio and the ability to have random read access. Block compression installs around as fast as solid compression with current homebrews. Block compression can however make heavy use of multithreading as every block is compressed/decompressed independently so theoretically it could be much faster but the speed is limited by your SD-Card anyways. If you just want to archive your dumps solid compression is the way to go. If you want to in the future be able to play your dumps without first decompressing them block compression is the way to go. The higher of a block size you specify the slower the random read access speed gets but the better of a compression ratio you get so the difference between solid and block compression depends on the specified block size. But be carefully with setting the block size too high or using block compression gets pointless due to the game having to wait too long for every read access. The reason block compression is the default for XCZ is because players expect to be able to play XCI files without having to install them while NSP always had to be installed first which is why I choose to use solid compression as default for NSZ. Keep in mind that there currently exists no software to play block compressed games but with NSZ 5.0 I will develop a way to mount them on PC so they can be used by every Nintendo Switch emulator.
Last edited by nicoboss,
Hi,
I have a plan to replace all my nsp to nsz, however I realize that nsz only has advantage on small to medium filesize. I did with level 22, long and block compression.
I extract the nsz and see that only the main nca has been compressed to ncz, the other has not. I wonder why not all of them. Some big files have multiple big nca and in this circumstance nsz cannot save much.
I have a plan to replace all my nsp to nsz, however I realize that nsz only has advantage on small to medium filesize. I did with level 22, long and block compression.
I extract the nsz and see that only the main nca has been compressed to ncz, the other has not. I wonder why not all of them. Some big files have multiple big nca and in this circumstance nsz cannot save much.
Nah, the average saving space percent is around ~40% across all the Switch Eshop library (it's ~30 TB for now) while using NSZ. That's much.
Last edited by mathew77,
How much can you save, for an example of Xenoblade Chronicles 3.
On my side it saves from 13.7 GB to 13.2 GB
May be I did something wrong
Similar threads
-
- Article
