RELEASE nsZip - NSP compressor/decompressor to reduce storage

Discussion in 'Switch - Emulation, Homebrew & Software Projects' started by nicoboss, Feb 1, 2019.

  1. nicoboss
    OP

    nicoboss Advanced Member

    Newcomer
    4
    Feb 1, 2019
    Switzerland
    nsZip itself is currently windows only but the NSZ python script from https://github.com/nicoboss/nsz should be platform independent. Because NSZ will probably be the successor on NSPZ there's no reason not to use the python script.
     
    dreambottle likes this.
  2. skinnyBIGGS

    skinnyBIGGS Member

    Newcomer
    2
    Apr 2, 2019
    Canada
    Great work guys . I.ll wait until its ironed out abit more a d works with goldleaf i dont like tinfoil
     
  3. Mat37

    Mat37 GBAtemp Advanced Fan

    Member
    6
    Dec 3, 2016
    France
    Does it matter ?
    @nicoboss Will the new version have retro-compatibility with the NSPZ format ?
     
  4. Deathscreton

    Deathscreton GBAtemp Advanced Fan

    Member
    7
    Oct 1, 2009
    United States
    I can't seem to get nsZIP to work anymore. It just opens up CMD, then closes immediately. I'm planning on switching to NCZ, but all of my old NSPs are compressed in the old format.
     
  5. awankelam

    awankelam Newbie

    Newcomer
    1
    Oct 19, 2019
    Indonesia
    I got JIT error message just right after I click RUN.

    It says, "couldn't load file or assembly 'netstandard. Version = 2.0.0.0. Culture=neutral" Bla 3x

    What is wrong? What should I do?
     
  6. awankelam

    awankelam Newbie

    Newcomer
    1
    Oct 19, 2019
    Indonesia
    I got JIT error message just right after I click RUN.

    It says, "couldn't load file or assembly 'netstandard. Version = 2.0.0.0. Culture=neutral" Bla 3x

    What is wrong? What should I do?
     
  7. awankelam

    awankelam Newbie

    Newcomer
    1
    Oct 19, 2019
    Indonesia
    I got JIT error message just right after I click RUN.

    It says, "couldn't load file or assembly 'netstandard. Version = 2.0.0.0. Culture=neutral" Bla 3x

    What is wrong? What should I do?

    — Posts automatically merged - Please don't double post! —

    Sorry for multipost. I got signal trouble here. Please delete the rest comments
     
  8. nicoboss
    OP

    nicoboss Advanced Member

    Newcomer
    4
    Feb 1, 2019
    Switzerland
    The next stable release will still have NSPZ support. In the end block compressed NSZ is very similar to the beautiful NSPZ format just without all the unnecessarily complexity which made NSPZ unfeasible for other software to implement. I might remove NSPZ/XCIZ support in the future if I see that there's no point in keeping it to not confuse people.

    Differences between NSZ and NSPZ:

    NSPZ/XCIZ:
    - GitHub Project: https://github.com/nicoboss/nsZip
    - Always uses Block compression allowing random read access to play compressed games in the future
    - Decrypts the whole NCA
    - Trims NDV0 fragments to their header and reconstructs them
    - Only supported by nsZip and unfortunately doesn't really have a future

    NSZ/XCZ:
    - GitHub Project: https://github.com/nicoboss/nsz
    - Uses solid compression by default. Block compression can be enabled using the -B option. Block compression will be the default for XCZ
    - Decrypts all sections while keeping the first 0x4000 bytes encrypted. Puts informations needed to encrypt inside the header.
    - Deleted NDV0 fragments as they have no use for end users as they only exist to save CDN bandwidth
    - Already widely used. Supported by Tinfoil, SX Installer v3.0.0 and probably a lot of other software in the future

    You don't have to use tinfoil. SX Installer v3.0.0 supports NSZ too. Solid compressed NSZ files are already ironed out very well and so widely used that it's unlikely a compatibility breaking change will ever be done. Goldfleaf support won't happen because of https://github.com/XorTroll/Goldleaf/issues/417 - obviously somebody could just fork Goldleafe and implement NSZ as it's open source but don't expect the official version to ever support NSZ.

    The portable preview versions come without any installer so missing requirements like the .NET framework will prevent nsZip from working. Try https://github.com/nicoboss/nsZip/files/3459201/nsZip_v2.0.0_debug_v1_Issue3.zip from built at 2nd August 2019 but keep in mind that this is NOT an official version and made to diagnose an issue. Even though this version comes with an installer it WON'T receive any updates and will contain bugs.
     
    Last edited by nicoboss, Oct 19, 2019
  9. Deathscreton

    Deathscreton GBAtemp Advanced Fan

    Member
    7
    Oct 1, 2009
    United States
    Nico, when I try the linked application, I get the same error I got form the other versions. The only version that actually works for me from your github is your v1 files. I've attached a screenshot of what happens when I load up the application. That CMD Prompt screen comes up, stays for a few seconds, then closes and nothing happens at all. Task Manager shows it opening, then closing suddenly. I've tried launching it through PS and CMD. Also just tried launching it via File Explorer as an Admin as well. No difference.
     

    Attached Files:

    Last edited by Deathscreton, Oct 19, 2019
  10. mrdude

    mrdude GBAtemp Advanced Fan

    Member
    7
    Dec 11, 2015
    A word of warning for anyone that uses this.

    I had a folder called 'compressed' on my desktop - it contained many files and folders from stuff I had downloaded and programs I had been working on. This app deleted that folder and all of it's contents without any warning......so use carefully - probably best on a drive or sandbox/virtual machine - until it has these bugs sorted out.
     
  11. nicoboss
    OP

    nicoboss Advanced Member

    Newcomer
    4
    Feb 1, 2019
    Switzerland
    Sorry! Never expect anybody to ever set the temp folder to a location where other folders with the name "decrypted" "encrypted" or "compressed" exists. nsZip needs these folders to store its temporary files and deletes them to clean them up afterwards. The behavior isn't really a bug but Desktop shouldn’t be the default location for temporary files (no idea why it’s desktop on latest preview) and there be a warning when selecting a temporary folder that already contains subfolders with such names. nsZip uses standard delete so files deleted that way can be recovered with any file recovery program as long the clusters where this file were stored aren’t already overwritten with a new file. Other than that, nsZip isn't dangerous in any way - take a look at its source code it you don't believe me.
     
    Last edited by nicoboss, Oct 19, 2019
  12. awankelam

    awankelam Newbie

    Newcomer
    1
    Oct 19, 2019
    Indonesia
    I use this debug version. But when in the progress I encounter issue that says the file is in use.. I used the stable version too, though .. But its still not going well.
     
  13. awankelam

    awankelam Newbie

    Newcomer
    1
    Oct 19, 2019
    Indonesia
    This for the detail... How can I fix it?
     

    Attached Files:

  14. nicoboss
    OP

    nicoboss Advanced Member

    Newcomer
    4
    Feb 1, 2019
    Switzerland
    Does this happen to every game you try to compress or only for a specific one? Very interesting. Never seen that one before. Very wired how it crashes in the middle of compressing blocks as there is nothing that really could go wrong there. Are you sure you don't run out of HDD space or RAM? Is your OS x64 or x86? What game are you compressing when this issue occurs?
     
    awankelam likes this.
  15. awankelam

    awankelam Newbie

    Newcomer
    1
    Oct 19, 2019
    Indonesia
    Its happened almost every nsp that I've tried. From 6 files only succed 1 file with very small size, about 70mb. The others that failed is about 100mb to 10gb. Is in process of conversion need so much space? I read on explanation said that need space about 5 times of file that we want to convert. But I tried the 100mb file still failed with such kind of error, "cannot access file .. Because used by another process" . FYI, I alocate the output & temp file on non C: disk that still have 20gb space left. Because on C: its only about 2gb left.
     
  16. nicoboss
    OP

    nicoboss Advanced Member

    Newcomer
    4
    Feb 1, 2019
    Switzerland
    I tried and can't reproduce any of your issues. I will continue investigate the reported issues and will fix them as soon I can reproduce them.
    Why don't you just use https://github.com/nicoboss/nsz instead? NSZ is the successor of the NSPZ File format and until NSZ is implemented into nsZip I don't really see any reason to use it.
     
    awankelam likes this.
  17. RisingHopperZeroOne

    RisingHopperZeroOne Advanced Member

    Newcomer
    2
    Jul 31, 2019
    Australia
    Is there a decompress script for nsz or is it compress only?

    EDIT: Saw how to decompress, question answered.
     
    Last edited by RisingHopperZeroOne, Oct 21, 2019
  18. razinx

    razinx Member

    Newcomer
    4
    Aug 17, 2009
    Canada
    Not having any luck with this tool. Any XCI I attempt seems to give all "Good" as its checking it, but then at the end I get,

    at LibHac.Nca.DecryptHeader()
    at LibHac.Nca..ctor(Keyset keyset, IStorage storage, Boolean leaveOpen)
    at nsZip.LibHacControl.ProcessNca.Process(IFile inFile, IFile outFile, Boolean verifyBeforeDecrypting, Keyset keyset, Output Out) in C:\Users\nico\Documents\nsZip\LibHacControl\ProcessNca.cs:line 15
    at nsZip.LibHacControl.ProcessXci.Decrypt(String inputFilePath, String outDirPath, Boolean verifyBeforeDecrypting, Keyset keyset, Output Out) in C:\Users\nico\Documents\nsZip\LibHacControl\ProcessXci.cs:line 76
    at nsZip.TaskLogic.CompressXCI(String xciFile) in C:\Users\nico\Documents\nsZip\TaskLogic.cs:line 138
    at nsZip.MainWindow.RunTask() in C:\Users\nico\Documents\nsZip\MainWindow.xaml.cs:line 182
    Unable to decrypt NCA header.
    Key Type: Common
    Key Name: header_key


    What am I missing? I have the prod.key in the .switch directory, so not sure what else I have to do to get this running.

    TIA
     
  19. nicoboss
    OP

    nicoboss Advanced Member

    Newcomer
    4
    Feb 1, 2019
    Switzerland
    As far I remember https://github.com/shchmue/Lockpick_RCM/releases should dump it. If not you can calculate it like this:
    header_key = header_key_source + kek_source (pkg2->ini1->FS) + aes_kek(pkg1->TZ)/key_gen_source(pkg2->ini1->spl) + master_key_00
    master_key_00 = master_key_source (pkg1ldr) + [decrypted keyblob = keyblob_key (pkg1ldr key source + sbk + tsec) + enc keyblob (boot0)]

    Anyways only use nsZip for NSPs and try to avoid using it for XCIs. It's only there as proof of concept and its implementation has a mistake leading in some metadata being lost. It was planned to be properly implemented in nsZip v2.0 an its work nearly completed but then NSZ appeared. The new NSZ/XCZ will be the successor of NSPZ/XCIZ and supported by homebrew so all focus is currently at this new file format. nsZip 2.0 will get NSZ/XCI support but the NSZ reference project has way higher priority right now. The new file format also still hasn’t XCI support yes but will probably get it very soon. See https://gbatemp.net/threads/nsz-homebrew-compatible-nsp-xci-compressor-decompressor.550556/
     
  20. razinx

    razinx Member

    Newcomer
    4
    Aug 17, 2009
    Canada
    Got it, thanks for the reply!
     
  21. Mat37

    Mat37 GBAtemp Advanced Fan

    Member
    6
    Dec 3, 2016
    France
    Does it matter ?
    When trying to decompress NSZ files (level 22 block compression) i run out of RAM for big games and can't decompress them. Is there a way around this ?
     
  22. nicoboss
    OP

    nicoboss Advanced Member

    Newcomer
    4
    Feb 1, 2019
    Switzerland
    nsZip doesn't even support the new NSZ file format yet. Do you mean https://github.com/nicoboss/nsz? If yes please create an issue under https://github.com/nicoboss/nsz/issues/new and I will look into it. Decompression definately shouldn't use much RAM and there's probably a memory leak somewhere.
    Please post questions related to NSZ inside https://gbatemp.net/threads/nsz-homebrew-compatible-nsp-xci-compressor-decompressor.550556/
     
Quick Reply
Draft saved Draft deleted
Loading...