Homebrew RELEASE Ryjin: A NSZ Converter Mod

nicoboss

Well-Known Member
Member
Joined
Feb 1, 2019
Messages
132
Trophies
0
Age
26
XP
1,196
Country
Switzerland
that game could never work to mount as a xcz since it would eat up all memory just to decompress it.
It wouldn't be hard to implementing mounting a block compressed xciz. Block compression (option -b using the nsz python script) allows random read access by compressing the game in independent chunks. We even already provide a block decompressor with random read access: https://github.com/nicoboss/nsz/blob/master/compressor/nut/BlockDecompressorReader.py
The only problem is that there is no open source xci mounter so we have to rellay on others implementing it or add nsz support to CFW which we can't do without help from somebody very familiar with the OS. As far I'm aware the solid compressed nsz was already added to SX Installer v3.0.0 so it might happen.
 
  • Like
Reactions: lordelan

linuxares

The inadequate, autocratic beast!
Global Moderator
Joined
Aug 5, 2007
Messages
13,299
Trophies
2
XP
18,132
Country
Sweden
It wouldn't be hard to implementing mounting a block compressed xciz. Block compression (option -b using the nsz python script) allows random read access by compressing the game in independent chunks. We even already provide a block decompressor with random read access: https://github.com/nicoboss/nsz/blob/master/compressor/nut/BlockDecompressorReader.py
The only problem is that there is no open source xci mounter so we have to rellay on others implementing it or add nsz support to CFW which we can't do without help from somebody very familiar with the OS. As far I'm aware the solid compressed nsz was already added to SX Installer v3.0.0 so it might happen.
If you know how (not sure how much you know about the Switch), why not make a opensource xci loader?
 
  • Like
Reactions: Chocola

nicoboss

Well-Known Member
Member
Joined
Feb 1, 2019
Messages
132
Trophies
0
Age
26
XP
1,196
Country
Switzerland
If you know how (not sure how much you know about the Switch), why not make a opensource xci loader?
XCI loaders are hard to write as the game cart slot was believed to be the main target for piracy. The LOTUS3 chip is still unbroken to this day and not much information can be found about it except: https://switchbrew.org/wiki/Gamecard_ASIC. SX easy way of implementing their XCI loader is highly illegal and could never be done like this in an open source project. In the end modifying FS would probably be the easiest way to get XCI loading to work but in that case adding NSZ support to CFW and maybe even NSZ loading would be easier.
In addition there's also a moral issue with XCI loaders. Assuming somebody owns a cartage, most people don’t see a good reason why he should mount his XCI dump over just inserting his cartage. This makes the majority considering it as piracy tool and so hurting the developer’s reputation a lot. Because of this support from other developers will be extremely limited as they refuse to help developing a piracy tool. Writing an XCI loader completely without any help seems unfeasible - at least for me. Please also keep in mind I developed nsZip with legitimate dumps and emulators in mind and don't want my work to be associated with piracy in any way.
Anyways if there will ever appear an open source XCI loader I will implement XCZ support for sure.
 
Last edited by nicoboss,
  • Like
Reactions: lordelan

linuxares

The inadequate, autocratic beast!
Global Moderator
Joined
Aug 5, 2007
Messages
13,299
Trophies
2
XP
18,132
Country
Sweden
What I have heard thanks to Atmosphere, all the calls are there. There is no need to modify the gamecart firmware anymore. But since I don't know jack how to code nor does it have to do with this topic. I might have gotten wrong info. And yes, it would essentially a piracy tool even if I see usefulness for homebrew games etc once it's production life is over. Man if I would like to install all my games I actually own (physical). I would need a couple of sd cards :P
 

nicoboss

Well-Known Member
Member
Joined
Feb 1, 2019
Messages
132
Trophies
0
Age
26
XP
1,196
Country
Switzerland
What I have heard thanks to Atmosphere, all the calls are there. There is no need to modify the gamecart firmware anymore. But since I don't know jack how to code nor does it have to do with this topic. I might have gotten wrong info. And yes, it would essentially a piracy tool even if I see usefulness for homebrew games etc once it's production life is over. Man if I would like to install all my games I actually own (physical). I would need a couple of sd cards :P
Yes, you are probably right. Atmosphere definitely made writing XCI loaders much easier but never looked into it. Do you remember the first method used for piracy? LayeredFS injection didn't require you to install a game. You modded a whole game into an installed one. With the knowledge gained over time compatibility would no longer be an issue. Instead of implementing an XCI loader I see a similar approached leading in directly launching NSZ's without installing them more likely especially if we ever implement NSZ into CFW.
 
Last edited by nicoboss,
  • Like
Reactions: linuxares

lordelan

Well-Known Member
Member
Joined
Jan 4, 2015
Messages
5,787
Trophies
1
Age
44
XP
6,516
Country
Germany
Instead of implementing an XCI loader I see a similar approached leading in directly launching NSZ's without installing them more likely especially if we ever implement NSZ into CFW.
This sounds awesome (if that's really possible).

However I wanted to clarify one thing at that XCI = piracy debate:
I'm one of the few people that highly prefer digital games over physical games.
The main reason for using USB Loader GX back then on the Wii was not that I was able to use hundred of pirated games but because I had not to get my lazy ass off the couch to change discs everytime.
Honestly I don't get it why people love cartridges so much. Especially nowadays when you have to download pretty big updates for your cartridge games (!) nonetheless and you still have to carry all of them around and swap them whenever you want to play something else.
Apart from reselling games (which I never cared for as I'm not only a gamer but a collector and don't want to sell my games) the only benefit of cartridges would be that they are still usable in 50 years when all eShop servers are down.
But even then you're having your "fun" with v1.0.0 of every game without any updates or DLCs.
That's why all our CFW opportunities are so fantastic. We can dump everything and keep it digital like forever without caring for Nintendo's servers.

The big downside of digital games, especially Nintendo ones: They are expensive. While I can get all Nintendo game cartridges cheaper than their digital counterparts, especially used from ebay, it's pretty rare Nintendo is doing any really good sale deals like Valve does in Steam almost everyday.

So dumping cheap bought cartridges to XCIs is a pretty legit way of benefiting from all advantages.
 

rogras

Well-Known Member
Newcomer
Joined
Apr 2, 2009
Messages
80
Trophies
1
Age
113
Location
TR
XP
528
Country

Attachments

  • mami.JPG
    mami.JPG
    53.1 KB · Views: 143
  • Like
Reactions: dabu-sapper

GnK23

Well-Known Member
OP
Member
Joined
Apr 6, 2016
Messages
170
Trophies
0
Age
35
XP
671
Country
United States
Please be careful when using this tool on updates. so far I only encountered 1 game with a problem. Roof Rage[010088100DD42800][US][v65536].
The error code returned this line
line 97, in <module>
nut.compress(filePath, 17 if args.level is None else args.level, args.solid, args.bs, args.output, args.threads
IndexError: list index out of range
@nicoboss @blawar I also tried it with the latest commit and same error.
IndexError: list index out of range


EDIT: It now works. New update on the first page
 
Last edited by GnK23,

blawar

Developer
Developer
Joined
Nov 21, 2016
Messages
1,708
Trophies
1
Age
40
XP
4,311
Country
United States
Please be careful when using this tool on updates. so far I only encountered 1 game with a problem. Roof Rage[010088100DD42800][US][v65536].
The error code returned this line
line 97, in <module>
nut.compress(filePath, 17 if args.level is None else args.level, args.solid, args.bs, args.output, args.threads
IndexError: list index out of range
@nicoboss @blawar I also tried it with the latest commit and same error.
IndexError: list index out of range

try it with the 1.0 release: https://github.com/nicoboss/nsz/releases/tag/1.0

I compressed that update fine.
 
  • Like
Reactions: GnK23

nicoboss

Well-Known Member
Member
Joined
Feb 1, 2019
Messages
132
Trophies
0
Age
26
XP
1,196
Country
Switzerland
Please be careful when using this tool on updates. so far I only encountered 1 game with a problem. Roof Rage[010088100DD42800][US][v65536].
The error code returned this line
line 97, in <module>
nut.compress(filePath, 17 if args.level is None else args.level, args.solid, args.bs, args.output, args.threads
IndexError: list index out of range
@nicoboss @blawar I also tried it with the latest commit and same error.
IndexError: list index out of range
Your keys.txt file is outdated when you get this error. You should always upgrade your keys.txt file when upgrading to a new firmware as otherwise you can's decrypt all your dumped games. I admit the error message is quite confusing as it's caused by sectionFilesystems[key] at Fs/Nca.py on line 209. I will make the error message more understandable in the next release which will come later today or tomorrow.
 
  • Like
Reactions: lordelan and GnK23

GnK23

Well-Known Member
OP
Member
Joined
Apr 6, 2016
Messages
170
Trophies
0
Age
35
XP
671
Country
United States
try it with the 1.0 release: https://github.com/nicoboss/nsz/releases/tag/1.0

I compressed that update fine.

Thank you it now works. I updated my files and I posted the new one on the first page
Your keys.txt file is outdated when you get this error. You should always upgrade your keys.txt file when upgrading to a new firmware as otherwise you can's decrypt all your dumped games. I admit the error message is quite confusing as it's caused by sectionFilesystems[key] at Fs/Nca.py on line 209. I will make the error message more understandable in the next release which will come later today or tomorrow.

I am currently using the 9.0.1 keys.

I noticed that after comparing the files that I use in the modified build commit 53f7026 and the NSZ version 1.0 is the absence of the block compression inside the _init_.py
if useBlockCompression:
bytesToCompress = nspf.size-0x4000
blocksToCompress = bytesToCompress//blockSize + (bytesToCompress%blockSize > 0)
header = b'NCZBLOCK' #Magic
header += b'\x02' #Version
header += b'\x01' #Type
header += b'\x00' #Unused
header += blockSizeExponent.to_bytes(1, 'little') #blockSizeExponent in bits: 2^x
header += blocksToCompress.to_bytes(4, 'little') #Amount of Blocks
header += bytesToCompress.to_bytes(8, 'little') #Decompressed Size
header += b'\x00' * (blocksToCompress*4)
f.write(header)

I also encountered problem here:
if useBlockCompression:
buffer = partitions[partNr].read(blockSize)
while (len(buffer) < blockSize and partNr < len(partitions)-1):
partNr += 1
buffer += partitions[partNr].read(blockSize - len(buffer))
if len(buffer) == 0:
break
compressor = cctx.stream_writer(f)
compressor.write(buffer)
compressor.flush(zstandard.FLUSH_FRAME)
compressor.flush(zstandard.COMPRESSOBJ_FLUSH_FINISH)
compressedblockSizeList.append(f.tell() - blockStartFilePos)
blockID += 1
blockStartFilePos = f.tell()
else:
buffer = partitions[partNr].read(CHUNK_SZ)
while (len(buffer) < CHUNK_SZ and partNr < len(partitions)-1):
partNr += 1
buffer += partitions[partNr].read(CHUNK_SZ - len(buffer))
if len(buffer) == 0:
break
compressor.write(buffer)

decompressedBytes += len(buffer)
bar.update(len(buffer))

Thank you @blawar and @nicoboss for your reply
 

nicoboss

Well-Known Member
Member
Joined
Feb 1, 2019
Messages
132
Trophies
0
Age
26
XP
1,196
Country
Switzerland
Hey, hope I'm not being an idiot (it's pretty normal for me), but I get this:
View attachment 183556
Not enough memory just means you run out of RAM. You can reduce the amount of threads or enabling your page file to solve that issue. However the main reason probably is the huge memory leak discovered today leading to solid compressed NSZ requiring way more memory then needed: https://github.com/nicoboss/nsz/issues/13 so you could also wait like 12 hours for it to hopefully be fixed and hopefully merged into Ryjin.
 
Last edited by nicoboss,

Site & Scene News

Popular threads in this forum

General chit-chat
Help Users
  • No one is chatting at the moment.
    NinStar @ NinStar: It will actually make it worse