Hacking 3DS rom dump info

[Jayro]

I edited it out, sorry about that. I was just trying to prove that they weren't hard to find is all. I meant no rule breakage.

 

[raulpica]

Lol, all I know is that posting links to such sights is a no no gbatemp, might want to edit it out. Just warning you.

Friendly tip: if you report something, DON'T quote it, otherwise the infringing content will also be in YOUR post.

Thanks.

 

[Pong20302000]

Lol, all I know is that posting links to such sights is a no no gbatemp, might want to edit it out. Just warning you.

rom trimming is perfectly fine

edit:
maybe i missed something as an admin may have been halfway through editing LOL

So where can we find a 3DS rom trimmer? I don't want these huge untrimmed files, I like them small. (I trimmed my DS roms myself, keeping the Wi-Fi functionality in all of them.) My R4i 3DS Gold loads any trimmed DS rom I throw at it.

you cant trim the roms
the best way is to keep them packed in the .rar files for now as that would be roughly the trimmed size anyway

 

[Jayro]

Maybe emulators and rom loaders will eventually add support for 7zip. But nah, that would make too much sense.

 

[Rydian]

Maybe emulators and rom loaders will eventually add support for 7zip. But nah, that would make too much sense require them to add in another library which increases the size and complexity of the compile which means more time spent each test build, and then they'd also have to deal with block versus individual compression and if a file has been compressed as a delta difference they would need to uncompress multiple files at once to load a single version of the ROM from the archive, and then they'd have to abide by both the GPL and the unRAR licenses/restrictions in order to distribute the libraries within the program anyways.

Something common, relatively-simple, and non-restricting like ZIP works.

7-zip is an archive and compression format and unsuitable for storing resources.

 

[andibad]

Something common, relatively-simple, and non-restricting like ZIP works.

7-zip is an archive and compression format and unsuitable for storing resources.

7-zip is used a lot cpu usage and memory .... than zip or rar.
but sometimes 7-zip is suitable for storing resources on limited space than rar or zip. and is opensource.

 

[porkiewpyne]

7-zip is used a lot cpu usage and memory .... than zip or rar.
but sometimes 7-zip is suitable for storing resources on limited space than rar or zip. and is opensource.

And as far as I know, is used to a lesser extent compared to .zip and .rar. And you have to remember that some people are ....er... not that tech-savvy. Yea, I do know people who do not know what .rar is.

 

[Rydian]

but sometimes 7-zip is suitable for storing resources on limited space than rar or zip.

ROMs contain a lot of non-repeating data so the best compress comes from storing multiple versions of a RAM via delta differences, but that's a more complex operation to extract. They need to extract the base file and the changes, then apply them in RAM (opposed to a single extract).

and is opensource.

Doesn't mean "free to do whatever with", still gotta' abide by the licensing.

 

[elisherer]

7zip has the advantage of streaming...so it might be easier to run a rom from inside a 7z file than from inside a zip file.

 

[Thorhian]

Friendly tip: if you report something, DON'T quote it, otherwise the infringing content will also be in YOUR post.

Thanks.

Oh thanks XD Good thing to remember XD

 

[Rydian]

7zip has the advantage of streaming...so it might be easier to run a rom from inside a 7z file than from inside a zip file.

From what I read, the streaming ability is just container-ability for multiple files, and doesn't actually do compression (since good compression needs to analyze the whole file/collection to discern the best patterns). In addition you don't actually run files inside of archives, they need to be extracted into RAM (or onto disk) anyways.

 

[elisherer]

From what I read, the streaming ability is just container-ability for multiple files, and doesn't actually do compression (since good compression needs to analyze the whole file/collection to discern the best patterns). In addition you don't actually run files inside of archives, they need to be extracted into RAM (or onto disk) anyways.

If that would have been the case then there was no benefit on using any of the compressions over the others and just use uncompression before play. 7z files can start reading chunks from a file inside the archive without decompressing all of it (as I did in 3dsexplorer... I can open the header of a rom file inside a 7z file, doing that on a zip file is more difficult).

EDIT: i think gz is even better.

 

[Rydian]

If that would have been the case then there was no benefit on using any of the compressions over the others and just use uncompression before play. 7z files can start reading chunks from a file inside the archive without decompressing all of it (as I did in 3dsexplorer... I can open the header of a rom file inside a 7z file, doing that on a zip file is more difficult).

EDIT: i think gz is even better.

From what I read the 7z format does use solid/block compression.

I guess the concern is if the user has a collection of 7z ROMs (each archive containing the various versions of a single game release), or if they have their whole collection in one archive.

 

[elisherer]

From what I read the 7z format does use solid/block compression.

I guess the concern is if the user has a collection of 7z ROMs (each archive containing the various versions of a single game release), or if they have their whole collection in one archive.

You are right, and that's the emulators makers' job to decide wisely. I suggest packing each in its own archive (as 3dsexplorer reads them today).

 

[Jayro]

This is why I prefer ROM trimmers to compressed archives... No extraction needed, and usually a smaller file. .__.

But I always trim my own ROMs, so I KNOW I have a good ROM. I never batch trim either. I trim, test in Desmume, test on my R4i 3DS Gold, THEN decide to either keep, or re-trim the ROM. So far I haven't had any trouble with the specific ROM trimmers I use for GBA and DS.

 

[Rydian]

Yeah for archiving 7-zip with delta differences for multiple versions of a ROM is great (I like ROM sites that do that, so I can grab an exact revision for a patch or investigation), but for saving space on the device, trimming is best nowadays. It'll use less space per-ROM at load time anyways since you don't need to have duplicates (compressed and uncompressed), load time is faster, etc.

 

[elisherer]

We still dont know id the 3ds support trimming because it seems it has a lot going on the power of two because of the encryption. the trimmed rom would have to be encrypted too... but all that are just speculations.. i'll stop talking about that now.

 

[Ris312]

in a 3DS rom, at offset 0x300 is a 4 byte value stored in little endian that is the used ROM size in bytes,
on the few 3DS roms that I've checked, they all seem to be filled with FF's after that point.

so in theory you can make a rom trimmer by:
1. check the 4 bytes offset 0x100, if it is 4E435344 in hex (NCSD) then its most likely a 3DS rom
(if not then give an error like 'Not a 3DS Rom')
2. reading 4 bytes at offset 0x300 and reverse the bytes (note: its stored as little-endian)
for example: 00 2C BD 1B = 1B BD 2C 00
3. convert that hex value into decimal to get the size in bytes. (example: 1B BD 2C 00 = 465,382,400 bytes)
4. cut the file at that point (or to be safe, check if all the data after that point is FFs, if not then give an error (e.g unable to be trimmed)

To restore the rom back to normal:
1. read 4 bytes at offset 0x104 and reverse the bytes (note: stored as little-endian)
(example: 00 00 10 00 = 00 10 00 00)
2. convert it to decimal (00 10 00 00 = 1,048,576)
3. then multiply by 512 which is the sector size (is it fixed to 512 or is there is an offset that contains the sector size?)
4. that will give you the full rom size in bytes (1,048,576 * 512 = 536,870,912)
5. then you just pad the file with FF's so it is the size of the full rom in bytes.
e.g 536,870,912 (Full Rom Size) minus 465,382,400 (Trimmed Size) means you need to add: 71,488,512 bytes of FF's.

The encrypted part of a 3DS Rom won't compress much, its only the FF section at the end of the rom that does,
because of the encryption, xdelta differences won't work and will end up being the full used size of the rom,
and merging 'region variant' games in an archive with solid compression won't help compression either
and will end up being the size of the ALL the games used rom size as only the FF part will compress.
(Unless a way to decrypt and then re-encrypt the roms is found, xdeltas and merging roms won't save any space)


After you trim a 3DS Rom you'll find that the trimmed size is almost the same as the compressed filesize,
So you could store 3DS roms trimmed to save a bit of space without having to have them compressed.
(and in the future if the roms needed to be full size, then you can just restore them to the original size)

 

[PsyBlade]

seems to work fine for me
the trimmed files were actually smaller that the ones compressed with gzip and bzip2


quick and dirty 3ds rom trimmer:

#!/usr/bin/env python
# -*- coding: utf-8 -*-
 
import sys, io, struct
 
print "trimming {} Bytes".format(sys.argv[1]),
rom = io.open(sys.argv[1], "rb+")
rom.seek(0x100)
assert rom.read(4).decode("ascii") == "NCSD"
fullsize = struct.unpack("<I",rom.read(4))[0] << 9
rom.seek(0x300)
trimsize = struct.unpack("<I",rom.read(4))[0]
rom.seek(trimsize)
while rom.tell() < fullsize:
    data = rom.read1(1<<16)
    if data == "": break
    assert min(data) == "\xff"
assert rom.read1(1<<8) == ""
assert rom.tell() <= fullsize
rom.truncate(trimsize)
rom.close()
print "to {} Bytes".format(trimsize)

and untrimmer:

#!/usr/bin/env python
# -*- coding: utf-8 -*-
 
import sys, io, struct
 
print "untrimming {}".format(sys.argv[1]),
rom = io.open(sys.argv[1], "rb+")
rom.seek(0x100)
assert rom.read(4).decode("ascii") == "NCSD"
fullsize = struct.unpack("<I",rom.read(4))[0] << 9
rom.seek(0x300)
trimsize = struct.unpack("<I",rom.read(4))[0]
rom.seek(trimsize)
assert rom.read1(1<<8) == ""
assert rom.tell() == trimsize
while rom.tell() < fullsize:
    rom.write("\xff"*min(1<<16,fullsize-rom.tell()))
rom.close()
print "to {} Bytes".format(fullsize)

 

[abdelmajidtolba]

Now ,guys,guys wait a second...
Explain in Human language , i stopped coding when came across pointers so i dont have a clue on how this piece code should be used (saved as bat file,recompiled ...etc)
mind to share a little of your wisdom.

(I trimmed my DS roms myself, keeping the Wi-Fi functionality in all of them.)

if trimming removes fucntionnality i hope you rot in hell !!!
For now the hole 3ds roms are 148 7z packed.
If trimming removes 30-40 Gb im all for it ,if not ,well ill see as one of various ''Mess with Original Backup'' LOL.

@Pong : expect a huge pack ill made will all icons ,High res covers for the 3ds (around 200 mb).
For a feature rom manager with high res cover (maybe??),cause i dont know about you guys but we reached 2013 (56K modem and dinosaurs are long gone) and my eyes hurts trying to read 3ds back or front covers.

@Ris312
Where have i seen this pseudo? an impression of Deja vu .