Hacking About CRC32 and Release List

[pekalicious]

Hello everyone,

I downloaded the ADVANsCEne release list which is an XML based release list. I understand that a number of tools use this file.

Here is an example game record:

CODE
ÂÂÂÂ3058
ÂÂÂÂ3058
ÂÂÂÂTomb Raider - Underworld
ÂÂÂÂEeprom - 4 kbit
ÂÂÂÂ134217728
ÂÂÂÂEidos Interactive
ÂÂÂÂ1
ÂÂÂÂXenoPhobia
ÂÂÂÂ4099
ÂÂÂÂ
ÂÂÂÂÂÂÂÂ63AB9330
ÂÂÂÂ
ÂÂÂÂ40BDAC38
ÂÂÂÂ09BC14BB
ÂÂÂÂ2980
ÂÂÂÂ730


I have ask two questions:

A) How exactly do I get a ROM CRC32?
I know I can calculate the file's CRC32 with a standard calculator library, but I think that the ROM has it's CRC32 somewhere in it's header. I am looking for a specification that tells me how exactly can I parse the one in the header. I have found a number of specification (such as this) but I can't understand anything. :/

As I understand it, if the CRC in the header and the CRC that I calculate are the same, then the ROM has not been hacked/trimmed. Is this true?

If there is anyone here who has parsed the CRC from a ROM, please explain how this is done in pseudo-code. I've been reading many source codes and still don't get it.

B) What are imCRC1 and imCRC2?
Are these the cover and screenshots?
Do I combine the imURL found in the start of the release list with these numbers?
i.e. "http://www.retrocovers.com/offline/imgs/ADVANsCEne_NDS/" + imCRC1 + ".jpg" or ".png" or ".something"
None of the above return a result.

If they are not, where do I find cover images?

Thank you in advance.[/url]

 

[SamAsh07]

Wow...wish I could help, all I can say is "What Is This??"

 

[Nollog]

imCRC1 and 2 do seem like the image crc's.
You can get a crc32 of a rom (or any file) by doing some maths that I never learned, or using something like hashcalc in windows.

 

[pekalicious]

imCRC1 and 2 do seem like the image crc's.
You can get a crc32 of a rom (or any file) by doing some maths that I never learned, or using something like hashcalc in windows.

I use a library that does the crc32 calculations and it works fine. The thing is, does the ROM itself includes a CRC32 in its header? Now that I think of it, maybe not.

So what I am looking for now is mainly the covers of the games. Like RToolDS. Does this tool have a server of its own that hosts the images, or does it use a standard Release List that includes these images? If the latter, where do I find the specs?

 

[Maikel Steneker]

imCRC1 and 2 do seem like the image crc's.
You can get a crc32 of a rom (or any file) by doing some maths that I never learned, or using something like hashcalc in windows.

I use a library that does the crc32 calculations and it works fine. The thing is, does the ROM itself includes a CRC32 in its header? Now that I think of it, maybe not.
No, it doesn't. A CRC32 value is called a checksum. It is designed to generate a unique code for every different file. Theoretically, two files can have the same CRC32. But when you change only one bit from a file, the whole checksum changes. Because of that, it's a pretty good way of determining if you have the correct rom.

So what I am looking for now is mainly the covers of the games. Like RToolDS. Does this tool have a server of its own that hosts the images, or does it use a standard Release List that includes these images? If the latter, where do I find the specs?

The images are on the ADVANsCEne website. OfflineList downloads them from there. If you take a look at the DAT you'll find this line:
CODE

http://www.retrocovers.com/offline/imgs/ADVANsCEne_NDS/

Inside this folder (on the server) there are a couple of other folders, each containing 1000 images (for 500 roms). You want to take a look at the first one, for example, you'd have to look inside the folder 1-500. Now if you want to see the boxart, you'd be looking for a file called 1a.png. For the screenshot, you'll need 1b.png. This works for every rom number.

And this is the result:
http://www.retrocovers.com/offline/imgs/AD...DS/1-500/1a.png
http://www.retrocovers.com/offline/imgs/AD...DS/1-500/1b.png

Remember that the numbers correspond to the DAT including XXXX releases. The first release in here is Metroid Prime Hunters: First Hunt. It's nuked (because it's a demo) so in the comment field it says XXXX, however for OfflineList the number is still 1.

I hope that helped you

 

[pekalicious]

imCRC1 and 2 do seem like the image crc's.
You can get a crc32 of a rom (or any file) by doing some maths that I never learned, or using something like hashcalc in windows.

I use a library that does the crc32 calculations and it works fine. The thing is, does the ROM itself includes a CRC32 in its header? Now that I think of it, maybe not.
No, it doesn't. A CRC32 value is called a checksum. It is designed to generate a unique code for every different file. Theoretically, two files can have the same CRC32. But when you change only one bit from a file, the whole checksum changes. Because of that, it's a pretty good way of determining if you have the correct rom.

So what I am looking for now is mainly the covers of the games. Like RToolDS. Does this tool have a server of its own that hosts the images, or does it use a standard Release List that includes these images? If the latter, where do I find the specs?

The images are on the ADVANsCEne website. OfflineList downloads them from there. If you take a look at the DAT you'll find this line:
CODE

http://www.retrocovers.com/offline/imgs/ADVANsCEne_NDS/

Inside this folder (on the server) there are a couple of other folders, each containing 1000 images (for 500 roms). You want to take a look at the first one, for example, you'd have to look inside the folder 1-500. Now if you want to see the boxart, you'd be looking for a file called 1a.png. For the screenshot, you'll need 1b.png. This works for every rom number.

And this is the result:
http://www.retrocovers.com/offline/imgs/AD...DS/1-500/1a.png
http://www.retrocovers.com/offline/imgs/AD...DS/1-500/1b.png

Remember that the numbers correspond to the DAT including XXXX releases. The first release in here is Metroid Prime Hunters: First Hunt. It's nuked (because it's a demo) so in the comment field it says XXXX, however for OfflineList the number is still 1.

I hope that helped you

Awesome! Thank you so so so much!

But what about the other tags? What is imCRC1? And duplicateID? Where are the specs for the XML?

 

[Maikel Steneker]

Awesome! Thank you so so so much!

But what about the other tags? What is imCRC1? And duplicateID? Where are the specs for the XML?

I don't know where the specs are, I figured them out myself. imCRC1 is the CRC32 value for the first image, imCRC2 is for the second image (1a and 1b in the previous example). The duplicateID is a number used to identify a game, so you can bundle the same games from another region in the same archive.

So, if this was the release list:
0001 - Game 1 (JP)
0002 - Game 2 (EU)
0003 - Game 1 (US)
0004 - Game 3 (JP)
0005 - Game 2 (JP)

The first game (Game 1) would get duplicateID 1. The second one would get ID 2. The third one (again Game 1) would get ID 1, because these are the same games from another region. Basically, the first release, no matter the region, of a game, gets the next duplicateID. There's an option in OfflineList to use these numbers to put the same games from all regions in one archive. In the case of 7Z, this saves a lot of space.

 

[pekalicious]

I don't know where the specs are, I figured them out myself. imCRC1 is the CRC32 value for the first image, imCRC2 is for the second image (1a and 1b in the previous example).

Well, the imCRC1 is really the CRC of the image, right? Because it's a hash code and not a number. Here is an example:

CODEÂÂÂÂÂÂÂÂ
ÂÂÂÂÂÂÂÂÂÂÂÂ2
ÂÂÂÂÂÂÂÂÂÂÂÂ2
ÂÂÂÂÂÂÂÂÂÂÂÂElectroplankton
ÂÂÂÂÂÂÂÂÂÂÂÂNone
ÂÂÂÂÂÂÂÂÂÂÂÂ16777216
ÂÂÂÂÂÂÂÂÂÂÂÂNintendo
ÂÂÂÂÂÂÂÂÂÂÂÂ7
ÂÂÂÂÂÂÂÂÂÂÂÂTrashman
ÂÂÂÂÂÂÂÂÂÂÂÂ256
ÂÂÂÂÂÂÂÂÂÂÂÂ
ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ94767CD4
ÂÂÂÂÂÂÂÂÂÂÂÂ
ÂÂÂÂÂÂÂÂÂÂÂÂ4CF966EA
ÂÂÂÂÂÂÂÂÂÂÂÂ82AA0065
ÂÂÂÂÂÂÂÂÂÂÂÂ0001
ÂÂÂÂÂÂÂÂÂÂÂÂ72
ÂÂÂÂÂÂÂÂ

So imageNumber is the number of the image, and the two CRCs are the CRCs of their respected image. OK, I got that now.

The duplicateID is a number used to identify a game, so you can bundle the same games from another region in the same archive.

So, if this was the release list:
0001 - Game 1 (JP)
0002 - Game 2 (EU)
0003 - Game 1 (US)
0004 - Game 3 (JP)
0005 - Game 2 (JP)

The first game (Game 1) would get duplicateID 1. The second one would get ID 2. The third one (again Game 1) would get ID 1, because these are the same games from another region. Basically, the first release, no matter the region, of a game, gets the next duplicateID.

Indeed.

QUOTE(Maikel Steneker @ Jan 26 2010, 08:28 PM)

There's an option in OfflineList to use these numbers to put the same games from all regions in one archive. In the case of 7Z, this saves a lot of space.

That is a very good idea. I will definitely use this.

Thank you again.

Do you know anything about trimming? I understand that you remove the 0x00s after an offset that is parsed from the ROM. But doesn't manipulating the ROM affect it's CRC32? I am asking this because if it does affect, then how can you know that a ROM is been trimmed? RToolDS has a column for this information (trimmed or not).

 

[Maikel Steneker]

Well, the imCRC1 is really the CRC of the image, right? Because it's a hash code and not a number.
So imageNumber is the number of the image, and the two CRCs are the CRCs of their respected image. OK, I got that now.
Yeah, I meant to say that.
QUOTE(pekalicious @ Jan 26 2010, 08:50 PM) Do you know anything about trimming? I understand that you remove the 0x00s after an offset that is parsed from the ROM. But doesn't manipulating the ROM affect it's CRC32? I am asking this because if it does affect, then how can you know that a ROM is been trimmed? RToolDS has a column for this information (trimmed or not).

Trimming is indeed removing the extra, empty space that fills up the ROM. If you delete or change this data, the CRC value will change as well. To verify trimmed roms, you'd have to use seperate CRC values for trimmed roms or find some way to restore a ROM to its original state (should be possible, as you know the size and the data which was deleted).

I haven't used RToolDS, but I'd guess it just takes a look at the filesize to determine whether a rom has been trimmed or not without verifying it. Personally, I keep my roms untrimmed and zipped. Only when I copy them to an SD card for usage I trim them with a batch trimmer like NDSTokyoTrim.

 

[pekalicious]

To verify trimmed roms, you'd have to use seperate CRC values for trimmed roms or find some way to restore a ROM to its original state (should be possible, as you know the size and the data which was deleted).

I haven't used RToolDS, but I'd guess it just takes a look at the filesize to determine whether a rom has been trimmed or not without verifying it.

Yeah, you can determine if a ROM is trimmed, but you need the trimmed part to calculate the CRC32 and find which game it is. Which is what I will do.

I see just one problem with this: the trimming algorithm. As far as I know, there are at least two. One that looks for the first empty data and deletes everything from there on, and the correct one which looks up the offset from where to trim. I found this in this thread.

This is why DS-X Manager's trimming is different:
Every rom has a set point for where the data ends. It's in the header. The trimmer I wrote writes up to that point. After that, there's 4 bytes of blank space, followed by 184 bytes the WiFi data. If you trim roms by just using the ROM Size and don't take into account the wifi you'd break download play (and maybe something else) on the carts that support it.

QUOTE(Maikel Steneker @ Jan 26 2010, 08:55 PM)

Personally, I keep my roms untrimmed and zipped. Only when I copy them to an SD card for usage I trim them with a batch trimmer like NDSTokyoTrim.

As do I.