ROM Hack Extracting puzzle data from Picross 3D

[Martoon]

Don't know if it's of interest to anyone, but I've been playing around with extracting the puzzle data from the Picross 3D ROM. Why? I don't know. I'm kind of interested in figuring out how they generate the clues for the puzzles when you create them. I thought it'd be nice to have the set of puzzles in a format my own programs could use. Maybe even make a Windows or Flash clone or variant.

Anyway, I've figured out that the puzzles are spread in groups throughout the ROM. They have the following format (I'll be using the "Blackboard" puzzle as an example):

0000 to 000F: Don't know. Blackboard has this: 20 04 05 1E 1E 03 C7 08 00 10 F0 4B 08 F6 01 0F. They always start with 20, and always have 00 10 F0 4B in those four bytes near the end. I'm thinking this might be some kind of color palette.

0010 to 0027: Puzzle ID name, padded with leading spaces. Blackboard has " 037_BLACKBOARD". It's always a 3 digit number, underscore, and all-caps name.

0028 to 0097: Don't know. Blackboard has: 00 00 00 00 00 00 00 00 71 05 C8 00 A0 01 C0 00 58 6B AC 35 10 42 08 21 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00.

0098 to 0187: Descriptive name of the puzzle, in two-byte Unicode, in five languages (48 bytes each, padded with trailing zeros). English is first. Blackboard has "Blackboard".

0188 to 056F: The actual puzzle data. 1000 bytes, which covers the maximum 10x10x10 size of a puzzle. It appears to go up in columns, starting at the bottom of the puzzle and going up, then going to the next column, etc., until a layer is filled, then repeating for all 10 layers. It appears to fill completely empty columns with zero, but empty spaces in partially-filled columns with 0x20. The non-empty spaces contain 0x30, 0x31, 0x32, etc., which are color indexes (I can see by looking at a puzzle, for example, that everywhere it has a 0x30 is the same color, and all 0x31s are their own color, as are 0x32s, etc.).

0570 to 05FF: Don't know. Blackboard has: 02 00 02 00 00 00 02 00 02 00 02 00 00 00 00 00 00 00 00 00 01 00 01 00 01 00 01 00 01 00 01 00 01 00 00 00 00 00 00 00 3D 00 3D 00 3D 00 3D 00 3E 00 3E 00 3D 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00.

Each puzzle uses a 0x0600 size chunk, and they always start on page boundaries that are multiples of 0x0200. They're packed together in small groups, but the groups are spread throughout the ROM.

I've written a Python script to find all the puzzle chunks, and write them out to XML.
CODEimport xml.dom.minidom

def hex_string(inStr):
ÂÂÂÂoutStr = ""
ÂÂÂÂfor byte in inStr:
ÂÂÂÂÂÂÂÂoutStr += '%02X' % ord(byte) + ' '
ÂÂÂÂoutStr = outStr[:-1]
ÂÂÂÂreturn outStr
ÂÂÂÂ
romfile = open("4926 - Picross 3D (USA).nds", "rb")
romdata = romfile.read()
romfile.close()

doc = xml.dom.minidom.Document()
root = doc.createElement("puzzles")
doc.appendChild(root)

dataIdx = 0
while (dataIdx < len(romdata)):
ÂÂÂÂchunk = romdata[dataIdx:dataIdx + 0x600]
ÂÂÂÂname = chunk[0x10:0x28].lstrip(' ')
ÂÂÂÂ
ÂÂÂÂif (len(name) > 4):
ÂÂÂÂÂÂÂÂif ((name[3] == '_') and name[0].isdigit() and name[1].isdigit() and name[2].isdigit()):
ÂÂÂÂÂÂÂÂÂÂÂÂhead1 = chunk[:0x10]
ÂÂÂÂÂÂÂÂÂÂÂÂ# name
ÂÂÂÂÂÂÂÂÂÂÂÂhead2 = chunk[0x28:0x98]
ÂÂÂÂÂÂÂÂÂÂÂÂenglishName = chunk[0x98:0xc8].replace(chr(0), '')
ÂÂÂÂÂÂÂÂÂÂÂÂhead3 = chunk[0xc8:0x188] # other languages
ÂÂÂÂÂÂÂÂÂÂÂÂdata = chunk[0x188:0x570]
ÂÂÂÂÂÂÂÂÂÂÂÂtail = chunk[0x570:0x600]

ÂÂÂÂÂÂÂÂÂÂÂÂpuzEl = doc.createElement("puzzle")
ÂÂÂÂÂÂÂÂÂÂÂÂpuzEl.setAttribute("name", name)
ÂÂÂÂÂÂÂÂÂÂÂÂpuzEl.setAttribute("englishname", englishName)
ÂÂÂÂÂÂÂÂÂÂÂÂpuzEl.setAttribute("header1", hex_string(head1))
ÂÂÂÂÂÂÂÂÂÂÂÂpuzEl.setAttribute("header2", hex_string(head2))
ÂÂÂÂÂÂÂÂÂÂÂÂpuzEl.setAttribute("tail", hex_string(tail))
ÂÂÂÂÂÂÂÂÂÂÂÂtn = doc.createTextNode(hex_string(data));
ÂÂÂÂÂÂÂÂÂÂÂÂpuzEl.appendChild(tn)
ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ
ÂÂÂÂÂÂÂÂÂÂÂÂroot.appendChild(puzEl)

ÂÂÂÂdataIdx += 0x100

outfile = open("puzzles.xml", "wb")
outfile.write(doc.toprettyxml())
outfile.close

That's what I've figure out so far. I'm guessing when you create your own puzzles and save them, they're saved in a similar format in the save game file. I may mess with that later, and create some diagnostic puzzles to try to nail down what those other chunks are (and how the color palettes are defined).

 

[Lyfeless]

Hello, I have done more digging and have a few bits of info to contribute to this thread. Hope my response is not too delayed.
For clarity, I'll be describing the different block sections by the names used in the provided python script above.

Header1: This section appears to be metadata for the level, although I haven't been able to decipher most of it. However, I did manage to find a few things. The first 3 bytes appear to be related to the level/background, although I wasn't able to find anything meaningful about what the numbers themselves mean. It may be a pointer to somewhere else in the rom? The 6th byte is the song, simply an number starting from 0 and counting up. Past that, the rest of the header doesn't make a whole lot of sense to me. I will note that a few bytes near the middle of the header seem to affect the level's size/display information, but I wasn't really able to make heads or tails of it.

Header2: Zero idea what this is. My best guess is that this is related to palette information, as it seems to have more data in it as the palette size increases, though this is likely a coincidence.

Data: Most of what's been figured out from this seems to be correct, this is the data for the level. The only thing of note I have to contribute is that I believe the order of generation for the puzzle loops through the horizontal row first, then the vertical column.

Tail: This is where most of the data I dug through is located. This block of data is used for storing the level hint/clue data. Reading through it is sort of complicated since they made use of a few techniques to reduce the size of the data (I'm not really sure why they went through the trouble of doing this if they were going to pad the leveldata out with a bunch of unused bytes anyways but Hal Laboratory has their methods I guess). The hint data is stored in blocks of 20 bytes, the first group of 20 being used for the side, the next group being the top, and the third being the front. Each side stores the hint data in 10 rows, 2 bytes per row. The side stores rows that go back to front, starting on the bottom and going to the top, the top rows go back to front, stored from left to right, and the front rows go bottom to top, stored left to right.
The way Picross stores its hints is a little weird. The leveldata never actually stores the value of the hint, instead simply storing a boolean flag to tell the game whether or not to display a hint on that row. 0 shows the hint, 1 hides it. The two bytes for each row store that row's hint flags as different binary digits, starting from the ones place and going up. For example, if a row had value 2F 02, that would translate to the binary number 00000010 00101111 (the way the game stores this, the first byte is the lower 8 digits, technically switching their places). This would correspond to the first 4 tiles closest to the front hiding their hints, followed by 1 tile showing its hint, then another hidden hint, etc.

For my purposes, this was just enough deciphered information for my purposes so I didn't dig into it much further than that. However there's obviously quite a bit of stuff I wasn't able to figure out. If I were to take an educated guess I would say actual palette data is stored elsewhere in the data, or perhaps is taking its color values from some build-in ds system (I don't know anything about how the DS actually works so take that guess with a grain of salt). I did notice the level names appear elsewhere in the data so it's possible external data like animations and colors are stored there.
Header2 really is the mystery to me. It seems like it's completely unused, as removing all the data from that block didn't have any noticeable impact on the level.
I will also note that the python script provided doesn't seem to run properly in modern python. I managed to create my own script to pull the data I needed but it's admittedly less than optimized and rather long.

This was a rather fun side project to put some time into, but I likely won't be looking into this further. I wish the best of luck to the next person interested in the file structure of this game and hope my additions prove useful. Thank you very much for the original post, it was an invaluable resource in my own search.