ROM Hack Decoding Tengen Toppa Gurren Lagann images

[Illidanz]

EDIT 2021:
The translation patch has now been released! See here: https://gbatemp.net/threads/release-gurren-lagann-english-patch.544148/
The tool used to translate the game can be found on GitHub here: https://github.com/Illidanz/GurrenTranslation

Original post:
Hello guys,
I'm working on setting up the tools for a translation of this NDS game.

I picked this up as an excuse to learn python, and I didn't think I was going really far with it since it was my first attempt to hack any ROM but after a couple of days I was able to extract the text from the scripts in the custom archives of the game, as well as some hardcoded lines from the arm9 binary.

I made a little progress, here is a bunch of screenshot of what I have so far (I don't really know japanese so the translation is probably inaccurate, I'm just testing the tools):
http://i.imgur.com/G2q6Rzk.png
(Original game above, translation below)

Right now I'm trying to decode the images in order to translate them (I'm pretty sure that the menu buttons are images as I can't find the strings anywhere, and there are a lot of images throughout the game to translate too), but I can't seem to make any progress whatsoever.
I have tried messing with crystaltile and tile molester but I wasn't able to do anything, so I thought I'd ask here for some help. Maybe it's something really simple but due to this being my first hack attempt I may miss something obvious.

Here are 3 sample files. I think the game uses the .KPC extension for static graphics and the .YCE extension for animated graphics.
The 2 KPC files are the backgrounds of the first zone (third screenshot in the image above).
BGA_001.KPC
BGB_001.KPC
AV01_01.YCE

I would really appreciate any help as I'm quite stuck on this and can't seem to go anywhere.

Thanks in advance for any help!

PS: if anyone is interested in the formats used in the game I can write something about them, just let me know.

 

[Illidanz]

Hello again! A little update: I made a lot of progress on the text side of the things, so I figured I would post it here as I feel it can be useful to other new rom hackers.

If you're not interested in the technical stuff, you can view some random screenshots in the imgur album here (again, don't mind the translation, I'm just testing): http://imgur.com/a/4t11P#0

The archives
The first problem was that the rom had no files inside it save for 6 big archives with a NFP extension.
After searching on the internet I found this japanese page that explained a slightly different version of the format.
It is pretty simple, and although the one of the game was a bit different the basis were the same.

The font
The second problem whas that the NFTR font was fixed width. After messing without much success with the arm assembly, I tried to follow this suggestion and it worked like a charm.
Here's a screenshot with the fixed width font: http://i.imgur.com/gM9uirl.png
And one with the new font, edited to fit in the given space: http://i.imgur.com/vOL4sdt.png

If anyone needs the font, you can find it here: GL_12FNT.zip
The zip file also contains a text file with the table needed to match the "bigrams" with their respective code.
Or if you prefer, you can import this image with NFTREdit (convert to bmp first): http://i.imgur.com/y3YTR5O.png

The text
After fixing the font, I had to figure out how the scripts file worked in order to shift the pointers when modifying the length of the text.

I made a little image showing how the scripting the game uses works, I thought it may be useful to others if some other game uses it (excuse my poor paint skills):

This is the smallest script with a japanese string I found, and it is the one that prints the string in this screenshot: http://i.imgur.com/aNjMGTj.png

This was pretty easy to figure out, what was harder (and took most of the time) was some other pointers that are found in the middle of the code section in complex scripts (more than 10kb big) and I think are used when different stuff happens in the same zone whether you talked to someone in another one or stuff like that.
These other pointers are in the form of 0x12 0x00 4-byte pointer and 0x00 0x11 4-byte pointer. However a lot of other stuff matches these "signatures" so you have to check that the first type points to a short equals to 0x0011 (little endian) and the second one points to a short equals to 0x310F (little endian). If they don't, it's probably something in the middle of a string or of another code type.
There probably is a better way to detect those but what I have now works so I don't complain.

So after figuring out the pointers, I made a (messy) python script that exports all the strings in the 500 script files to a single text file where they can be translated, like this:

Spoiler

!FILE:EV_004.SPC
やだ、穴掘りシモン！=Ew, it's Simon|the Digger!
毎日土まみれになって=What's the fun of
何が楽しいか？？=digging everyday?
なんかへんな物=He's carrying
持ってるし=something weird, too.
キモイよね～～=He's so gross...
あ、こっち見てる=Oh, he's looking|this way!
気をつけないと=Careful, he might
穴掘られるかもよ！=drill a hole in us!
(Note: the | character is used by the script to modify an int 0x1C bytes before the start of the string to make one-line messages become two-lines messages, and replaces itself with a line break.)

And then the script reimports them back shifting all the pointers. Due to the fact that the english strings have pointers at the end of them, they need to be shifted as well, even if they are not modified.

A lot of other strings (about 1000) like the ones for the combat and menus are hardcoded in the arm9 binary, so I made another script that finds them with a brute-force method (it looks for one or more shift-jis encoded japanese character followed by 0x00 in specific sections of the binary) and then exports them in a file like the one above. Obviously the editing of these strings is a lot more limited as you have to keep them shorter or of double the size (thanks to the font that has 2 characters per glyph) of the original string.

What's next?
After all this I was able to translate the first few minutes of the game up to the first combat without the game crashing/acting up.
The script of the game is very similar to the one of the anime so I used the official one from the recent BD release and I google-translated the rest so that will need to fixed by a proper translator.

Text-wise, I still need to figure out how minigame files work, there are probably more pointers I didn't found, and I'm still missing a few lines (the ones from the tutorial in the first battle).

The big thing that's left to start a proper translation project is the image format. I messed with it a little more but I still wasn't able to figure out the format. I will try again as I get some time in the next days.

Meanwhile, if anyone has any idea or suggestion to convert the images referenced in the first post, please share!

 

[Rydian]

Not very useful post, just wanted to say this has been a very interesting read!

 

[jjjewel]

I only looked at the 3 image files you upload so this info might not apply to every file. And I used CrystalTile2 to check them, so my explanation will be based on CrystalTile2's tools.

KPC File Format
(It's long and has pictures, so I put it under spoiler tag.)

Spoiler

A bunch of 01 01 from 0x06 to 0x0B might have something to do with each sub file's compression. (I didn't check, but it's very likely.)

@ 0x0C image width / 8
@ 0x0E image height / 8

@ 0x14 Map offset
@ 0x60 Tile offset
@ 0x84 Palette offset

If any of these files start with 0x10, it's very likely that the file is compressed with LZ77 type 10.
The map/tile/palette work similarly to NSCR/NCGR/NCLR (without the header).

Ex. For BGA_001.KPC
@ 0x0C image width / 8 = 0x50
@ 0x0E image height / 8 = 0x20

@ 0x14 Map offset = 0x00C0
@ 0x60 Tile offset = 0x0EE0
@ 0x84 Palette offset = 0x41A0

Image width = 0x50 * 8 = 0x280 = 640
Image height = 0x20 * 8 = 0x100 = 256

@ 0x00C0 First byte = 0x10 so it's probably compressed.
@ 0x0EE0 First byte also = 0x10 so it's probably compressed.
@ 0x41A0 First byte is 0xE0 so it's not compressed.

You need to decompress the compressed files. (CrystalTile2 has a tool under Tools menu to decompress part of a file.) Then you can go to the palette offset an use CrystalTile2's Edit/Data to Palette function and export the palette so that you can import it later when you view the decompressed tiles.

My method of extracting picture is to use another program called NDS Editor to pack the decompressed files as .narc and use CrystalTile2 to view the mapped tiles. Then you can just export the image.

It looks like this.

The other file would look like this when you extracted it.

Anyway, to edit and insert the file back, you'll have to combine the extracted files into KPC format and update all the offsets.
(The offsets I wrote above are the obvious ones that I could define. But I didn't try to figure out what other numbers might lead to.)

*You can download both CrystalTile2 and NDS Editor from http://filetrip.net/.

I didn't want to make the post too long so I didn't explain everything in details. Let me know if there's any part you can't follow.

YCE File format

It seems like animation file with a bunch of tiles and palettes in one file.
Most palettes seem to start with 0x1F7C.
(If you have problem viewing the image, check if the image is compressed. The sample file you uploaded doesn't have any compression but other files might.)

For example, the first picture uses palette at 0x090C and the image offset is at 0x00CC.

<-- Sorry it's so tiny.

There are some bytes before the image that probably tell you how the tiles are arranged if you want to write image extractor. But if you just want to edit them, you can use CrystalTile2's Edit/Data to Palette and then use Tools/Tile Composition Tool to edit and insert image.

 

[Ericthegreat]

I like gurenn lagann a lot, but if thats the same game that I played where you fight other robots over and over, it was a really shitty game. (Maybe it gets better?)

 

[Illidanz]

jjjewel: Wow, thanks a lot! I don't think I would have guessed all that stuff without your help.
I have tried to open other files and your guesses seems to be right, so I will try to write a script to export them the next days.

On another note, yesterday I was messing around with the .3DG files, which I thought were just the 3D models, and I noticed they were just standard NSBMD files with an additional header and footer, so I removed them to check what they were and to my surprise, there were a lot of images I didn't thought I would find in there. I was able to view some of them with the nsbmd viewer/nsbtx extractor, but on a lot of other ones they crashed, and on others they weren't able to extract all the textures, so I wrote yet another script to export the images and palettes in convenient png files.
Here are a couple of examples:

Spoiler

Still, no UI buttons or opening logo so they are in the KPC files (and one or more animated UI buttons are in YCE files from what I saw)

I like gurenn lagann a lot, but if thats the same game that I played where you fight other robots over and over, it was a really shitty game. (Maybe it gets better?)

The game might be a bit unhortodox gameplay-wise, but it's not that bad once you get used to the controls. The plot and music are great though!
Obviously if you don't understand japanese and you just do the battles it gets boring pretty fast, it's sort of like playing a visual novel while just looking at the pictures. That's why I would like to translate it.
There's a great english review here if you want to read more.

 

[Ericthegreat]

jjjewel:
The game might be a bit unhortodox gameplay-wise, but it's not that bad once you get used to the controls. The plot and music are great though!
Obviously if you don't understand japanese and you just do the battles it gets boring pretty fast, it's sort of like playing a visual novel while just looking at the pictures. That's why I would like to translate it.
There's a great english review here if you want to read more.

Hey looking at the pictures can be just as "exciting"

 

[jjjewel]

@ Illidanz. I just happened to see the part that looked like palettes and worked from there. Luckily the palettes weren't compressed. Otherwise I might not see them. It was quite a fluke that I figured it out. Anyway, glad I could help.

For the buttons and logo, you might try to make a memory dump with an emulator when it loads the graphics. And then try to look for tiled images in the dumped file. If you find the images you want, then search in your rom or your decompressed graphic files for something that look the same.

P.S. It feels so refreshing to drop by NDS hacking board and see something that's not pokemon-related. LOL

 

[Illidanz]

@ Illidanz. I just happened to see the part that looked like palettes and worked from there. Luckily the palettes weren't compressed. Otherwise I might not see them. It was quite a fluke that I figured it out. Anyway, glad I could help.

For the buttons and logo, you might try to make a memory dump with an emulator when it loads the graphics. And then try to look for tiled images in the dumped file. If you find the images you want, then search in your rom or your decompressed graphic files for something that look the same.

P.S. It feels so refreshing to drop by NDS hacking board and see something that's not pokemon-related. LOL

I was able to find the opening logo in a KPC file and what I think is new game button in this YCE one.

After messing around with the options I was able to see the first image in crystaltile: http://i.imgur.com/vdWaShP.png (palette starts at 0x85C)
But the tile composition tool doesn't seem to work for me, I exported the image (tried different formats) and tried to import it in the tool but nothing happens. Am I missing something? Or maybe I got the size of the image wrong, I wasn't able to find them so I just tried to guess.

 

[jjjewel]

I was able to find the opening logo in a KPC file and what I think is new game button in this YCE one.

After messing around with the options I was able to see the first image in crystaltile: http://i.imgur.com/vdWaShP.png (palette starts at 0x85C)
But the tile composition tool doesn't seem to work for me, I exported the image (tried different formats) and tried to import it in the tool but nothing happens. Am I missing something? Or maybe I got the size of the image wrong, I wasn't able to find them so I just tried to guess.

Tile composition tool will require some manual work. You can view the tiles as 8x8 and then resize your window or use Shift+Arrow keys. And then drag and drop the tiles to the tile composition tool.
(An example for NCGR tiles is here; https://sites.google.com/site/otomegameguide/tutorials/ct2-graphic-edit#TOC-Discontinuous-graphics The concept is similar.)

It'll take a little bit of time and be careful because CrystalTile2 tends to crash a lot. (Anyway, I started hacking graphics before this tool was released and things were much tougher back then. So even though it might crash every 10 minutes, I wouldn't complain much. )

 

[Illidanz]

Tile composition tool will require some manual work. You can view the tiles as 8x8 and then resize your window or use Shift+Arrow keys. And then drag and drop the tiles to the tile composition tool.
(An example for NCGR tiles is here; https://sites.google.com/site/otomegameguide/tutorials/ct2-graphic-edit#TOC-Discontinuous-graphics The concept is similar.)

It'll take a little bit of time and be careful because CrystalTile2 tends to crash a lot. (Anyway, I started hacking graphics before this tool was released and things were much tougher back then. So even though it might crash every 10 minutes, I wouldn't complain much. )

Oh I have to drag the image on the composition tool, ok. I thought I had to use the import button to load the image first.
And it wasn't that bad! The program never crashed actually, at least on this one. I made a quick test with a crappy font and it works just fine: http://i.imgur.com/IGQsKHn.png
Thanks again for the helpful input

I think I have pretty much everything covered for now, just need to find the other buttons, so I'm off doing some polishing on the tools.

Meanwhile if a translator interested in the project happens to pass by, please contact me!