ROM Hack Deviations from the nitroSDK formats (maybe versions thereof)

[FAST6191]

I probably should not start a new project right now but I thought I would lay the groundwork at least.

Most people doing anything on multiple DS roms will encounter the SDK formats soon enough ( http://www.romhacking.net/documents/469/ , http://llref.emutalk.net/docs/ , http://kiwi.ds.googlepages.com/sdat.html , http://kiwi.ds.googlepages.com/nsbmd.html , http://nvwr.googlecode.com/svn/trunk/libs/formats/ and so on) and they make a lot of DS hacking somewhat easier (presumably programming as well) but certain games have been observed to do things differently leading to the whole affair coming crashing down.
Now this is a well known problem in reverse engineering and indeed many of those that define proprietary formats will subtly shift the format which breaks reverse engineered parsers but I have yet to see anything resembling a kind of V1, v2, custom tweak analysis similar to the likes of the big reverse engineering projects other than maybe when people were looking at different region saves (not sure about v00 and v01 or early pre wifi and later wifi versions).

Monster tale appeared to have something extra in the SSEQ (tracker/sequence audio format common in SDAT audio which most DS roms use) files but I have not confirmed that http://gbatemp.net/index.php?showtopic=292661

Pokemon has done lots of this over the years with the most notable being the nameless narc files which led to narctool 0.1-p and I am told some of the graphics formats there ignore the layout information and have it defined somewhere else. Pokemon has a history of being a pain to hack so whether this falls under different formats, incomplete specification or obfuscation is probably open to debate.

Phoenix Wright was supposed to use some graphics formats differently as well ( http://gbatemp.net/topic/303529-tinke-072/page__st__225 ) or at least different to current implementation from tinke.

Sands of Destruction had some graphics formats that might not be standard http://gbatemp.net/topic/329423-unpacking-an-archive-in-a-ds-rom-iear/

NSBMD... I do not know if this is differences in formats or just that the format has not been completely reverse engineered but I am sure we have all seen the existing tools fall short. Certainly I have seen texture animations and similar done manually and the NSBCA/NSBTX formats ignored). I grabbed the old Metroid Prime Hunters model viewer the other day ( http://filetrip.net/nds-downloads/utilities/download-dsgraph-10-f29517.html ) and although popular thought has it as the precursor to NSBMD (certainly nsbmdtool appears to owe a nod to the graphviewer program for metroid and both are open source so comparing should be easy enough) I have not seen anybody confirm it.

NFTR (font format) had a few minor tweaks as time went on ( http://gbatemp.net/topic/105060-nftr-editor/page__st__135 ).

VX video format from ActImagine/mobiclip later became MODS and there appear to have been a couple of versions of this ( http://gbatemp.net/topic/125374-player-of-vx-file/ ) but the format itself has not really be reverse engineered yet.

BMG (a text format but one usually only seen in first and second party games or download play in my experience) I am not sure I have seen a complete reverse engineering of yet although unicode and fairly basic examples of the elementary pointer types does not leave a lot of room to change much.

Custom compression? As far as most common types are concerned (type 10, type 11, type 40....) most of it is nothing major (a different flag type, a different range....) if you know how the compression works in the first place. I might have some proper custom compression though (El Tigre Make my mule) but I do have to confirm that (it might just be "headerless").

Procyon Audio (you probably know it by the extension SAD)- not strictly an SDK format as far as I know but common enough to be noted. There is however a disparity between the luminous arc stuff and the professor layton version of the format which wins it a spot on the list.

ADX- ADX, AHX, ADH. Most of that is done but apparently there is a ADX2 format although that seems to be seen in the PSP rather than the DS.

I am not sure if I want to try relating the likes of NSBMD and (N)ARC to the GC ( http://hitmen.c02.at/files/yagcd/yagcd/chap15.html#sec15 ) and Wii equivalents at this point but that is a possible future direction. Going down that leads to comparing formats between games either on the same system (Super Princess Peach and New Super Mario Brothers probably want to be compared at some point if they have not already) or different systems (Mario Kart DS being a fairly big focus here on various levels). I am not really looking at non SDK formats myself but if you have an interesting one or want to compare games in a franchise/from a given developer (the ph/pb format used in Inazuma 11 changed between versions for example) do feel free to post an analysis.

I probably should note somewhere in this that either parts of larger files have been seen by themselves (the NTFP section of a palette was often seen by itself in earlier games and not in what would later be the NCLR palette format) or could be precursors to formats (same example from a moment ago probably applies).

Also if you have any good developer leftovers that explain a format and might give a clue there ( http://gbatemp.net/topic/320192-japanese-programming-madness/ has some examples and links to them and the SMAP file from Zoids Saga DS), we all love levels, audio and graphics that did not make the final cut but unless part of that is directly useful to reverse engineering (that is to say it is composed of primitives and example layouts in the case of a level) probably better to find another thread.

Anyone got anything worth noting in this arena?

 

[pleonex]

Pokemon games have some nitro files so weird. IIRC the last thing I was investingating was the NFTR fonts, I have more or less figured out it, they have one sectiong inside another... I need to find the flag to know that. But also some NCGR files that have their tile data encoded.

The problem with BMG files is that some of them are BigEndian...

And about the AHX files that I have seen seems, are not like Wikipedia says (from I took the code for ADX encoding). I was trying to remove the AHX header from them and then try to play in a common player (I have tried with mpg123 too) but they give problems and can't be played. It's strange because you can check that they have the mpg2 header in each frame but it seems that their frame size are not like it says.

 

[rastsan]

As to the bmg, jjjewel may have it right - here.

If you are looking for series or a type/genre of game that has the same format and uses the nitro format (except for the text). Well master of the monster lair type games (date ni game, my world my way) all share a common bin format that is just a very easy to take a apart container file and file structure in common. The english versions with other english versions and the japanese with the other japanese. If anything they would be a good study on file redirection. As in they have the original japanese files in their folders with the english (en) folder holding english versions of those files/folders. (Okay with what appears to a new way to implement the same files but still - the .tbl)...

maybe not for this but still worth a mention.
Oh and if you are looking for a slight variation of format Elminage series uses what appears to be ncg, ncl, nsc but with extra info tacked on the front and end of the files. this information is just a fancy way to chain the files together. With direct mention to certain other files. so the ncl mentions the ncg(ccg or nccg) . the ccg is just a packed version of ncg that also mentions cce. the cce is just a nsc again with the extra header footer. Pleonex has done some documentation on these file formats. though technically they could be a genre of their very own. tinke can handle these files easily but editing the map files will need more dissemination. This game contains csv format text and has in some of those files the psp version of the same text. Along with script files that eventually i may get to pulling apart.

now if you are looking for a prime example of non nitro format that does have versions the shiren the wanderer series and its sir0 headered files need to be mentioned. you can see the 'evolution' if you go through the games by release date. getting more complex than going back and getting less complex. I should mention that these same sir0 text files can also be found in 9 person 9 door 9 hours (i hope i got the name right) with a slight difference as file structure but clearly (to me) the same file type.

 

[pleonex]

Yes, SIR0 are so different from game to another. In fact, inside the same game (999) they are used in different ways: the first one inside the /cha folder for sprites images, the second one for text, the third one it's encoded with AT6P (I wrote a tool to decrypt them) and it's for background images and finally a fourth one for sound files IIRC. Furthermore if you go to another game with files with that header you'll see that they're almost different (as far I have seen).

Some time ago, I started to see that some nitro files like NCLR and NCGR have a new section. Specially in the NCLR format the section with header "PMCP" has been added, still no idea what it could be but maybe something related to use with map files. The same for NCGR files, some of them with the section "SOPC" totally unknown for my part.

About the ncl and so. In the lastest Layton game, London Life folder, there are like new nitro format for images: NCCL for palettes, NCCG for image data, NCSC for map and NCOB for sprite data. They are interesting because if you look at them, it seems to be an evolve of previous formats, with a lot of more sections, the NCOB files have now the animation and image data too and all of them have a common section, "LINK", which give the relative path to know what palette file, image file... it's linked to that file. I have implemented them in tinke but I haven't figured out every section of them, specially in NCOB files (some of them have 15 sections!). That sections are like frames, all of them start with the 4 bytes for header and another 4 bytes to the size of the section so you can skip them without problems (like the NCLR, NCGR, NSCR, NCER and NANR).

 

[FAST6191]

Yeah I have seen sir0 in Zombie Daisuki as well although that was another ChunSoft game so probably not a surprise.

I had hoped to make a kind of table along the lines of

precursor - Mid 2006 - Simple magic stamp and file size witnessed.
1.0 - Early 2008 - First instance of ???? in ****. Encompassed precursor but added ability to store multiple versions of the file.
1.1 - Early 2009 - New section used for YYYY was spotted
1.11 - Early 2011 - Pointer section expanded to 24 bits (in 32 bit sections).
1.2 = Late 2011 - Internal file level compression spotted with upper 8 bits in pointer section being used as compression flag.

As you say though games do odd things and various developers throw things in container formats, add extra headers/trailing headers to files or ignore abilities the formats afford so that method might miss half the story. I have seen quite a few use otherwise standard formats but use different extensions (and not just games that moved directly to using numbered files) as well or pack files that usually get found in other files (el tigre again shoves a bunch of swav files in the pkg format it uses).

Anyhow I am now faced with writing many of the tedious sections in my current rewrite of the rom hacking docs and if I get distracted now I sense it will join the last few attempts to rewrite them I made in the bitrot that is my hard drive these days (something I would really rather not happen to this version) so I am afraid I will probably not be able to add much to this for the time being, thanks for the extra information though.

 

[Julio Sotomayor]

OK, I have a complete text dump of SCP.BIN from Elminage 2 and I personally spaced every single sentence together as well as one man can. I used a macro to eliminate the majority of the hexadecimal garbage symbols (I'm using Notepad++ because it has integrated SHIFT-JIS for this)-- It's as put together as can be expected. Any translator worth his salt could easily translate this document now. And yes, I know, it is unbelievably huge. No one's gonna do it. Whatever, who cares.

(I've been using the Elminage: Original ISO to know where to find things, obviously)

The second thing I have to report is that I did something similar with the ETC.BIN and did some _very quick and dirty_ "marks" inside WindHex32 to denote where things such as the Mage/Cleric/Alchemist spells begin, where the section on class changes begins, etc. I think those two .BIN's are probably the most important ones to translate to have a playable game in English, and one of the two .BINs can be machine-translated. So we have good news there.

Now for the life of I can't find ANYWHERE, either inside the Elminage 1 ISO or the 2nd game's ISO where the battle commands are! It's insane! But it's true; I've done a relative search on every singe .BIN that's inside the Elminage: Original ISO using every single one of the fight commands, you know: fight, defend, escape, etc. I have turned up nothing! It's insane! I wanted to find that first after I was finished arranging the game script from SCP.BIN and quickly go over to the Elminage 2 files and do a quick battle menu translation; but like I said I just can't find those commands.

The only places I haven't been able to access are the DATA.BIN(s) from either game as they're all completely zero'd out inside WindHex32 even though they are very huge files.

Aside from further arranging the script dump from SCP.BIN (hey, it's not perfect) a little better, and getting started on translating everything inside ETC.BIN which is where the majority of the game's non-dialog content is contained I don't know what else I should do.