GBATemp ROM hacking documentation project (new 2016 edition out)

Discussion in 'NDS - ROM Hacking and Translations' started by FAST6191, Jan 28, 2008.

  1. luke_c

    luke_c Big Boss

    Member
    3,587
    32
    Jun 16, 2008
    Land of England
    Great guide FAST, helped me a bunch [​IMG] just wondering, is it absolute neccesary to learn C#/C++ for this, i'm already filled up to my head in coursework ._.
     


  2. psycoblaster

    psycoblaster Divine

    Member
    2,132
    2
    Jan 26, 2008
    Seoul.. (in Korea)
    The reason why programming helps is because it helps you do repetitive work, and it just makes the whole task easier.
    Manually increasing the file size, editing pointers, rewriting control codes etc can be a pain. With programming, you can make this whole work easier by making it as simple as editing a text file.
     
  3. Sp33der

    Sp33der GBAtemp Fan

    Member
    435
    1
    May 31, 2008
    Netherlands
    Not really good with C# syntax, so what's textDump?
     
  4. psycoblaster

    psycoblaster Divine

    Member
    2,132
    2
    Jan 26, 2008
    Seoul.. (in Korea)
    before I knew C#, I made Darth make programs for me ;D
    jk
    well I had to do everything in a hex editor, manually writing down every new pointer for each line.
    Why not just spend that time learning how to make a program that calculates everything for you?
    List textDump = new List();
    In C# library, there is a Generic List class. http://msdn.microsoft.com/en-us/library/6sh2ey19.aspx
    For more info about the < > (Generic programming), look at http://en.wikipedia.org/wiki/Generic_programming
    After a variable type, a name is given to it.
    So basically, what this line of code do is creates a new List which can only contain strings (So no need to cast objects), and the name of this List is textDump.
    new List();
    This part of code means you will create it as a new List of strings with no parameters.

    I'll tell you though, that the easiest way of learning how to program is to just learn the basics, and then look at source codes of different programs.
    (That's how I learned)
    I started out with JAVA, just learning the basics. Then I took a look at Darth's codes, and just learned from there. (The internet is also a good reference when you are looking for specific classes that can do what you need)
     
  5. FAST6191
    OP

    FAST6191 Techromancer

    pip Reporter
    23,361
    9,156
    Nov 21, 2005
    My thoughts.
    As mentioned much of hacking once you get past the reverse engineering is long winded and tedious if you go it by hand. Fortunately computers are designed for long winded repetitive tasks (actually they are good at little else which many people and coders seem to forget).

    Fortunately we have file types which means they are at some level predictable. This however is where things get interesting. Rom hacking has changed somewhat with the rise of DS hacking; snes era hackers would work in small, fairly quiet and very tight teams where DS teams are often 10 people or more and will be fairly open and people will often come and go as they are needed.
    Also seen as we are now dealing with file types on what amounts to fairly complex files (snes, GBA and the like would have code fairly similar to the hardware: see GBA pointers generally being 08XXXXXX) which are often game specific and with teams including people of less technical skill things also changed things also change.

    If you look at hacking documents aimed at the likes of the snes they will often focus around getting premade tools (table files, text extraction tools, tile editors and so on) to work with games whereas on the DS can afford to or needs to do things differently. This also marks a shift in ideas as now rather than a simple tool you can tweak to a rom you need to make your own (even if you have a tool configuring it will likely be analogous to programming something yourself).

    This is where programming comes in:
    For a lot of my internal hacks or stuff I play around with myself I will tend to use a hex editor and a spreadsheet: you can set the width of columns and the amount of bytes they contain in a hex editor* (making a crude form of matrix) and then import these into a spreadsheet.
    Alongside this various hex editors (I am still using hex workshop for day to day stuff) feature nice search abilities which means you can search for offsets or given strings that give things away (text will often end a line with something and you can grab these and add one to get the pointer location for example).
    Spreadsheets feature some serious hex manipulation tools (open office includes them as standard, MS office you may need to install the "analysis toolpack" although I have not used some of the newest versions at length so I am not sure about them) which can be good to work out file formats: it is fairly easy to take the previous cell from the current one or do fun things like that which is great for reverse engineering.

    *it might involve pasting into a text file first and you may also wish to know that 0d0a is the main method to start a new line in windows using ASCII (linux and mac tend to use either 0a or 0d although they will accept the windows method, windows standard notepad and wordpad will tend to ignore a single 0a or 0d which is why some files you download will not display properly in them).

    As well as reverse engineering you can also do shifts quite easily and if you do a search again you can often get a list of new pointers. From here it is just a matter of pasting back into your file of choice.

    However as I mentioned your team may not be so comfortable with this (the simple example is things like flags in the text to make it bold or something) and if you are doing something like narc files (bad example as there are many tools for this but I am sure you can see what I am aiming for) or doing something with the binary then it gets either tedious or complex meaning you probably have to make a program.
    Now comes the debate as various languages have their strengths and weaknesses, generally from a hacking perspective what you want is
    Binary manipulation: few hacks can get away with "standard" formats so this is a must.

    "random" external file manipulation of any type: some languages are designed for a given format or even a given file (sql type languages spring to mind)

    Graphical interface: if you are making a tool you may need to make it so people can manipulate things, of course if you are just making a tool to recalculate pointers you could make it command line quite easily.

    The most common from a hacking perspective are
    internal formats: things like hex workshop have the ability to define file types, most notable use would be Keshire's work on the Jump Ultimate Stars sprites.

    plain C: An old language but a very good one, all the criteria are fulfilled.
    C++: A tweak on plain C and was taught more often in the late 1990's and early 2000's which means most hackers you will meet who were around then will know it.
    C#: A tweak on C type languages by Microsoft although nowadays it is an iso standard and with things like mono other operating systems can use it at some level too. While the above languages share many similarities things are easier to port and/or remake between C# type programs and as there is a framework a lot of the work has already been done for you. Considered by most to be a successor to visual basic (an older language from MS that some older tools are written in, not held in high regard these days).

    All forms of C are similar enough that you can reasonably transition between them all (certainly a C programmer could, as C++ and C# feature more it can be harder to "go backwards"), resource issues aside (a well written c program can invariably beat C++ and c# in size, memory usage and probably speed; hence why it is used for most DS programs), I counsel against landing yourself with something like borland /turbo C++ (it might be easier in the beginning but it will cause headaches later).
    Personally I am useless at any form of C coding (I know the basics but I never took it that far), I plan on rectifying this problem as C is the most commonly used language for computer coding (especially on the consoles) which is very useful to know when reverse engineering (many emulators (ab)use this fact).

    python: deufeufeu did a lot of work with this and for quick and dirty scripts you can not really beat it. It fulfils all the criteria I mentioned above very well. Other languages that share a similar mindset like LISP, perl, lua and scheme are also worth a mention.

    autoIT: not so good at binary manipulation (it works but the tools available are "ground up" only: you will have to shift, add and and perform logical operations to get it where you need it to be) but does make a nice frontend quicker than any other language I have ever used (good for a series of command line tools).

    java: not so common in hacking circles for some reason as it does do what it needs to.

    assembly: very little work on the PC side of things is done in assembly although with the rise of video and audio hacking this looks set to change a bit. Any assembly will probably be "inline assembly" (assembly code that sits inside a program and gets assembled when the rest gets compiled).
     
  6. Sp33der

    Sp33der GBAtemp Fan

    Member
    435
    1
    May 31, 2008
    Netherlands
    Never heard of autoIT hahaha, soms like some varient of BASIC.

    Now my main programming language is Python, now looking at psychoblaster's sample I'm guessing you open the file, reads it and put it in a list?

    And should I just stick with Python or use C#, since most hackers I've seen actually use that language, and I guess I could learn faster looking at samples which are mostly made of C#.
     
  7. Lockon Stratos

    Lockon Stratos GBAtemp Regular

    Member
    262
    0
    Apr 1, 2008
    Sylvarant - Tales of Symphonia
    I tried learning the basics of C# but then I can't find the tutorials and I've sort of learnt about some parts of C# but I don't understand how when you type in strings or any piece of C# code, how it would make something actually appear on the screen like making it work properly. That's one of my main problems.

    PS - FAST can you make that download link another format aswell?
     
  8. tom9927

    tom9927 Member

    Newcomer
    35
    0
    May 21, 2009
    thamks guys alot of the links u linked me too help alot hmm interesting this is gona take some time to learn still reading alot of it

    eidt so far im reading

    http://www.romhacking.net/start/

    alot of usefull stuff and sould be me homewotk for today D ill continue and hopefully when iv learned alot ill start a project up

    projects i hope to do are

    3535_Super_Robot_Taisen_K_JPN_NDS-XPA full english

    reason is i have alot of free time on my hands so im going to learn and hopefully with my free time hack this rom to make it fully english




    UPDATE ok guys iv been reading Hexadecimal(The Basics of Bases) i was learning thiat college so i know a little so thats no problem onto Hex Lesson

    thansk to everyone who made the guides [​IMG]

    ill keep you guys updated stay tuned [​IMG]







    urgent help needed

    i was reading hex editor and he says

    I also suggest grabbing a copy of Castlevania , because it’s my favorite game and the one I use for this document

    which castlvaniagame is he on about nes snes gba ds?
     
  9. DarthNemesis

    DarthNemesis GBAtemp Maniac

    Member
    1,211
    41
    Feb 19, 2008
    United States
    The original, for NES.
     
  10. Lockon Stratos

    Lockon Stratos GBAtemp Regular

    Member
    262
    0
    Apr 1, 2008
    Sylvarant - Tales of Symphonia
    I've really lost hope in an attempt to translate or rom hack since I'm not getting anywhere with my knowledge of rom hacking.
     
  11. Ian10234

    Ian10234 Member

    Newcomer
    41
    0
    May 27, 2009
    United States
    In front of you
    I wasn't sure where to ask this so ill just ask it here. I unpacked a rom but i want to watch the cutscenes (and possibly edit them) Im pretty sure i found them but how do I open them? They're MODS files.
     
  12. Nugg2396

    Nugg2396 GBAtemp Regular

    Member
    239
    0
    Jun 9, 2009
    Malaysia
    Great job on the thread [​IMG]

    and to Ian10234, My name is Ian, and my birthday is on October (10) 23rd!! Lol
     
  13. killakk

    killakk Member

    Newcomer
    27
    0
    Feb 13, 2010
    United States
    Thank you!
     
  14. TempusC

    TempusC GBAtemp Regular

    Member
    229
    0
    Nov 22, 2006
    Canada
    I use python. It’s easy to learn since the console lets you experiment quickly. It’s easy to use since it isn’t such a syntax stickler.

    The struct module allows for easy unpacking and packing of files.

    The string.encode/decode functions and the unicode module allow for conversion between SJIS and unicode (or *shudder* ascii).

    It’s easy to test and to modify since it runs from scripts, so you can try each step out as you code them.

    It works with tkinter, wxWidgets, and PyQt, which are easy to use cross platform interface builders (also available for C/C++).

    It’s completely cross platform - as a Mac dev, I appreciate not having to run a VM or gimp the code for it to work on Windows.
     
  15. FAST6191
    OP

    FAST6191 Techromancer

    pip Reporter
    23,361
    9,156
    Nov 21, 2005
    I made it for another thread and I am too lazy right now to reword it to a more general post but here is a quick how to on crystaltile2, I will see if I can work it into an earlier post as well.

    <b>Introduction</b>
    I know of no guides and while it is a very nice app it is still a very low level application however you slice it so a true guide would read more like a general hacking guide than how to use crystaltile2.
    It is a Chinese application but there are translations, the English one was made by cory1492 and myself (apparently) but between high level technical language, a bit of machine translation at points and the volume of text that often repeats parts of it are a bit crude.
    A note before we start- the basic application is easy enough on your machine but start playing with some of the even basic search methods and you will soon eat memory faster than any memory leak you have ever seen. The more advanced ones will be a true test of your machine.

    I will not be covering emulator integration in much detail because frankly I do not know enough about it to say much other than it exists and as of the later versions it has rather nice support for things like nef files (about as close as we come to decompiling- think of it as ASM with awesome premade comments) and you select your emulator (while it is running/in memory from the file DS emulator options, No$gba is the main one) From there it can snatch the odd thing like a palette or some memory from it which can be useful for basic editing if your palette is dynamic, compressed or otherwise hard to get at in the rom.
    I have however been looking to do this sort of thing since I first discovered the app so here we go:

    <b>General usage/theory</b>
    There are 5 main features of the application and several niceties associated with the whole application.

    tile viewer, hex editor, disassembler, tile editor and text editor represented by the 5 icons on the right hand side of the icon list or the view pulldown menu. The pulldown menus do change depending on what one of these you have selected, they are however usually context relevant (no graphics in the text editor for instance) and the general tools menu should be there throughout (though that too changes). Good practice is to have the editor relevant to what you want open when doing things of that nature.
    Each one is perhaps not as good as a more specific app and indeed the same applies to some of the other options but for breaking into a new rom I know of no better tools and while there are better sub apps most are perfectly functional.
    Where it really comes into play is when you use the DS file system support and some of the extra features which I will cover in a few paragraphs.

    DS file system support. First you need to click on the DS looking icon also on the right hand side.

    This will open a sub window showing the DS file system. The pulldown menus have some good options that I encourage you to scan through.
    Generally speaking though there are three things you want to be looking at on this page

    The icons on the left of the files- these give clues as to what they are and rather nicely if they are compressed.
    The file name for obvious reasons.
    the sub file categorisation on the right hand side.
    The number, address and size data is useful for hacking but here you are probably looking at a more specific hack if you are looking at this. On the other hand if you have opened the entire rom or a subfile in a hex editor it can be useful for tracking it down.

    Basics here are a double click sets the start address of the file you just clicked on in the tile viewer, disassembler and hex editor.
    Right click provides more options, most importantly is the sub file sort (f2 is also a shortcut) and compression support but extract and import are great too.
    Format support in crystaltile2 is the second big strength after DS file system support. Many many file types are supported including most of the big SDK formats like SDAT, NARC and the graphics NCER, NCLR and so on.
    SDAT, .bin (assuming it is something like utility.bin) and NARC will open new windows much like the one you were just working on although in the case of SDAT there is some somewhat broken playback options.
    Right click on the graphics will allow you to do one of three things (more on graphics here: <a href="http://tahaxan.arcnor.com/index.php?option=com_content&view=section&id=7&Itemid=36&lang=en" target="_blank">http://tahaxan.arcnor.com/index.php?option...=36&lang=en</a> )
    This is load the data, order the data (tiles can be in an order in the rom) and apply a palette.
    This leads to stuff like <a href="http://pix.gbatemp.net/32303/gundam2.JPG" target="_blank">http://pix.gbatemp.net/32303/gundam2.JPG</a>

    Compression support is another strength, for the longest time crystaltile2 had some of the best compression support in general rom hacking tools (indeed some of the new LZ flags were supported long before anything else). It works in much the same way as import and export files does but it can use compression algorithms at the same time.
    More general compression options are available under the tools menu including a general compression app and rather nicely a compression search that rivals anything else I have seen.

    After this you have a NFTR editor (a relatively common font type in DS roms) which includes font conversion abilities (as in you grab a PC font and it will spit out a NFTR font)
    Multimedia editor- more aimed at animation than sound. Would probably get a call from Nintendo's legal types were it not in China.
    Below that is a rather crude OCR app that in my version at least seems to have escaped translation but it is the only one I have ever met in rom hacking. I use them day in day out in video and images but for rom hacking...... wow.

    Onto the main tools side of things. You have a tile viewer, hex editor, disassembler, tile editor and text editor.

    <b>Tile viewer</b>
    Assuming you have not wound DS formats into it the tile editor is a fairly full featured one. It is not as pretty as some others (indeed I still keep a copy of TilEd 2002 around for really crude work) but it has great features like nonstandard tile size (7x13- not a problem).
    First thing to note is the keyboard shortcuts which do nice things like change starting offset which is nice for dodging headers and whatever else. Everything else is self explanatory aside from perhaps tile form(at) which allows you to chose between all the various image modes (and there are a lot) that crystaltile2 supports. Most useful are GBA 4bpp, GBA 8bpp (the modes used by the GBA and DS, note that most GBA stuff is 4bpp while 8bpp is very common in the DS) and nds1bpp (useful for a certain type of packing used sometimes in fonts) but play with it all.

    If you higlight a selection of tiles you can export it to a more common format (word to wise, stick with lossless if you plan on dragging it back into the app), 1:1 is a "regular expression" style exporter.


    <b>Hex editor</b>
    A hex editor, rather basic but for the table support and inbuilt support for shiftJIS (back when it was one of the very few that did making it a very valuable tool)
    Use of tables is chosen in the box on the left (same place as the tile editor before it) and more general codepages can be chosen by clicking on the section above the text decoding (probably saying Western European (Windows) at first).
    In the box the options other than table support is choice of known codepages
    choice of sort options (1 byte, 2 bytes or 4 bytes)
    colour character- not as useful as it might be in other areas as it can be a bit hard to read at pace unlike tiles but to each their own.

    Data to palette and palette to data do as they say on the tin and grab the hex and stick it in the palette and vice versa. A nice shortcut for some but probably not all that useful as most games I have ever tangled with tend to opt for the RAM to do things to the palette or in the case of the DS have it wrapped in some format.

    <b>Disassembler</b>
    Rather basic crude disassembly supporting ARM9 and ARM7 including THUMB for each. You force change between these modes in the same way you changed premade code tables (clicking where it says ARM? .
    You can get base address at the top of the "effective address" list.
    You can an offset (goto address) too in the box on the left

    Assembly is a topic for coverage elsewhere but you have effective address, hex code for the instruction and the mnemonic and registers/addresses* it deals with.

    *this is one of the things NEF files can sort for you- if a given address is a given thing the NEF file will make it change to an even more readable format.

    <b>Tile editor</b>
    Unless you have an image made up from DS format options it will just have the tile you have selected in tile viewer.
    Fairly basic, you snatch colours from the palette by clicking on the button and choosing it,
    left is char, right is BG.

    usual warning about palette based graphics- a change of one thing can have effects on another if things are reused which they often are and likewise they are palette not bitmaps so changing a colour in the palette may not necessarily translate to the real game.
    Much like the other palettes and the hardware itself you have 16 available (0 through F).

    <b>Script editor</b>
    Originally a standalone app from the same author, most useful is probably the table support and the script search (it is an upgraded version of the search strings option (namely table support) you may have seen in various hex editors), under the tools menu it is called Ambassador search.

    If you already know text editing the rest is fairly obvious. Exit code= line end/new line sort of thing (0D0A in windows .txt for instance, 0000 in bmg files (a somewhat uncommon DS SDK text format))

    Geared towards the GBA mainly but can be pushed towards the DS side of things.

    In the translation I have in front of me "censorship" allows you to narrow down the broad/"crude"* search but it will require some tweaking/defining what you want.

    *the methods/algorithms behind these techniques are anything but crude but the results they produce are far from "magic application" grade.

    Annoyingly the relative search (I personally use monkey moore for such things: <a href="http://www.romhacking.net/utils/513/" target="_blank">http://www.romhacking.net/utils/513/</a> ) is back on the hex editor menu under the search or the tools pulldown menus for the version I have in front of me. It is a fine relative search with many options including 4 byte search which is insane (I have only ever heard of 4 byte fonts- 4294967296 possible characters, 16 bit is 65536 which is more than enough for pretty much every character in every commonly used language).
     
  16. jjjewel

    jjjewel GBAtemp Maniac

    Member
    1,009
    293
    Dec 17, 2009
    United States
    FAST6191, I'm just getting curious about NEF you mentioned in your article. Do you have any links that give more information about it? (I searched and only got Nikon Electronic Files and I don't think that's something related to rom hacking. [​IMG])

    I totally have no idea about ASM hacking but that's something I think I should start to learn at some point.
     
  17. FAST6191
    OP

    FAST6191 Techromancer

    pip Reporter
    23,361
    9,156
    Nov 21, 2005
    I do not have a great deal of info on them or even that much experience with it but I will give it a go:

    NEF is a file format apparently used by Nintendo but supported in the developers version of no$gba (a far more interesting prospect for hackers) and later in the line of things crystaltile2, in practice is acts like an external comments system ("the commands at a given address do X" sort of thing) mixed with a debug info (breakpoints and whatnot).

    The reason this sort of thing is interesting is because when you first disassemble a rom (the DS is quite nice as you have the filesystem- older consoles are not so lucky) you get several thousand lines of barely comprehensible instructions ( http://gbatemp.net/index.php?showtopic=39979&hl= is worth a scan through as it has a bit of info/provides a good example) and indeed owing to developers leaving/reading the game text/pictures/levels and the like from/in the binary and the fact the ARM processors in the DS having two differing instruction sets (ARM and THUMB: http://nocash.emubase.de/gbatek.htm#cpuoverview ) it may indeed be entirely useless (your basic disassembler is a very very crude tool- it is on a par with the ASCII readout of your hex editor in that any good it shows is luck, use of standards or you manually guiding/prodding it to show as such). Anything that cuts this down and makes it more manageable is welcomed, using the NEF files you can essentially comment "out" the things like setting the stack pointer, changing CPU modes and ultimately leave you looking at the juicy stuff like what instructions deal with your chosen interest for hacking or more broadly what deals with what. Similarly it can kind of cut down on the memory/register management side of things that most of ASM is concerned with and most high level languages seek to do away with)- I am loathe to use the word decompiler but they are in the same circles.

    Hopefully I am allowed to paste it here but here is the relevant section from the no$gba debug help (note the passage on SRL- you normally see that format in roms as the file sent over wireless for download play):
    no$gba help quote


    I would argue it is not that relevant to the hacker new to ASM, you would probably be better served reading up on the likes of http://crackerscrap.com/index.php?p=docs http://gbatemp.net/index.php?showtopic=444...t=0&start=0 and http://www.romhacking.net/?category=&P...itle=&desc= (pretty much all of the documents there but the ones on VFW, compression and VBA-SDL are the big three)

    desmume (now a proper dev version exists) is also good enough for some ASM work (it plays well with later roms too unlike no$gba).
     
  18. jjjewel

    jjjewel GBAtemp Maniac

    Member
    1,009
    293
    Dec 17, 2009
    United States
    [​IMG] FAST6191, thank you so much for the info and all the links. I started reading some of them already. [​IMG]
     
  19. Krobelus

    Krobelus GBAtemp Regular

    Member
    163
    2
    Apr 9, 2009
    Canada
    Vancouver
    I have a question; I used thenewpoketext to create the tmp folder of Pokemon Platinum. Then after I closed thenewpoketext.

    Now that I've finished using ppre I want to compile the tmp folder, but thenewpoketext wont compile it.

    How could I compile my tmp_Pokemon Platinum folder?

    Thanks
     
  20. loco365

    loco365 GBAtemp Guru

    Member
    5,458
    2,674
    Sep 1, 2010
    I dunno about reviving stickies here... But what is the catch on looping in a ds sseq? I have been trying to crack that for a while and haven't gotten too far.