GBATemp ROM hacking documentation project (new 2016 edition out)

Been spending a lot of my days redoing the guide* and getting some stuff up on filetrip for it. I have very nearly finished the text section (just some basic usage on Atlas and Kruptar even though I am not entirely sold on their usefulness for a lot of what we deal with on the DS) and as part of that though I was checking out alternative and although I am late to the party I had not seen it discussed around here
Links
http://www.magicteam.net/index.php?page=programs
http://filetrip.net/pc-downloads/applications/download-oriton-2050-f29376.html


Oriton is a table making program and for some purposes it may edge out TaBuLar (the only real downside is it does not have nice options for adding the various orders of Kana and the Roman alphabet- right click and you can add a codepage but not a lot else or you can just visit http://www.rikai.com/library/kanjitables/kanji_codes.sjis.shtml and engage in a bit of copy and paste). It does however have serious support for 16 bit and beyond encodings where others fall flat and a really nice regular expression style input box as well as Kruptar support (the other main competitor to Cartographer and romjuice for text extraction).
Basically if you play around with table files on a semi regular basis definitely give it a go.

*sitting around 85K words with a tiny bit on 2d, a lot on 3d, some worked examples in audio hacking, more or write the entire video section, some more ASM stuff, maybe some more on stats and level editing and a bit on python to do. Alas it is a bit raw right now in places or I would do a preview release.

 

Double post I know but I just wanted to note in this thread beyond an edit to the first page and a portal post that the 2012 rewrite is well under way and a very early preview of them is up on filetrip. PDF only for now but I will convert them to HTML properly in the near future (right now it wants to make 9 megs of HTML and pictures on a single page). Suggestions, typo fixes and more are welcome. Part III needs a fair bit of work and there are a handful of GBA topics that want some work (some more on sappy audio and affine sprite transformation) in Part II but part II which contains discussion of the techniques used in rom hacking and GBA/DS hardware is for the most part done.

Filetrip download

Spoiler: List of contents of new documents

I Introduction 11
II Rom hacking concepts 14
1 Basics 14
1.1 Hexadecimal 14
1.1.1 Representation 16
1.1.2 BCD (Binary coded decimal) 17
1.1.3 Big and little endian 18
1.1.4 Signed values, floating point and fixed point 18
1.2 Hex operations 23
1.2.1 Shift 23
1.2.2 Rotate 24
1.2.3 Flip 24
1.2.4 Boolean logic 25
1.2.5 Hex Mathematics 27
1.3 Patching and patch making 28
1.4 File systems and operations 29
1.4.1 GBA 30
1.4.2 DS 30
1.4.3 GC (gamecube) 31
1.4.4 Wii 32
1.4.5 Xbox 32
1.4.6 Xbox 360 33
1.4.7 PS1 and PS2 34
1.4.8 PS3 34
1.4.9 PSP 34
1.4.10 Saturn 35
1.4.11 Dreamcast 35
1.4.12 Amiga 35
1.4.13 PC and related hardware 35
1.5 Finding the object of your interest 36
1.6 Abstraction 37
1.7 Tools of the trade continued 38
1.7.1 Hex editor 38
1.7.2 Tile editor 50
1.7.3 Spreadsheet and command line 54
1.7.4 Compression 56
1.7.5 Music 57
1.7.6 ASM 58
1.8 Basic file format concepts 62

2 Graphics 64
2.0.1 Aliasing 65
2.0.2 Haloing 66
2.0.3 Bit depth 66
2.1 Palettes and colours 66
2.1.1 GBA colours (15 bit) 66
2.2 Tiles 67
2.2.1 1Bpp 67
2.2.2 4 Bpp 68
2.2.3 8Bpp 68
2.2.4 GBA3 Xbpp 69
2.2.5 GBA2 4BPP 71
2.2.6 Bitmap 74
2.2.7 Known formats 74
2.2.8 Crystaltile2 export and import 74
2.2.9 Avoiding gradients, AA, lossy/noise and such things 79
2.3 Layout, timing, OAM and special effects 79
2.3.1 Introduction to the OAM and BG modes 79
2.3.2 Timing 80
2.3.3 GBA and DS OAM (sprites) 81
2.3.4 GBA and DS BG modes 84
2.3.5 Basic animation 88
2.3.6 Window feature 93
2.3.7 Special features (flipping, affine transformation, alpha and such) 93
2.3.8 Basic DS layout formats and mapping 95
2.3.9 Video memory handling and alignment 98
2.4 3d 98
2.4.1 Basic 3d (bones, coordinates, keyframes) 99
2.4.2 Viewpoints 101
2.4.3 Textures and material colours 102
2.4.4 Models 103
2.4.5 Lighting/shadows 104
2.4.6 3d smoke and fog 105
2.4.7 Animations 106
2.4.8 DS 3D hardware 107
2.4.9 The shift of the 3D to DS 2d 109
2.4.10 NSBMD 109
2.4.11 Non NSBMD 119
2.5 Notes and further reading 120

3 Text 121
3.1 Tables 121
3.1.1 Relative searching 123
3.1.2 Corruption and alteration 130
3.1.3 Memory viewing and corruption 132
3.1.4 Frequency analysis 134
3.1.5 Language analysis 136
3.1.6 Pointer and encoding/hex analysis 137
3.1.7 Assembly tracing 137
3.1.8 Font viewing 137
3.1.9 Language comparing 138
3.1.10 Table creation tools 138
3.2 Pointers 141
3.2.1 Special cases and non pointer concepts 142
3.2.2 Example reverse engineering of pointers 143
3.3 Markup, control codes and placeholders 148
3.3.1 Worked example 148
3.4 Fonts 153
3.4.1 NFTR 154
3.4.2 Common hacks 160
3.5 Scripting and layout 163
3.5.1 Layout and limits 171
3.6 Text extraction and insertion 172
3.6.1 Text extraction 172
3.6.2 Text insertion 175
3.7 Language detection in DS games 177
3.8 Translation hacking 178
3.8.1 The types of Japanese characters and how they work 179
3.8.2 Japanese glyphs/characters and observations on the language 182
3.8.3 On language 183
3.9 Japanese text editors and translation tools 184
3.9.1 General Japanese capable text editors 184
3.9.2 Rom hacking tools 184
3.9.3 CAT tools 185

4 Multimedia 186
4.1 Sound 186
4.1.1 SDAT (NDS) 190
4.1.2 Others 198
4.1.3 Tracker formats 199
4.1.4 General rule of thumb for custom audio formats 200
4.1.5 Common DS SDAT audio hacks (undubbing, injection,
tweaks and relinking) 200
4.1.6 GBA audio 220
4.2 Video 224
4.2.1 General video theory 225
4.2.2 Mods/VX/act imagine by Mobiclip 226
4.2.3 RAD/Bink 226
4.2.4 Criware 227
4.3 Cut scenes 227

5 Game logic 229
5.1 Levels and Stats 229
5.1.1 Example tools 231
5.1.2 Level editing techniques 232
5.1.3 Stats 243
5.2 Compression 244
5.2.1 Lossy 245
5.2.2 Lossless 245
5.2.3 Basic theory of the actual implementations 246
5.2.4 Compression at hexadecimal level 251
5.3 Cheating 254
5.3.1 General cheat making 255
5.3.2 GBA cheat making 256
5.3.3 DS cheat making 258
5.3.4 Basic making of a cheat 262
5.3.5 Cheat prevention methods and frustrations 265
5.3.6 Instruction editing cheating 269
5.4 Programming concepts 272
5.4.1 Functions and procedural programming 272
5.4.2 IF ELSE 273
5.4.3 Recursion 273
5.4.4 Iteration 273
5.4.5 Loops 274
5.4.6 Turing complete 274
5.4.7 Fundamentals of Assembly 274
5.5 Assembly 278
5.5.1 ARM 279
5.5.2 GBA Assembly specifics 280
5.5.3 DS Assembly specifics 285
5.5.4 The GBA and DS compared 289
5.5.5 On controls 290
5.5.6 Hooking 291
5.5.7 GBA cart as extra memory for DS hacks 292
5.6 Non specific assembly discussion 292
5.6.1 Language mod example 292
5.6.2 Non code in ASM 296
5.6.3 Destructive vs non destructive assembly editing 296
5.6.4 Polymorphic and dynamic code 297
5.6.5 Slowdown and speedup 299
5.6.6 Cryptography (encryption, checksums and signatures) 300
5.6.7 Save editing 306
5.6.8 Interpreted languages 307
5.6.9 Game AI, game logic and game theory 307
5.7 Flash cart and emulator theory 311
5.7.1 GBA 313
5.7.2 DS 315
5.8 Rom hacking "protection" 317

III Examples, oddities and techniques319
6 Crystaltile2 general usage guide 319
7 GBA tracing 325
7.0.1 Worked examples 326
8 DS tracing 326
8.1 Cart read command 327
8.1.1 Basic lookup and methods for it 327
8.1.2 Header reverse engineering/generated values 327
9 Reverse engineering various ROM images 327
9.1 Large archive on top of filesystem 328
9.1.1 Tony Hawk 328
9.1.2 Star Wars - The Force Unleashed 328
9.1.3 El Tigre Make my mule 328
9.2 Compression 328
9.3 First Person Game 329
9.4 Platformer 329
9.5 Fighting games 329
9.6 Role playing games 330
9.7 Racing games 330
9.8 Puzzle 330
9.8.1 Mahjong game 331
9.8.2 Tetris 331
9.9 Other genres 331
10 Developer leftovers 331
11 Workarounds 332
12 Moving to a new system 332
13 Developer tricks aka thinking like a game developer 333
13.0.1 Level and mechanism design 334
13.0.2 Sprite and palette reuses 335
13.0.3 Pre rendering 335
13.0.4 Speed blur and fog 335
13.0.5 Loading covers 335
13.0.6 Optimisation of loading 336
13.0.7 3d imagery in general 336
13.0.8 Procedural generation 337
13.0.9 Noise on images and sound 338
13.0.10Using the limits of the system/working to them 338
13.0.11Network coding 339
14 Game design and media 339
15 Python, batch files and programming for rom hacking 340
15.1 ROMulan Data Extractor/Injector 340
15.2 Programming languages 341
15.3 Python 341
16 PC program hacking 342
16.1 Debugging 343
16.2 Decompilation 343
17 Version control and project management344
17.1 Project and team management 344
17.2 Version control 346
18 Interesting links and further reading347
18.1 Links 347
18.2 Further reading 347
IV File formats (specications, methods and known formats)349
19 General things about the DS 349
20 Generic DS nitro SDK format 349
21 General file reverse engineering 349
21.1 Headers 350
21.2 File sizes 350
21.3 Multiple versions of the game 350
21.4 File names and extensions 350
21.5 Tile viewers 350
21.6 Pointers and such 351
22 Sound 351
22.1 SDAT 351
22.2 SSEQ 352
22.3 STRM 352
22.4 SWAR 352
22.5 SWAR 353
22.6 BANK 353
22.7 Other formats 353
23 Graphics 354
23.1 NCER 354
23.2 NANR 355
23.3 NCGR 355
23.4 NSCR 355
23.5 NMCR 356
23.6 NFTR 356
23.7 NSBMD 356
23.8 NSBTX 357
23.9 NSBCA 357
24 Packing format 358
24.1 NARC, ARC and CARC 358
25 Text 358
25.1 BMG 358
V Glossary, index and such 360
26 Glossary 360

 

Works fine for me. Still mirror if you want it http://trastindustries.com/romhacking2012preview2.pdf I can not promise I will keep it up to date though so best if you do not share the link far and wide.

Edit, on an entirely unrelated note and although I am quite happy with arm-eabi assemblers and armips some might like http://arm.flatassembler.net/ which I ran across the other day.

 

Owing to no$gba being free and having changed domain (only a basic redirect seems to be in place) I thought it best to update it as several links inside the document were kind of broken as a result, the specs have been updated as well so I am hoping the headings/urls have not changed within the gbatek document itself. I have been updating pieces over the last couple of years, playing editor and have even added a couple of new sections. However if something was not finished in the previous release then chances are it is not finished now. Equally some tools have been updated a bit, especially vgmtrans, no$gba and desmume, so some of that is a bit outdated (and not just from my old XP machine using the themes of even older versions of windows).
I may well have a bit of free time over the coming months so I will be doing more, however with the no$gba changed domain it felt worthwhile to spin out a new version. Still just PDF for the time being, HTML (which will probably have some videos for a few things) and the source of the document will have to come later. I did also have a version with slightly more sane borders in the works but that got sidelined for the time being (a lot of the images sizing and placement has been tied to text/column size).
Speaking of image sizes if they are too small to see it should just be scaling so you can zoom in on your PDF viewer and see them at the size they should be at.

You can get it on filetrip:
http://filetrip.net/nds-downloads/u...-rom-hacking-guide-2014-preview-1-f32908.html
I will have a version on my server at some point as well.

I have since moved to Linux for my main machine so there may be some fallout thanks to having to migrate from mitex on windows to texlive on Linux (I am not cool enough to use tex/latex so I use Lyx, getting it back to the point where it could spit out a PDF saw me have to learn more about tex and latex than I really wanted to do yesterday afternoon). It does seem to be reflected in the font and some of the pictures.

I don't think I will frontpage this one for the time being but if people want to share it then feel free.

If you encounter broken links in GBAtek then please tell me and I will see what can be done, likewise if there is some image placement is too off (I did decide to do some stuff with boxes to keep related text and images together) or some other broken URL then do also tell me.

 

I tends to help to say what game you are on about. Egg Gauntlet would then appear to be a bonus mode that was cut from the eventual release of Sonic 2 remastered, however for the first version (later updates removed them) the files for it all were left in there. I imagine you could stitch it back together and have it available somewhere easy to select but given I only learned what it was about 3 minutes ago I do not know a method offhand.

Mother 3 beta is new to me. Is this an official beta or some kind of beta of the translation? The answer remains mostly the same either way but it can change how I play it. Generally it depends upon how the extra stuff from the beta was removed, how it was coded originally and what space you have to play with (mother 3 being especially noted for a lack of space, doubly so with the translation applied).
If the devs just removed an item from the map and called it a day you just add it back to a map/shop and you have it back. If they removed the art and the code for it then you get to add that back in which is easier said than done. If they replaced it wholesale (128 item slots, 128 items, 2 removed and 2 new items added for instance) then you have a fairly hard hack on your hands, not that the one before it always easy.

 

With google code possibly being on the way out ( https://code.google.com/p/support/wiki/ReadOnlyTransition ) and otherwise having a few broken links I thought I would update some things, I need to find another copy of cracker's newer ARM ASM kit ( armasmkit.rar , see also https://web.archive.org/web/20100323015315/http://crackerscrap.com/projects.php ), it is probably on an old hard drive somewhere though. I have not added much (a very short section on RPG randomisers is the main thing, some of the more advanced parts of GBA audio hacking are still unfinished, http://www.feshrine.net/hacking/Fire Emblem Music Hacking Tutorial.pdf has more if you really want it). That said I have continued to tidy some stuff up, modernise a small handful of things, expand on some other things and generally tweak things as I see them.
Still no HTML version, though I have moved the page format (if not much of the text/margins) to A4 from US letter which it was before in anticipation of this.
I plan to get back into this far more so this will just be an early preview and only posted to sort some of the broken links issues. As such I will likely not frontpage it here, though if you want to update your mirrors or share it with your mates then go for it.
Download
http://filetrip.net/nds-downloads/u...-rom-hacking-guide-2016-preview-1-f33419.html

Google code wise I have left a few things in there and put github links wherever they need to be. I am slightly concerned as some google code pages had compiled binaries where github did not always carry them (or it puts them under the releases tab). I will address that at a later date though.
romhacking.net have recently gone register to download to help dodge some issues with google calling them a malware pusher (very much a false positive). I did not change much here, hopefully most of the key tools should be on filetrip. When I get back to editing in earnest I will mirror everything I can.

I also spoke to the monkey moore author as well so just to help searches update then he moved from google code to github, and linked me to https://github.com/rjricken/monkey-moore should you want such things.
Most others seem to have a "moved to" link, if not a straight redirect.

 

The website stuff is ancient if you have been looking at that, I abandoned it long ago so if it is that you are referring to then it is never going to get fixed.
Just downloaded a fresh copy of the pdf from filetrip. Worked fine in http://www.gnome.org/projects/evince 3.something and has worked fine in other readers I have tested it in for other computers since the version made earlier this year. That looks like an encoding issue for your setup, I am not sure what it would be but if you have forced it to try to decode a certain language then it might be that.
There are a bunch of blank tables at the end as I never got around to filling them in but had the tables there ready to go. Most things that would be covered in it are covered in http://llref.emutalk.net/docs/ and http://www.romhacking.net/documents/469/ and other links scattered throughout the relevant sections.

 

Hmm, got that as well. Probably still an encoding issue somewhere along the line as it seems to match length after a fashion and is repeatable. I certainly did not intentionally protect or wish to stop anything like that.

Spun off another version using a different exporter
http://trastindustries.com/randomfiles/romhacking2016_copy_1.pdf
Not sure what I have changed since last time but nothing major. Others reading if you are happy with what you have then no need to get this, I can't promise that will still be there at some point in the distant future.

Image quality will probably not be as great, though nothing was intended in that as a photo and it all appears just as readable as it ever was, and some of the footnotes and formatting needs work but it does appear to allow you to copy from it.