ROM Hack Pokemon DP Source Code How to use it

[Razor1993]

Hey Yall,
I know,sharing data from the Leak is forbidden, thats not what this Thread is for

This Thread is only for Learning purposes to understand DS Pokemon Game Development.
Can a Genius pls explain the following questions for me:
- How to Setup an development enviroment for that leak?
- How to compile a working rom?
- Is there any more documentation i can learn from?

Thanks in advane
Best wishes Razor1993

Edit:

Here is the way to setup a development envirement for the leaked Source Code:

1. Get/Find yourself the "PKMN Build Machine v2" from the Leaks. Thats the Development Enviroment for all Pokemon Games that you need to run on Virtualbox with Windows XP x32.
2. Now Setup Virtualbox with Win XP 32Bit and give as much ram as you want.
3. On the Step where your are going to configure the Hard Drive Choose "Use an existing virtual Hard disk drive" and choose the VMDK File that came with the PKMN Build Machine v2.
4. The Source Code of Pokemon DP us already in c:\gen4src.
   
5. To Build that, just simply go to Desktop and then: Build Scripts/Gen4/DiamondOrPearl/Release/ and click on Build Enviroment.bat.
   
6. That is gonna compile that Source Code to a Rom in Desktop/RomOut

 

[FAST6191]

" - How to Setup an development environment for that leak?"
Not going to happen. DS source code made within the standard development environment (which pokemon and probably 99% of other games were -- you have a few random scripting language games that kind of defy tradition) have a rather custom setup -- it is still C family in the end but it was all fairly locked down and twisted in towards itself (see precompiled headers) in addition to the usual having to play to the DS hardware ( http://problemkaputt.de/gbatek.htm ) rather than Windows or Linux or whatever calls and responses to input and output. There is a very early leaked version of the devkit but this was years before the games in question were properly started and the later stuff (which only a handful of private individuals had access to, not sure even they do any more as things used to phone home a lot and we are now years out of DS games being current) saw many further developments to the libraries, formats and the like meaning it is nearly worthless here.
Basically if you are coming here from something like the original xbox where you download a relevant version official SDK and go from there then not so much. This in addition to most of the things made available to generate music, 3d graphics and whatnot being plugins for now ancient versions of high end professional software. No homebrew devkit or library setup ever came close to wanting to replicate the official software setup either.
" - How to compile a working rom?"
Not going to happen. At least short of a radical reworking of the code for you to either donate the precompiled stuff and be limited by it or figure out what it expects (you can do much from context and basic searching will tell you more)
" - Is there any more documentation I can learn from?"
Any C and C++ tutorial should do, as will any existing analysis of pokemon mechanics of these games (there are usually a fair few) and maybe their predecessors (not like much changes between games).
If you are good enough at C you can spend the dozens of hours needed to replicate the headers and related stuff such that you can compile it with devkitpro or whatever, or the hundreds you might want to port out the code and remake the game from the ground up (but with the code there to maybe copy and paste a bit at a time and be assured you are doing it 1:1 with what the original devs wanted but be far more extensible or able to be ported to other platforms far more easily).


By all means read the source code (assuming you wish to shoulder the legal issues anyway) so you can write things like https://www.dragonflycave.com/mechanics/gen-i-capturing as a result but for whatever aspect you care about here, learn how a format is supposed to be read (or maybe what parts are there but are not used at all by the game), find any hidden functionality buried within the game, maybe do an analysis to figure out an exploit or what caused a bug, maybe there are some amusing or enlightening comments, or you could direct an assembly hack without first having to find the relevant assembly in the binary, work backwards from that to get a higher level understanding of the mechanics involved and then proceed to edit things (we have had something resembling compilation and injection done on other things as well so that could also be a possibility).

 

[CHEMI6DER]

I think compiling the source code is doable, I even almost got it to compile. I used a Windows 7 VM onto which I installed the TWL SDK (and a ~2010 version of it can be easily found on the internet, just google it), CodeWarrior for DSi (should be alongside the TWL SDK download), and Cygwin with Ruby and an old version of GNU make (cause the newer one stalls at one of the SDK files for some reason, version 3.80 worked fine for me). Additionally you'll need to convert the source files from Shift-JIS to UTF-8 if you want this to compile on a non-Japanese version of Windows (since otherwise Ruby errors out). I didn't quite get it to compile yet because it turned out my Windows VM was broken and many SDK tools would error out with 0x0150002 on launch, so I'm reinstalling Windows to try this again.
Again, in theory this should be compile-able if things go right. I'll post an update in case I get this to compile with more detail on what I did.

 

[FAST6191]

Did we see a proper release of the DSi/TWL stuff?

I was only ever really aware of the early stage codewarrior thing, a couple of 3d plugins from later stuff, the few things fished out from ROMs that the devs left in and ensata (the in house emulator that was technically the first emulator available, long since surpassed by others though I see we with these leaks now have new options).

Some of the people making flash cart firmwares had access to other things via friends in the industry but they kept that too themselves (fear of watermarks, endless phone home annoyances and more besides).

 

[CHEMI6DER]

Oh yeah, all the SDK stuff has been leaked online. CW, TWLSDK and TWLSYSTEM, which is all the proprietary stuff required to do s/w development. The most recent one that can be found even has a CW license for up to some point in 2016. (but what I'm using is a bit older, sourced from SDKs.rar which has DSi, Wii, GC and even a WiiU thing)

Also, I've gotten a bit further in my compilation attempts. Now that I got the Windows issue sorted out, I have new problems with it. The tool GF made for converting xls to csv errors (which makes a Ruby script error out because the file doesn't exist...and so the make routine for the maps never exits successfully)...this probably requires MS Office, since this program makes references to Excel's OLE server...maybe I can bodge a thing to convert these without the tool, or make it work somehow
The SDK has a weird thing called make-verinfo, and I don't understand what it is for...it's just a .cc and .ld file in the SDK build tools folder...but the SDK makefile calls it as if it's a command
The compiler seems to be working fine and produces object files...I'm kinda concerned how Cygwin turns paths into a mess (like C:\TwlSDK\build becomes \cygdrive\c\c\twlsdk\build, instead of the correct \cygdrive\c\twlsdk\build), which seems to cause some sort of error in the end, but somehow stuff manages to get compiled even though the path is messed up...

 

[Razor1993]

OK, thx to Cosmo, I found a way how to compile that Leaked DP Source Code. But I have to say that the Source Code of dp is such a mess... not compareable to the dissasemblys... I couldnt edit a single file... and there are so many folders with so many confusing names...

Anyways... Here is the way to setup a development envirement for the leaked Source Code:

1. Get/Find yourself the "PKMN Build Machine v2" from the Leaks. Thats the Development Enviroment for all Pokemon Games that you need to run on Virtualbox with Windows XP x32.
2. Now Setup Virtualbox with Win XP 32Bit and give as much ram as you want.
3. On the Step where your are going to configure the Hard Drive Choose "Use an existing virtual Hard disk drive" and choose the VMDK File that came with the PKMN Build Machine v2.
4. The Source Code of Pokemon DP us already in c:\gen4src.
   
5. To Build that, just simply go to Desktop and then: Build Scripts/Gen4/DiamondOrPearl/Release/ and click on Build Enviroment.bat.
   
6. That is gonna compile that Source Code to a Rom in Desktop/RomOut

 

[MarioBrosThe]

OK, thx to Cosmo, I found a way how to compile that Leaked DP Source Code. But I have to say that the Source Code of dp is such a mess... not compareable to the dissasemblys... I couldnt edit a single file... and there are so many folders with so many confusing names...

Anyways... Here is the way to setup a development envirement for the leaked Source Code:

1. Get/Find yourself the "PKMN Build Machine v2" from the Leaks. Thats the Development Enviroment for all Pokemon Games that you need to run on Virtualbox with Windows XP x32.
2. Now Setup Virtualbox with Win XP 32Bit and give as much ram as you want.
3. On the Step where your are going to configure the Hard Drive Choose "Use an existing virtual Hard disk drive" and choose the VMDK File that came with the PKMN Build Machine v2.
4. The Source Code of Pokemon DP us already in c:\gen4src.
   
5. To Build that, just simply go to Desktop and then: Build Scripts/Gen4/DiamondOrPearl/Release/ and click on Build Enviroment.bat.
   
6. That is gonna compile that Source Code to a Rom in Desktop/RomOut

When I run the vm the VMDK file, all I get is a black screen. Might you know why this is happening?

edit: figured it out

 

[Razor1993]

When I run the vm the VMDK file, all I get is a black screen. Might you know why this is happening?

edit: figured it out

We or someone should make a documentation on the DP Source, so we can read it better

 

[CHEMI6DER]

We in fact should make a doc. Not just on the code itself, but how to set up the build environment for it too. Cause, as good as the build VM is, it has problems IMO:
1 - it's pretty large when it's unpacked
2 - it's a VMWare VM, not VirtualBox (trying to just boot the VMDKs on VBox resulted in a black screen for me), which means you need VMW to run this thing...and as I found out, VMWare doesn't play very nice with Linux for some reason. Made my whole computer hang up a few times (and no, my h/w is good enough and I did not hit the pagefile)
From what I could see with the VM, I was pretty close with what I was doing with my attempts. (but I had to alter the source files, while this setup compiles the original source code)
I guess I can document whatever the VM had in case anyone wants to try to make their own (and I'll probably test this later myself if I have enough disk space):

Spoiler

- Japanese version of Windows XP (Google's your friend here, easy enough to find), you can install an English language pack into it if you want to
- MS Office 2003 SP3 (you specifically need Excel, so make sure you install it)
- CodeWarrior 1.2 for DS (this is the IDE and compiler thingy)
- NitroSDK 3.1 (the DS SDK)
- NitroSystem (libraries for dealing with SDK 2D, 3D, sound and font formats, also has tools for making those files and plugins to export intermediate files from s/w like 3DS Max, Maya, Photoshop and Softimage (or at least they added in the latter in TwlSystem, but GF, being a 1-st party dev, had their hands on it pretty early it seems))
- NitroMiddleware-libVCT 1.0.1 (a voice chat middleware library?)
- NitroWifi 1.1 (the DS Wi-Fi library)
- NITRO-DWC 1.2 (some sorta networking library)
- Cygwin of some version (I wonder if it can be replaced with MSYS2, since IMO Gygwin is a pain in the rear end to use)
- Ruby (the folder is called ruby187, so I guess version 1.8.7?, but I think a newer version should work just fine)

Additionally you'll need to add a few environment variables in Windows (Ggoogle how to do it. It's easy, just a pain that the window to edit these in Windows is very tiny and can't be resized)
For all of the above components I'd suggest you to install it somewhere in the root of the C:\ drive and avoid having non-ASCII characters and spaces in the file path to avoid potential issues.

Spoiler

CW_NITROSDK_ROOT and NITROSDK_ROOT need to be set to path where the NitroSDK is installed (ex. "C:\NitroSDK")
CWFolder_NITRO to path where CodeWarrior is installed (this folder should contain ARM_EABI_Support and ARM_Tools folders among other CW stuff)
NITROSYSTEM_ROOT to where NitroSystem is located
NITROVCT_ROOT and NITROLIBVCT_ROOT to NitroMiddleware-libVCT path
NITROWIFI_ROOT to NitroWifi path
NITRODWC_ROOT to NitroDWC path
LM_LICENSE_FILE to path to the CodeWarrior license.dat file (this VM seems to have a hacked license file with doesn't have a date set for any of the components)
You'd also want to add the bin folders of Cygwin and Ruby to PATH.

I guess that's about it for the build environment. To compile the code, go into the source code folder, open the Cygwin bash prompt and command "make clean" then "make" to compile everything.
Additionally, you'll need CVS if you want to check out different versions of the code base and compile them...it didn't work for me any good using CVS on Linux and then sharing the checked out repo folder with the VM, so a much better way IMO is to use cvs-to-git to convert the whole repo to Git and use it like that (maybe it's just me, but I'm way more familiar with Git and it's less of a pain to work with compared to CVS)

As for the source itself, I can give some info on what I found out as of now going through it and reading stuff online:

Spoiler

- convert/message/src has all the game texts, seemingly the makefile to recompile these from xml to dat isn't called at build time, so you'd need to call make clean and then make in convert/message for your message edits to apply.
- version file has info for what version of the game to build and whether to enable debug features or not
- uncommenting "NITRO_FINALROM = yes" in the main Makefile will make a ROM that can be ran on a flashcart on retail DS h/w
- fonts are in src/graphic/font.narc. With the source code there was also a leaked tool called dpk_fontconv (it's in the "imatake" directory btw), which was made to convert font for Korean localization. But you can exploit it to insert your own characters into the font or even extract the final binary font from the game into a font sheet (same in format as whatever BMPs SDK's fontcvtr outputs). For some reason though making an entire font from BMP kinda messes the game up...so I guess the source code of this utility can be studied to understand the format completely and to make a proper utility for editing these.
- convert/message/pokemon_dp.xlor has the text character map for the game...it is also in the fontcvtr format...now I wonder, why GF didn't just use NFTR fonts for their game, instead opting to make their own format, and then finally switch to NFTR in BW only to make stupid edits to the format for no good reason (and that's actually the reason NFTRedit crashes when trying to open BW/B2W2 fonts, not because it's an early version of the format it doesn't support, contrary to what I've said in another thread). There's evidence that DP fonts were at some point NFTRs, since the text files refer to fonts as font_talk.nftr or font_system.nftr...and also as a final tidbit about the fonts I want to add that the font used since BW was actually first introduced in DP but only in the original Japanese version. The localization team decided to have the old style for the fonts in their version for some reason (as can be seen in the NFTR(!) files passed between GF and NOA in "yama_work").
- seems to me like GF had a special tool to edit the text files and probably other things in the game, but those didn't leak with the source sadly. Also, as observed from the intermediate SDK files for 3D models, GF used (and still uses) Autodesk Maya for their 3D work.

 

[Pokemon_Tea_Sea_Jee]

https://gbatemp.net/threads/so-the-...pearl-just-leaked-online.570625/#post-9145458

 

[delete12345]

We in fact should make a doc. Not just on the code itself, but how to set up the build environment for it too.

Just posting to ask if there are any updates on the VM?