Homebrew What is the state of DS(i) emulators in 2020 for homebrew dev's?

[Archerite]

Hopefully this is the right sub-section of the forums to ask about emulators for the DS itself. I have been working on a few homebrew projects for the DS and one of them has a few special needs that not a lot of emulators can provide. It's a remake of an old game and I do not own the rights for the graphics and I work around it by converting them at run-time on the device itself. This works great on Linux, Wii, GameCube and 3DS which have enough system RAM to work with and a way to load files from an SD card. The DS does not have either natively with only 4MB RAM and SD only from a slot-1 or slot-2 cart. The only emulator so far that supports R4 emulation in slot-1 seems to be Desmume and that is what I have been using so far to test my homebrew on. Unfortunately I am running into issues that seem to be related to the data cache on the ARM9 processor when I test my builds on hardware...and it crashes or gives different results than the emulator.

To work around the limited RAM I have modified desmume a little bit to enable it's slot-2 Memory Pak emulation by default. Normally it only enables it when the DS Browser ROM is detected. For a while to allowed me to test a lot of things on the emulator but as said...it keeps failing a lot when run on hardware. I have been reading up on the automatic data caching of the ARM9 cpu and I am suspecting it to be the cause of a lot of my issues!

My requirements mostly come down to this:
- support loading homebrew obviously
- support for R4 filesystem from a directory
- support for SLOT-2 Memory Expansion Pak (or other external RAM like SuperCard)
- support DSi emulator (mostly for the 16MB RAM)
- support for DSi SD card would be nice
- supports config override's from the command line
- debugging options would be appreciated
- opensource preferred
- has a Linux version

Searching online it seems there are only about three DS emulators that are still being worked on:
- Desmume
- no$gba
- MelonDS

If there is no existing emulator that meets my requirements I might consider modifying desmume considerably to have a better GUI and configuration options. But if something better already exists then I would be just wasting my time which is already very limited and could also be spend on working on my game. Hopefully someone can give me a few pointers on where to find good DS(i) emulators that meet my requirements (or at least most of them).

 

[Coto]

All I know is that no$gba is focused on retail compatibility than Desmume, which boots almost everything NTR related. I'm not sure of DSi.

Do note, on the DS later games (2008+) did use sound streaming because the 4MB was very little to work with. So while having more ram is nice to do more things doesn't guarantee you are taking the right approach when it comes to data processing and/or rendering.

Also, 99% the reason ARM9 code is working in emulators and not in real hardware, is either cache emulation or unhandled/missed interrupt. Here's pseudo code dealing with caches in NTR hardware:

Do note ARM7 does not need these methods because it doesn't have a cache unit (it does have the prefetcher, but that is something else irrelevant now)

Executing code: (say you want code ahead to ensure last program state is consistent, such as initializing IRQs)

1. Drain write cache (absorbs and performs older code that may be on cache, and carries these from the ARM9 into the AHB bus).
2. Invalidate Data / Instruction TCMs (all of it)
3. Run code

Data Processing: (say you want to decompress and move a BMP bitmap from EWRAM -> VRAM):

Example: https://bitbucket.org/Coto88/toolch...ommon/arm_driver_shared/utilsTGDS.c#lines-606

1. Do work on EWRAM source address
2. Clean and Invalidate Data / Instruction TCMs. (EWRAM has TCM as well, so this must apply between EWRAM source address/buffer and desired EWRAM address/buffer size)
3. Copy to target VRAM (note, can be either DMA or CPU copy)

Optionally, there may be need to save whatever contents was in VRAM before, and in this case you need to repeat these steps in VRAM source address.

 

[Coto]

Sorry for double posting, but you can use the neat :

https://gbatemp.net/download/tgds-ramdisk_dldi.36517/

which is a DLDI driver that maps a RAMDISK image to GBA Slot-2 (0x08000000 + (32*1024*1024) - 1). If you want to test stuff in Desmume through DLDI emulation. Sadly this means the ext-RAM must be discarded, but ...

 

[Archerite]

Besides the data cache I am also thinking data alignment of the buffers is a bigger issue on the DS than it seems to be on other platforms. I read somewhere that ARM cpu's do not like unaligned memory and since the ARM9 has that additional data cache I started thinking that might be it. Therefore I am also thinking that changing the malloc() calls to memalign(32) might help reduce the misalignment of the buffers.

Spoiler: Some offtopic details about why I need more RAM

I am limited by the data structure of the original files as explained before. They are using zlib compressed streams and sometimes are less than 500 bytes...the biggest one containing the level is about 1.8MB when uncompressed! The sound effects are using a 1MB sample buffer but they are actually 800-900kb in size. Already reduced it from 2MB what it was before. I have not counted the graphics separately but as they are now after building the spritesheets at runtime in four textures I have:
- 256x256 (66kb) for the HUD
- 1024x1024 (1025kb) for the tiles
- 1024x1024 (1025kb) for the level objects
- 1024x1024 (1025kb) for the player sprites and animations

For these alone I am already at nearly 6MB of RAM....ofcourse the limited screen resolution and VRAM might require me to scale the graphics to 50% in size and then it would be more like 3372kb. And hopefully using tiles instead of spritesheets gives even a little more savings on RAM.

For now I have completely disabled the sound to save that extra 1MB of RAM while I am writing the "Driver layer" for the game to work on the DS. This is working well enough that I now have it loading at least in desmume but it still crashes on hardware whenever it feels like it while loading the data. At that point there is not even 2MB in RAM!

And that sample buffer only contains the sound FX and not even the music yet. These need to either be converted from their current format into something playable or I need to port the mikmod library that can play them. But that also needs a buffer to store stuff in increasing the required RAM even more.

That's why I need external RAM on the DS with a SLOT-2 expansion module. On the DSi none of that is required as the 16MB should be enough to fit everything that I need to keep in RAM. It's just that the crashes on hardware are not very motivating when it works fine on the emulator.

Until now in my game engine I took the "quick-and-dirty" make it work fast method, and now it's time to optimize things a little more. While it works better than I thought at the start, the DS forces me to optimize things. In the end the other platforms will benefit from the more efficient code I think. Or at least the complicated mess of some parts become more maintainable.

The code you linked to is not directly usable for me since I use libnds. But the pseudo code translates perfectly into the function calls I found in lot's of projects that also use libnds: DC_FlushAll() and DS_InvalidateAll() but it's unclear to me where and when I should use or add these functions. From what I learned it should be before ANY dmacopy is started but I don't use dma at the points the code crashes! Or at least where I think it crashes. That guru mediation screen is still very cryptic to me...and sometimes it seems my code exits from main and returns to homebrew menu before I can even read any output at all!

Thanks for the pointers and I will at least try a few things of flushing the cache even more than I already do. And using memory aligned memory buffers to see if that makes a difference.

 

[Coto]

Then if you ran out of memory, then NDS is not suitable for your needs, or you are using the hardware incorrectly.

On GBA there was even less memory , 256K but devs managed to make great games on it. : D

 

[Archerite]

Then if you ran out of memory, then NDS is not suitable for your needs, or you are using the hardware incorrectly.

On GBA there was even less memory , 256K but devs managed to make great games on it. : D

My specific method of working around the license of the original games graphics requires the extra RAM to convert everything at runtime. If I would do that during compile time and embed the stuff inside the ROM I would not have most of the issues I am running into right now. But I like a challenge! It took me three months and a massive headache to figure out how the 3DS textures need to be generated without relying on tools on a PC. It was easier on the Wii/GC but I forgot how long it took to figure it out there.

The real problem is that it crashes before I even run out of memory! And only on hardware. I will admit my code is far from optimized for the DS and it needs to much RAM for it right now. I am still considering to resort to convert them once at runtime or have it done by the installer, but because of the legal stuff of the graphics license I want to prevent this.

It's not that I disagree with your comment that the NDS might not be suitable. But more that I like to find a way to make it possible. If the RAM expansion pak did not exist I would not have even considered it! Looking at the DSi with it's 16MB of main RAM I have a little more breathing room to get my game engine working on the DSi GPU....which is identical to the DS ofcourse. And when it works on them....hopefully most of the issues are solved and it's easier to make it work on the GBA too.

Just to make it perfectly clear: My end goal is learning about the different console hardware and not so much porting the game itself. That the game is ported as the end result is just a really nice bonus that others can enjoy besides myself

The final system I want to port this game engine to is the Sega Megadrive/Genesis and the SNES. Which also require a flashcart to provide the required RAM to convert the graphics into something the consoles can use. I don"t want to think about the loading times on those two yet though, hahahaha

sorry about going a bit offtopic, but I felt it was needed to explain my point.

 

[catlover007]

couldn't you also preprocess the graphics on the DS and cache it on disk?

If you're game crashes on hw it could also be that it's accessing invalid memory regions, which generates a data abort, which isn't emulated by melonDS or desmume (though idk about no$gba).

the DSi support was recently merged into master in melonDS, though it has pretty much no tools for debugging code.

EDIT: if you consider making serious contributions to any emulator, we accept PRs for melonDS, like I said, debugging is still missing and improvements would be appreciated.

 

[Archerite]

couldn't you also preprocess the graphics on the DS and cache it on disk?

If you're game crashes on hw it could also be that it's accessing invalid memory regions, which generates a data abort, which isn't emulated by melonDS or desmume (though idk about no$gba).

the DSi support was recently merged into master in melonDS, though it has pretty much no tools for debugging code.

EDIT: if you consider making serious contributions to any emulator, we accept PRs for melonDS, like I said, debugging is still missing and improvements would be appreciated.

Like I said there is a thing about the graphics license that I want to prevent "altered version to be distributed" which a cache on disk makes possible. Maybe I am a little paranoid on that little thing...but better safe than sorry. But maybe just during the development process I could at least make the game work on the DS with preprocessed graphics and not be frustrated at this early stage already.

Since all other platforms I have ported the game to are more or less the same as OpenGL in their function names and concepts I decided to do the same on the DS, and just use the videoGL functions of libnds. This in combination with aligning my memory buffers helped a lot and I get some parts of the game on screen already! It's only some debug arrows and the "mini map" that only stay on screen while the "player" is in the upper left corner....this is progress and it also works on the DSi hardware! The same build unfortunately does not work on the DS...but I might just be using to much memory with all the conversion buffers.

I have looked at melonDS and it could not run my game because I could not find any options for R4 emulation or specify a path for the DSi SD card. If one of these is supported and my game at least runs on it then I can take it into consideration. I only said to extend desmume since I have downloaded the source and it compiles on my ubuntu 18.04 PC....melonDS does not because it requires a newer CMake version.

I did however look at the sourcecode of melonDS on github and the existing issues and pull requests etc.... There is one PR that looks like it has what I need in emulating SLOT-2 SuperCard's. While I would like the SD version to be emulated testing with the CF version is perfectly fine. And for the issue with CMake I can setup a VM or docker image with ubuntu 20.04 to compile inside of that. The most important thing is that it is actively worked on I guess and it seems both emulators still have updates in the last few months on github.

I am not sure what a debugging interface should look like though, but seeing whats in VRAM and the active OAM seems obvious. And the ability to disable some backgrounds at will....or a "split" view to see what's on all layers separately seems really useful to me. No idea if that would be the case for everyone but for my game on the DS there is a need for things like this to optimize the code and graphics to fit in memory, VRAM and all the background layers

 

[catlover007]

Like I said there is a thing about the graphics license that I want to prevent "altered version to be distributed" which a cache on disk makes possible. Maybe I am a little paranoid on that little thing...but better safe than sorry. But maybe just during the development process I could at least make the game work on the DS with preprocessed graphics and not be frustrated at this early stage already.

Since all other platforms I have ported the game to are more or less the same as OpenGL in their function names and concepts I decided to do the same on the DS, and just use the videoGL functions of libnds. This in combination with aligning my memory buffers helped a lot and I get some parts of the game on screen already! It's only some debug arrows and the "mini map" that only stay on screen while the "player" is in the upper left corner....this is progress and it also works on the DSi hardware! The same build unfortunately does not work on the DS...but I might just be using to much memory with all the conversion buffers.

sure, I see, another idea: if you're careful, you could process the graphics in small chunks as well.

I have looked at melonDS and it could not run my game because I could not find any options for R4 emulation or specify a path for the DSi SD card. If one of these is supported and my game at least runs on it then I can take it into consideration. I only said to extend desmume since I have downloaded the source and it compiles on my ubuntu 18.04 PC....melonDS does not because it requires a newer CMake version.

it's all still WIP, currently redirecting SD card to the host filesystem is not support, only having one big SD blob (I know it's not ideal). Also it's not our fault, if you choose an distro which prioritises "stability" over having half way recent software releases included

I did however look at the sourcecode of melonDS on github and the existing issues and pull requests etc.... There is one PR that looks like it has what I need in emulating SLOT-2 SuperCard's. While I would like the SD version to be emulated testing with the CF version is perfectly fine. And for the issue with CMake I can setup a VM or docker image with ubuntu 20.04 to compile inside of that. The most important thing is that it is actively worked on I guess and it seems both emulators still have updates in the last few months on github.

I am not sure what a debugging interface should look like though, but seeing whats in VRAM and the active OAM seems obvious. And the ability to disable some backgrounds at will....or a "split" view to see what's on all layers separately seems really useful to me. No idea if that would be the case for everyone but for my game on the DS there is a need for things like this to optimize the code and graphics to fit in memory, VRAM and all the background layers

Looks like, you've found Asie's PR. Developement paused, so we prioritised another PR by Chagall, which adds slot 2 capabilities as well, though it supports only gba rom carts and the solar sensor iirc.

While it's true that there's still a change once in a while for desmume, developement on the core emulation mostly halted and you will probably never see DSi support implemented for it.

no$gba has great debugging support, it's a pity that it's compability is rather bad, but there's a lot of stuff to copy.

 

[Archerite]

sure, I see, another idea: if you're careful, you could process the graphics in small chunks as well.

That's an option as the graphics are already packed into "AnimationSets" but I am also restricted to the orginal data format in how much I have to load into memory at once. I can however do it in even smaller sections and when an animation set is no longer needed unload it to free the memory. This is currently only done when they are all processed. Thanks for the idea.

it's all still WIP, currently redirecting SD card to the host filesystem is not support, only having one big SD blob (I know it's not ideal). Also it's not our fault, if you choose an distro which prioritises "stability" over having half way recent software releases included

I understand why the SD card needs to be a disk image and not loose files since that mages sence for how things work on the DSi. This is actually the same method used by Dolphin for Wii SD card emulation and it's perfectly fine for me. It's quite easy to mount the image and add or remove files on it on Linux.

Hahaha, yeah I am slightly behind on updating my distro for multiple reasons. Mostly I hate reinstalling every 9 months on the other releases and upgrading never quite works well enough for me to rely on for work. I am actually using Xubuntu 18.04 but in 20.04 there was an annoying issue that made me wait a little longer.....something stupid and I totally forgot what it was. I have had it on my laptop and reverted back to 18.04 because of it. I am already happy Linux is supported at all since no$gba is not even considering it from what it looks like to me. But like I said I can spin up a docker or VM image with 20.04 that has the correct version of CMake to compile it. I'll figure that out in a few days when I get the time for it.

Looks like, you've found Asie's PR. Developement paused, so we prioritised another PR by Chagall, which adds slot 2 capabilities as well, though it supports only gba rom carts and the solar sensor iirc.

While it's true that there's still a change once in a while for desmume, developement on the core emulation mostly halted and you will probably never see DSi support implemented for it.

no$gba has great debugging support, it's a pity that it's compability is rather bad, but there's a lot of stuff to copy.

Desmume has some limited amount of DSi support as my game stops working when I set the console mode to DS Lite. I'll give melonDS a better change and look into how I need to setup the SD card image and such. I have three hardware DSi's so if anything is needed from them it's no problem to extract it.

While there might be issues with no$gba as an emulator I see it highly recommended as a debugger for GBA and DS. And the technical GBATEK documentation page is priceless!!! I have spend many many hours reading it over and over again to learn about the GBA and NDS hardware. When I started working on my game engine I mostly based my design on how the DS works with it's backgrounds actually.....and most info came from that page.

I did see it was an old PR about the Slot 2 expansion carts but maybe I'll fork it and continue the unfinished parts? I did see the memory expansion pak was missing code for the unlock register for example Like I said I will setup an environment to compile melonDS on my system somehow and checkout how the DSi SD card emulation needs to be selected or activated.

Thanks for the tips

 

[catlover007]

the DSi mode is still a bit in flux, so there's no definitive guide on how to use it, though these two blog posts should contain the necessary information:
http://melonds.kuribo64.net/comments.php?id=103
http://melonds.kuribo64.net/comments.php?id=117

the best thing would be (especially if you want to be involved in developement) if you join us on IRC (#melonds on irc.badnik.net). The Asie (author of the slot2 peripherials PR) usually can be found there too.

 

[Archerite]

the DSi mode is still a bit in flux, so there's no definitive guide on how to use it, though these two blog posts should contain the necessary information:
http://melonds.kuribo64.net/comments.php?id=103
http://melonds.kuribo64.net/comments.php?id=117

the best thing would be (especially if you want to be involved in developement) if you join us on IRC (#melonds on irc.badnik.net). The Asie (author of the slot2 peripherials PR) usually can be found there too.

I remembered I had a VM already setup with Xubuntu 20.04 ready to be used. After installing a couple of packages I could compile melonDS! Already modified it by the way, since the code that reads "sd.bin" was disabled!

Unfortunately one of the bios dump tools gives me "FAT init failed" on the DSi XL so I can't boot the DSi menu yet. I did have a NAND backup already from when I just modded my DSi a while ago and it included bios dumps (I think) but they won't boot. Bios for the DS mode is booting so at least the emulator partially works

My game won't boot at all since I really require a file system to read from

Any ideas on how to solve this?

I'll try again tomorrow to run the tool again, maybe the DLDI patching failed or something and I need to do it manually?