Apart from that, a huge THANK YOU for actually addressing and providing some insight on the technical feasibility of the concept, instead of just commenting to inform me that my thread sucks. I really appreciate the insight.
And yes, I began to wonder how exactly it could be able to register the various text transitions, placement, and dialogue changes, and how long of a delay we would be looking at before the translation overlay could be thrown up on screen. And if the text auto-scrolls or something for a particular game, probably all sorts of problems. And yeah, the 3DS being touchscreen-based for menu navigation and whatnot, makes sense there isn't any kind of accommodations in place for overlays.
I was sort of piecing together that this would likely be an unfeasible technical nightmare, and the few bits of technical insight I'm seeing is pretty much confirming it.
Actually, now I managed to re-read both messages without missing anything, and it seems like I got the gist of it, but missed some of the fine details, so I'll try to address those as well.
First of all, the big one: you will sadly not be having this in Luma anytime soon. It's already struggling a bit with code size, so there is a policy in place where only the absolute "critical" and most useful (mostly hardware-related) utilities are allowed to be in there, where it makes actual sense to have it in an easily-accessible overlay, instead of opening an installed .cia/.3dsx and poking things there. So basically something you'd want to change about the hardware mid-game.
The 3DS has a GPU, which can only output to a rendertarget, meaning there is *always* some sort of memory which holds the contents of a rendered scene; unlike fixed-function hardware, where there are "instructions"/"settings" in VRAM and other hardware registers to reconstruct an image *directly* to the LCD, every frame, like how the Game Boy and DS do.
Well, the DS actually has a 48-scanline (or 96-scanline? I forgot) internal buffer for the 3D engine, but the 2D engine only gets up to 2 scanlines worth of internal buffering in certain cases. The Game Boy itself even pauses the pixel clock just so it has time to fetch data from VRAM.
Anyway, I went on a tangent.
None above matters that much, but what does matter is that games also un-mangle the rendertarget to the framebuffer (using dedicated GPU hardware with DMA), which *does* get displayed on the LCDs, and that can be used for our purposes.
There are various ways to access this framebuffer from "My code" (using Visual Studio terminology, because I can't word it any better now), but there is no clean method in userland. You will eat the cost one way or another, usually involving a balance of technical debt and use of limited hardware resources.
My homebrew uses the DMA method, so it works on old3DS.
Well, the DS actually has a 48-scanline (or 96-scanline? I forgot) internal buffer for the 3D engine, but the 2D engine only gets up to 2 scanlines worth of internal buffering in certain cases. The Game Boy itself even pauses the pixel clock just so it has time to fetch data from VRAM.
Anyway, I went on a tangent.
None above matters that much, but what does matter is that games also un-mangle the rendertarget to the framebuffer (using dedicated GPU hardware with DMA), which *does* get displayed on the LCDs, and that can be used for our purposes.
There are various ways to access this framebuffer from "My code" (using Visual Studio terminology, because I can't word it any better now), but there is no clean method in userland. You will eat the cost one way or another, usually involving a balance of technical debt and use of limited hardware resources.
- NTR goes full-out on RAM use and technical debt. It just copies itself into every relevant process, and accesses things directly. Very bad, but sadly this is pretty much the fastest method within the constraints of the 3DS OS. But old3DS be damned due to not enough RAM.
- You can also DMA it into your own process (and this is what my homebrew does). This is kind of bad, as it wastes the already extremely limtied memory bandwidth of the system, and it also trashes the cache due to having hardware-overwritten local memory, you *need* to invalidate the cache, otherwise you will read cache remnants instead of the actual memory contents. Not only that, it also steals cache from the game, meaning everything will run slower as a result. This is especially noticable on old3DS, where the OS will appear to hang due to running way too slow due to cache misses and slow memory and huge IPC latency combined.
- You can use Luma's memory mirror. The problem is that this is not just uncached, but *strongly ordered*, which in very short means that it can be even slower than uncached memory (due to (re-)ordering of memory accesses being blocked), which in itself can cause other weird issues, and also slow down the system, possibly even more than just uncached, if the conditions are just unfortunate. At least it won't trash the cache. But sadly this only works with Luma, it doesn't work in any other CFW, and obviously not in an "unmodified" 3DS (with minimal set of patches allowing unsigned code execution).
My homebrew uses the DMA method, so it works on old3DS.
Looking back at my previous reply, feels like some AI wrote the technical parts (well, I was basically asleep due to the heart problems, no wonder!), so let me clean it up, while elaborating further on the challenges.
The software parts (with regards to translator APIs and stuff) still stand though.
Sadly the more I think about it, the less your original proposal sounds feasible.
On old3DS, there is not enough RAM (assuming we can't use game memory) to hold *one* extra framebuffer (I know that because I've made old3DS-compatible video streaming *from* 3DS to PC), so the idea of local translate is pretty much dead right there. Your only option is to stream the video feed to a PC or phone, and use that to feed into an image translator. And sending back the frame... again, not enough memory to even hold the framebuffer to be *sent*, so local old3DS image translate is pretty much dead idea. You'd have to read the translated image on your phone/PC, and there is sadly seemingly no way to work around this in a way that works with every game.
However, on new3DS the situation looks a lot more feasible. Considering that the only game that uses the highest memory mode on new3DS is not even a game (CTRAging), it's safe to assume that every game (that doesn't mess with WiFi, unlike Ocarina of Time or Super Mario 3D Land for example) will work with new3DS, and "My code" will have plenty of RAM at all times to hold multiple framebuffers.
Not only that, the L2 cache in the new3DS carries basically all of the performance. Even with the CPU set to old3DS speeds, the speed of the 3DS is only marginally reduced. But with L2 off, and CPU at new3DS speed, somehow the system becomes SLOWER than an old3DS

Anyway. Video streaming is also faster on new3DS, and has less impact on the system. In fact, it could possibly even handle video decoding at the same time, at some okay framerate. Only "okay" when streaming simultaneously both from and to new3DS, but when only receiving, it can do 60FPS just fine.
So if we went the video streaming route, we could pretty much stop here.
Assuming text processing is done outside of the 3DS, you could possibly speed up the framerate and reduce the latency by doing the text overlay 3DS-side (either by pre-processing the text PC-side if the API gives back text with positions, or extract the new text from the image if a completely new image is sent by the API, and just keep the overlay on for a few frames every few frames).
However, if you want a fully localized (no pun intended) translation suite, problems still arise.
Don't remember the exact numbers, but I think you have around 64MB of RAM to use for "My code" on new3DS, if running alongside the game. I'm not sure what local OCR and translation code could fit into that much RAM use, but I don't think there is more than you have fingers.
Oh, and if whatever algorithm you use needs a lot of file IO (to index into some huge database stored on the SD or in the .cia), you're just screwed. Even just doing one file IO every frame can slow down a few games significantly. Low framerate is acceptable for some peoople, but *unstable* framerate is unacceptable for everyone I know, and made everyone sick, so that is definitely not an option.
May be still possible, but don't expect miracles, or if such software even exists at all that meets all of these requirements.
A bonus mention: if you don't have internet, but you can localize this entire translation fandango operation on your PC and/or phone (idk, huge image database, some "AI" running in Docker, idk what local options exist for this), you can run a HTTP proxy on there, and in the 3DS WiFi settings set the proxy settings to point to your phone and/or PC. This way you can play on the go, assuming your phone and/or laptop can hold enough charge, and/or have access to a power outlet.
tl;dr:
Don't expect old3DS to work to be able to do this at all due to not enough RAM. Your *only* option is streaming 3DS video to a PC and/or phone, and view the phone/PC screen there with the translated content. Slightly more comfortable than holding a phone to the 3DS, but not by much.
On new3DS, there are better options. None great, but much more possible. Although fully local translate might be unfeasible. Someone dedicated may make it work, but chances are extremely slim. The video streaming method however is really much feasible, just janky, and suffers from low framerate and high latency.
Last edited by Sono,