Is it possible...?

[Sono]

Apart from that, a huge THANK YOU for actually addressing and providing some insight on the technical feasibility of the concept, instead of just commenting to inform me that my thread sucks. I really appreciate the insight.

And yes, I began to wonder how exactly it could be able to register the various text transitions, placement, and dialogue changes, and how long of a delay we would be looking at before the translation overlay could be thrown up on screen. And if the text auto-scrolls or something for a particular game, probably all sorts of problems. And yeah, the 3DS being touchscreen-based for menu navigation and whatnot, makes sense there isn't any kind of accommodations in place for overlays.

I was sort of piecing together that this would likely be an unfeasible technical nightmare, and the few bits of technical insight I'm seeing is pretty much confirming it.

Actually, now I managed to re-read both messages without missing anything, and it seems like I got the gist of it, but missed some of the fine details, so I'll try to address those as well.

First of all, the big one: you will sadly not be having this in Luma anytime soon. It's already struggling a bit with code size, so there is a policy in place where only the absolute "critical" and most useful (mostly hardware-related) utilities are allowed to be in there, where it makes actual sense to have it in an easily-accessible overlay, instead of opening an installed .cia/.3dsx and poking things there. So basically something you'd want to change about the hardware mid-game.

Spoiler: Technical tangent about the 3DS GPU and frame capture and other unrelated stuff

The 3DS has a GPU, which can only output to a rendertarget, meaning there is *always* some sort of memory which holds the contents of a rendered scene; unlike fixed-function hardware, where there are "instructions"/"settings" in VRAM and other hardware registers to reconstruct an image *directly* to the LCD, every frame, like how the Game Boy and DS do.
Well, the DS actually has a 48-scanline (or 96-scanline? I forgot) internal buffer for the 3D engine, but the 2D engine only gets up to 2 scanlines worth of internal buffering in certain cases. The Game Boy itself even pauses the pixel clock just so it has time to fetch data from VRAM.
Anyway, I went on a tangent.

None above matters that much, but what does matter is that games also un-mangle the rendertarget to the framebuffer (using dedicated GPU hardware with DMA), which *does* get displayed on the LCDs, and that can be used for our purposes.

There are various ways to access this framebuffer from "My code" (using Visual Studio terminology, because I can't word it any better now), but there is no clean method in userland. You will eat the cost one way or another, usually involving a balance of technical debt and use of limited hardware resources.

• NTR goes full-out on RAM use and technical debt. It just copies itself into every relevant process, and accesses things directly. Very bad, but sadly this is pretty much the fastest method within the constraints of the 3DS OS. But old3DS be damned due to not enough RAM.
• You can also DMA it into your own process (and this is what my homebrew does). This is kind of bad, as it wastes the already extremely limtied memory bandwidth of the system, and it also trashes the cache due to having hardware-overwritten local memory, you *need* to invalidate the cache, otherwise you will read cache remnants instead of the actual memory contents. Not only that, it also steals cache from the game, meaning everything will run slower as a result. This is especially noticable on old3DS, where the OS will appear to hang due to running way too slow due to cache misses and slow memory and huge IPC latency combined.
• You can use Luma's memory mirror. The problem is that this is not just uncached, but *strongly ordered*, which in very short means that it can be even slower than uncached memory (due to (re-)ordering of memory accesses being blocked), which in itself can cause other weird issues, and also slow down the system, possibly even more than just uncached, if the conditions are just unfortunate. At least it won't trash the cache. But sadly this only works with Luma, it doesn't work in any other CFW, and obviously not in an "unmodified" 3DS (with minimal set of patches allowing unsigned code execution).

Basically, if you want any speed, you can use the NTR method on a new3DS only.
My homebrew uses the DMA method, so it works on old3DS.

tl;dr: the 3DS has a GPU, which basically™ means that there is always a framebuffer to display on the LCDs, and we can use that, with magic™, to access the displayed contents in one way or another.


Looking back at my previous reply, feels like some AI wrote the technical parts (well, I was basically asleep due to the heart problems, no wonder!), so let me clean it up, while elaborating further on the challenges.
The software parts (with regards to translator APIs and stuff) still stand though.

Sadly the more I think about it, the less your original proposal sounds feasible.
On old3DS, there is not enough RAM (assuming we can't use game memory) to hold *one* extra framebuffer (I know that because I've made old3DS-compatible video streaming *from* 3DS to PC), so the idea of local translate is pretty much dead right there. Your only option is to stream the video feed to a PC or phone, and use that to feed into an image translator. And sending back the frame... again, not enough memory to even hold the framebuffer to be *sent*, so local old3DS image translate is pretty much dead idea. You'd have to read the translated image on your phone/PC, and there is sadly seemingly no way to work around this in a way that works with every game.

However, on new3DS the situation looks a lot more feasible. Considering that the only game that uses the highest memory mode on new3DS is not even a game (CTRAging), it's safe to assume that every game (that doesn't mess with WiFi, unlike Ocarina of Time or Super Mario 3D Land for example) will work with new3DS, and "My code" will have plenty of RAM at all times to hold multiple framebuffers.
Not only that, the L2 cache in the new3DS carries basically all of the performance. Even with the CPU set to old3DS speeds, the speed of the 3DS is only marginally reduced. But with L2 off, and CPU at new3DS speed, somehow the system becomes SLOWER than an old3DS

Anyway. Video streaming is also faster on new3DS, and has less impact on the system. In fact, it could possibly even handle video decoding at the same time, at some okay framerate. Only "okay" when streaming simultaneously both from and to new3DS, but when only receiving, it can do 60FPS just fine.
So if we went the video streaming route, we could pretty much stop here.
Assuming text processing is done outside of the 3DS, you could possibly speed up the framerate and reduce the latency by doing the text overlay 3DS-side (either by pre-processing the text PC-side if the API gives back text with positions, or extract the new text from the image if a completely new image is sent by the API, and just keep the overlay on for a few frames every few frames).

However, if you want a fully localized (no pun intended) translation suite, problems still arise.
Don't remember the exact numbers, but I think you have around 64MB of RAM to use for "My code" on new3DS, if running alongside the game. I'm not sure what local OCR and translation code could fit into that much RAM use, but I don't think there is more than you have fingers.
Oh, and if whatever algorithm you use needs a lot of file IO (to index into some huge database stored on the SD or in the .cia), you're just screwed. Even just doing one file IO every frame can slow down a few games significantly. Low framerate is acceptable for some peoople, but *unstable* framerate is unacceptable for everyone I know, and made everyone sick, so that is definitely not an option.
May be still possible, but don't expect miracles, or if such software even exists at all that meets all of these requirements.

A bonus mention: if you don't have internet, but you can localize this entire translation fandango operation on your PC and/or phone (idk, huge image database, some "AI" running in Docker, idk what local options exist for this), you can run a HTTP proxy on there, and in the 3DS WiFi settings set the proxy settings to point to your phone and/or PC. This way you can play on the go, assuming your phone and/or laptop can hold enough charge, and/or have access to a power outlet.

---

tl;dr:
Don't expect old3DS to work to be able to do this at all due to not enough RAM. Your *only* option is streaming 3DS video to a PC and/or phone, and view the phone/PC screen there with the translated content. Slightly more comfortable than holding a phone to the 3DS, but not by much.
On new3DS, there are better options. None great, but much more possible. Although fully local translate might be unfeasible. Someone dedicated may make it work, but chances are extremely slim. The video streaming method however is really much feasible, just janky, and suffers from low framerate and high latency.

 

[Kwyjor]

Respectfully, what is "possible" on Retroarch with vastly more powerful PC hardware is completely immaterial to what is "possible" or technically feasible on 3DS

Quite the contrary, no one is going to somehow develop something for the 3DS that is somehow more capable than what is currently available via Retroarch. If the current state of Retroarch's auto-translate function is not acceptable to you then no one is likely to start improving it by writing something for a dead platform like the 3DS.

 

[The Real Jdbye]

Hey, guys. Moron here, equipped with an ill-conceived concept of a theoretical 3DS application, and absolutely no practical or technical knowledge whatsoever.

So. Here I am, playing the Japan-only Legend of Xanadu via TemperPCE on my favorite electronic device in the entire world -- my trusty, reliable, fun-dispensing NN3DS -- thanks in no small part to all the actual intelligent, knowledgeable, and talented developers among this community that continually innovate and trick it out to its full potential.

Just trying to reach the first of the beautiful-looking side-scrolling sections I've seen screenshots of to experience it for myself. No fan translation for this one, though, and it's a bit of a text-heavy JRPG. So, tad bit of a language barrier going on. Thanks to the wonders of modern technology, however, I have a free phone application that uses the camera to translate Japanese text into English in real-time. Obviously, the results aren't necessarily the quality and thoughtful kind of translations that actual, dedicated fan localizations with a human touch tend to provide, but even when the English is occasionally utterly broken, it provides enough of a basis to decipher and at least piece together the general idea. Constantly hovering my phone screen over my 3DS screen can get a little tedious, but it's still amazing in and of itself that I can even do this at all. I'm thinking of maybe trying to do a run of SegaGaga with this method now. I don't see an actual translation for that one ever seeing the light of day with all the apparent technical hurdles associated with its English text-size that its various developers have faced during their numerous failed passes at it, and at this point, it's better than nothing.



Aaaand, that's when I got to wondering to myself, in my moron-mush brain, if it would be technically possible at all to create a similar sort of application natively for the 3DS. I have no idea how these phone apps actually work, of course. Just seems that it essentially interprets the text on-screen, matches it with its native-language dictionary, correlates it with the appropriate equivalent in the alternate language dictionary that's to be output, and then simply outputs the swapped text to the user. These dictionaries would probably have to be built-in to this 3DS app somehow, and I would surmise it would have to work as some sort of Luma/Rosalina application acting as something of an overlay that you can use independently and in conjunction with any other app of your choosing... that is, if Luma can actually read the on-screen text in real time, if at all. No actual clue there (or anywhere). I'm just suspecting that maybe that's technically possible somehow, simply due to the fact that you can take screen shots through Luma? I know that's pretty broad and general, though. Just because it can screen cap a native image doesn't mean it is somehow capable of actually reading or interpreting any visual information on the screen.

Reading specific visual information on-screen, natively, in real-time, is likely a ridiculously fantastical pipe-dream of a concept on my part, now that I take into account all the potential variables in text size, pixel size, resolution, fonts, etc., etc., across wildly varying types of software. This is all just occurring to me now, and yeah... this theoretical native-screen-reading app I've dreamt up in my head is probably just not how this type of shit works or reads information, and any specific software would probably need to be individually programmed for compatibility to be effectively read via the game's code itself, rather than its on-screen visual output in real-time... which then pretty much renders my entire concept moot, I suppose. These cell phone translator apps use a camera and visual information from an actual lens, adjusting the text size accordingly with draw distance and whatnot. So, you probably need an actual camera lens to pull-off reading real-time on-screen text. The 3DS has a camera of its own as well, of course. So maybe this theoretical application would actually be technically achievable through its camera in the same way the actual smart phone applications work. But who really needs that when we all already have our phones for that?

My pipe-dream is a Luma application powered by voodoo magic that can somehow detect, read, and translate native on-screen text information, and overlay the translated text box over the original foreign text in real-time as you're playing, just as the camera-operated smart phone apps do, and that you can utilize universally across pretty much anything. I'll go look for a genie lamp and get back to you guys.


Well, anyway. If there's anyone here who has any idea what they're actually talking about regarding the technical logistics of any of my nonsense here, and would care to indulge me in any genuine development insight or technical break downs to properly put my toddler's-understanding and cave-drawn development roadmap for this idea in clearer context, I would be very interested and appreciative of that. Or any other input, really.


Thanks, and enjoy your day.

Sure, Google has a translate API that anyone can use.
The catch is, that API isn't free. It's not very expensive, but translating a whole game it would probably add up.
There are some other options that are free, like Yandex, but I'm unsure if they support OCR. Doing OCR directly on the 3DS and only sending the text to the API would reduce wifi usage a lot and work with more translation APIs, but doing OCR on the 3DS might be too slow, it would be no good if the game needs to pause for several seconds (or more) every time it needs to process an image into text. And the OCR won't be nearly as good as Google's (which is best in class). Also, the free APIs probably have heavy usage limits.
Google does offer a free trial of their APIs which allows a lot of use but only for 2 weeks.

 

[Kwyjor]

So, Retroarch has an auto-translator on PC. Good for Retroarch on PC. It's immaterial.

Have you even bothered to take a minute to look it up and see what it looks like – something that people have actually done, with existing tools, for free, out of their own interest, thereby demonstrating what is actually possible – or are you content to only speculate about how things ought to work in the ideal world?

 

[impeeza]

Hey, guys. Moron here, equipped with an ill-conceived concept of a theoretical 3DS application, and absolutely no practical or technical knowledge whatsoever.

So. Here I am, playing the Japan-only Legend of Xanadu via TemperPCE on my favorite electronic device in the entire world -- my trusty, reliable, fun-dispensing NN3DS -- thanks in no small part to all the actual intelligent, knowledgeable, and talented developers among this community that continually innovate and trick it out to its full potential.

Just trying to reach the first of the beautiful-looking side-scrolling sections I've seen screenshots of to experience it for myself. No fan translation for this one, though, and it's a bit of a text-heavy JRPG. So, tad bit of a language barrier going on. Thanks to the wonders of modern technology, however, I have a free phone application that uses the camera to translate Japanese text into English in real-time. Obviously, the results aren't necessarily the quality and thoughtful kind of translations that actual, dedicated fan localizations with a human touch tend to provide, but even when the English is occasionally utterly broken, it provides enough of a basis to decipher and at least piece together the general idea. Constantly hovering my phone screen over my 3DS screen can get a little tedious, but it's still amazing in and of itself that I can even do this at all. I'm thinking of maybe trying to do a run of SegaGaga with this method now. I don't see an actual translation for that one ever seeing the light of day with all the apparent technical hurdles associated with its English text-size that its various developers have faced during their numerous failed passes at it, and at this point, it's better than nothing.



Aaaand, that's when I got to wondering to myself, in my moron-mush brain, if it would be technically possible at all to create a similar sort of application natively for the 3DS. I have no idea how these phone apps actually work, of course. Just seems that it essentially interprets the text on-screen, matches it with its native-language dictionary, correlates it with the appropriate equivalent in the alternate language dictionary that's to be output, and then simply outputs the swapped text to the user. These dictionaries would probably have to be built-in to this 3DS app somehow, and I would surmise it would have to work as some sort of Luma/Rosalina application acting as something of an overlay that you can use independently and in conjunction with any other app of your choosing... that is, if Luma can actually read the on-screen text in real time, if at all. No actual clue there (or anywhere). I'm just suspecting that maybe that's technically possible somehow, simply due to the fact that you can take screen shots through Luma? I know that's pretty broad and general, though. Just because it can screen cap a native image doesn't mean it is somehow capable of actually reading or interpreting any visual information on the screen.

Reading specific visual information on-screen, natively, in real-time, is likely a ridiculously fantastical pipe-dream of a concept on my part, now that I take into account all the potential variables in text size, pixel size, resolution, fonts, etc., etc., across wildly varying types of software. This is all just occurring to me now, and yeah... this theoretical native-screen-reading app I've dreamt up in my head is probably just not how this type of shit works or reads information, and any specific software would probably need to be individually programmed for compatibility to be effectively read via the game's code itself, rather than its on-screen visual output in real-time... which then pretty much renders my entire concept moot, I suppose. These cell phone translator apps use a camera and visual information from an actual lens, adjusting the text size accordingly with draw distance and whatnot. So, you probably need an actual camera lens to pull-off reading real-time on-screen text. The 3DS has a camera of its own as well, of course. So maybe this theoretical application would actually be technically achievable through its camera in the same way the actual smart phone applications work. But who really needs that when we all already have our phones for that?

My pipe-dream is a Luma application powered by voodoo magic that can somehow detect, read, and translate native on-screen text information, and overlay the translated text box over the original foreign text in real-time as you're playing, just as the camera-operated smart phone apps do, and that you can utilize universally across pretty much anything. I'll go look for a genie lamp and get back to you guys.


Well, anyway. If there's anyone here who has any idea what they're actually talking about regarding the technical logistics of any of my nonsense here, and would care to indulge me in any genuine development insight or technical break downs to properly put my toddler's-understanding and cave-drawn development roadmap for this idea in clearer context, I would be very interested and appreciative of that. Or any other input, really.


Thanks, and enjoy your day.

ANYTHING is possible if you spend enough time and money.

 

[flanter21]

There is this 3gx plugin which gives you a japanese dictionary so you don't have to switch back and forth. But for your initial question, maybe image translation isn't a good idea, but a plugin might be able to find the text in RAM, translate it and then read it out into an overlay (or just modify the RAM). That probably wouldn't work for emulators though.