I'm hella late. Wouldn't it be theoretically possible to make a 3DS Capture mod using the top screen's ribbon cable then? Or even more just make a replica of the Katsukitty board? The second option apparently isn't all that special it was more so the proprietary software that did all the real work.
If you can get the signal out somewhere then you can redirect it wherever you want it to go, and ribbon cables are usually a good point to intercept things. However as you are unlikely to be able to go on digikey or whatever and buy a compatible chip for the 3ds top screen, or something you can readily get into a format an existing chip can handle, you are left with building something to handle it. Your average PIC is not going to do it so you are in CPLD (maybe) or FPGAs to handle it as making your own custom chips suitable for this is that little bit too pricey still (it can be done and we have seen it but it is usually for larger run devices), and those things that will handle it are usually fiddly and annoying (you can stick a preprogrammed PIC in something with a power lead and ground and it will do, FPGAs though need coaxing into life and a whole bunch of stuff put around them to get them to work and that gets annoying very fast*). 400 x 240 at either 30fps or 60fps if you are doing a non 3d capture... it is not outrageous to stuff over a simpler protocol as raw frames (webcams have done more for many many years now, indeed I would probably look at converting it to some kind of webcam type format rather than something more custom like some video cameras, something like HDMI, or converting it to analogue for capture that way).
*nice enough overview
Assuming the ribbon cable sends analogue/final stage signals and the screen displays those (it is possible for screens to do decoding and extras onboard) you also have to figure out what goes there. While a custom connector is pin layout is absolutely nothing to write home about then when the dust has settled there are only a few ways screens in the wild work and you can fairly easily figure out what each pin on the ribbon does if you want to risk a 3ds by desoldering pins, if you are just probing it is more annoying but still within reason and able to do some more active things like ground out or send things high and look at the results... if you are being a proper big boy hacker you could also add an intercept to remap pins or invert them but I will skip that one for now.
As mentioned before you will also have to make or source your own custom ribbon cables which is easier said than done -- no company likes doing low cost small runs but ribbon cable manufacturers are worse than most in my experience. You could possibly get it done at home with some fairly minimal tooling/setup (either etching or milling some metal backed kapton being where I head first, maybe also getting a punch and die set made up) but at that point you are introducing other skills.
None of that is out of the realm of possibility for a lone engineer or small team to make (we quite demonstrably saw it happen several times over the years) but I would put it at the level of beyond what I might expect someone to come off the street willing to learn to be able to handle, at least not with a timeframe up around 18 months. Cost wise it is not likely to be low either and then we are back where we started**, though maybe this team could open source it or something.
**see all the comments here, or indeed everywhere else this sort of thing has been discussed, about the price. Such things are always the way in gaming, and why I don't bother doing anything for it most of the time. If you have the times, tools and talent, or money to make up for a lack in any of those, then so be it but most do not.