Remember that Retroarch is a reference implementation of the Libretro API. It's designed to do everything, and its feature set is large and growing. It's impressive, even, that such a large feature set is so portable, but let's not forget the breadth of the project in general.
The best solution for 3ds, albeit less flexible and featureful, would be to have something like lakka but for 3ds, that stays stable for 3ds, and uses cores optimized specifically for 3ds. This would of course fragment things, and without dedicated maintenance would be less reward than the efforts of simply fixing current retroarch for 3ds.
So don't read this and think, "wow, I should just fork the oct 20 builds, strip them down, and enhance them!". If you truly have the capability to do that, you should instead spend your time looking into how retroarch for 3ds can be fixed in its current form, unless you're VERY dedicated and willing to backport from main retroarch for the long haul.
I'm just pointing out that we really shouldn't expect 3ds to be a first-class citizen, and because of retroarch's nature, we can expect it to break.