Dosbox doesn't run well because it has no 3ds related code implemented.
Retroarch allows it to run but that's it.
There's still a lot of room for improvement.
For example, co-routines should be implemented. Or at least some form of threading.
Which also should allow us to make use of the ARMv4LE dynamic recompilation already available with dosbox.
The 3DS's ARMv6 is fully capable of running the ARMv4 instuction set if i'm correct.
It does however need a dedicated developer with the appropriate knowledge and non biased opinion about retroarch.
A direct port of dosbox would also be a possibility if one feels like it, since it's only dependency is SDL 1.2.
With the current fragmented state of the 3DS 'scene', i wouln't count on any developer being intrested enough tho.