Firmware modded 3ds is a given really but beyond that.
Do you know any programming already? Most applications are built to take advantage of other libraries or features of the OS, porting then involves recreating whatever is necessary or adapting things, as well as working to hardware constraints.
There a handful of other means of doing things but https://devkitpro.org/ is probably what most would suggest as a basic C family compiler and series of libraries (and examples of their use).
If you have to learn to program on top of this (and you will need some fair programming skills -- the nice simple languages seen on the PC are not really a thing on the 3ds, and porting is not something you can sort of muddle through by changing a few variables or maths routines like we see with people doing some small mods for open source/source available games). https://gbatemp.net/threads/so-you-want-to-learn-to-program.371255/ might be a start there, though again you will want to be heading down the C/C++ and hardware knowledge path rather than the more simplistic stuff. This can take a while as well.