This is completely useless aside from feeding someone's fantasy. There are two main types of buffer overflows: Heap and Stack overflows. Stack based overflows write more data to a buffer on the stack than has been allocated. This can allow for the overwriting of a function pointer or return address which could be executed to allow the running of your own code. Heap overflows affects the heap data area. Program data can be stored here and overwritten. Heap overflows are much less common, and we probably will not see one for a long time due to their complexity.
IF the system doesn't have buffer overflow protection in it, the most likely soft-mod exploit that we would see is a string buffer overflow. These can occur if an input field pushed onto the stack is not validated. By changing the string, you could possibly jump to another area in the code. If you jump to an area you control, then you have basically hijacked the thread.
For an overflow like this to occur a LOT of things need to line up perfectly. The system first, must not have any overflow protection. If it does, even if you found an exploit, the code execution would be halted. The second is that a developer needs to accidentally on purpose forget to validate user input that is pushed onto the stack. If both of these work, then you would need to be able to point and execute your own code. Which, on the 3DS could be a project in itself (seeing as not much is known about it currently).
*Please note that's for a stack overflow using the stringbuffer.
I hope that clears some things up for people...