Notice
You may know that I do NOT intend to promote piracy. I believe PSP hacking scene was so successful because less people regarded piracy as their goal.

What is ARM9
3DS has two CPU and they have different architectures. I'll call them with the name of their architecture: ARM9 and ARM11.
The main reason why it has two CPU is compatibility; ARM9 is compatible with DS while ARM11 is used to improve the performance. However, the design also improves the security by letting ARM9 play the role related to the security.

Vulnerability on ARM9
ARM9 has all permissions on 3DS and it can do most things. For example, ARM9 can make ARM11 execute arbitrary code. It means ARM9 is always the target for hacking.

Practice
ARM9 has several interfaces and you'll hack it with them. The most largest part is PXI.

PXI
PXI is the interface for Process9, a process running on ARM9. Process9 has services which executes commands sent by ARM11. 3dbrew.org has the list for those services.
http://3dbrew.org/wiki/PXI_Services/

The protocol is also explained on the site.
http://3dbrew.org/wiki/PXI_Registers/

The site also has descriptions for commands, but they are incomplete.
http://www.3dbrew.org/wiki/Services_API/

As you can see, it doesn't have so many commands. That's the advantage of the design. You must hack those commands, which are the most secure part.
But it doesn't mean 3DS can't be hacked. It depends on who does.

Debugging
Unfortunately, we don't have a debugger working "well". The largest reason is that few people tried to do that. But I don't recommend you to make a new debugger. As I said, the system is divided into ARM9 and ARM11 and ARM9 plays only the security-related role. It's nearly impossible to communicate with PC via Wi-Fi or IR.
However you don't have to abandon. I developed an exception handler for ARM9 on rxTools which displays exceptions on the display. I recommend to use this.

rxTools
It hooks SWI on ARM11 because it needs ARM11 to handle LCD. So something must call SVC on ARM11. Anyway, "something" should always call SVC even if you don't.

When an exception occurred, the handler sets the address of the framebuffer to the top of the VRAM (0x18000000) and it writes exception information to VRAM. You can change the color of the background by clearing VRAM before an exception occurred because the rest is left as it is. Otherwise it shows garbage on VRAM.


You can see what is the "garbage". In the next, I'll explain the information described on the display.

Type
The document by ARM is good for this.
http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.ihi0014q/I84795.html

If you have "Data Abort" with a writing instruction, it may be exploitable.
If you have "Prefetch Abort", it may be exploitable.
If you have "Data Abort" with a reading instruction, disassemble and study well. If you can't figure out, try to input some valid value, or ask someone skillful.

Registers
ARM provides documentation.
http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.ddi0337h/Chdedegj.html

If you got LR control, you are likely to exploit it. The final goal is to control PC (and execute your own code) .