Wall of text incoming. I don't know absolutely every step, but this is how I understand it so far, beware of some technical jargon. I'm not 100% sure on every detail here, so someone like
@Mrrraou or
@Selver can help fill in the gaps I missed; this is just how I understand it, as I said.
To explain what it is and how it works, you have to get into how the entire New 3DS boots, because ARM9LoaderHax hijacks how the 3DS (both old and new) boots from the moment you turn the power button on. When you hit the power button, a lot of things are going on in order to take you to the home menu; it's not just hit the button and the game starts like the old DS or Gameboy. The 3DS has ROMs (read only memories) that get executed after the power turns on, one for each CPU (ARM11, and ARM9). After those execute and initialise a few basic things, the ARM9 CPU gets the NAND (sysNAND) ready for reading, and eventually NATIVE_FIRM (which is basically the kernel of the 3DS operating system, as well as a few other things) will be brought into memory from FIRM0, which is a place in the NAND that it's stored (also take note that the New 3DS has a "secret sector", which is how it decrypts FIRM on 9.6+, a key is stored there). If it fails to load for any reason, the ARM9 will try to load NATIVE_FIRM from FIRM1, which is a different area, and is there as a fail-safe, to prevent bricks (but also note that FIRM1 is smaller than FIRM0, and this is important). Assuming that the system was able to load one of the two NATIVE_FIRMs, it then lets the ARM9Loader take over, which decrypts some things, and then brings up the ARM11 so that the 3DS OS can truly start. This is the normal boot process.
Now that that's established, here's ARM9LoaderHax. The process begins the same, but instead of executing as normal, garbage was installed into the secret sector in order to make Kernel9 (this is the code that usually runs on the ARM9, if I'm not mistaken) decrypt to garbage. This garbage is specially crafted so it jumps to our code just after the bootROMs finish executing. Since the bootROM can't decrypt FIRM0, it panics and then goes to FIRM1 to see if it can salvage the boot. FIRM1 is left intact on purpose, so that it can be loaded into the place where FIRM0 would've been. But since it's smaller, not everything is overwritten from where FIRM0 was (and FIRM0 has some of our code written into the end of it). Since the secret sector is decrypting things to garbage, FIRM1 also fails, and it ends up taking us right to the end of where FIRM0 was (which is where ARM9LoaderHax begins). A9LH will then look for a payload on the SD card, and then allow that payload to do whatever it wants (typically, boot a CFW, or run a recovery mode like Decrypt9).