Is it correct that the signed file that is fakesigned is the equivalent to the Stage 1 Bootloader? Which is probably ARM9Loader or something similar? The idea behind this is that there is a chain of trust established in many devices. So the bootrom doesn't verify the home menu, an ARMLoader of some kind would.
There's only really two steps that can really be called a bootloader.
On the O3DS, the bootrom just directly loads the firmware (FIRM/NFIRM/NATIVE_FIRM) and then NFIRM runs parallel to userland software like the home menu until the console is switched off or it firmlaunches into another mode (e.g. TWL_FIRM/AGB_FIRM for backwards compatibility). In this case you essentially have a simple single stage bootloader.
The N3DS made a few changes. Nintendo wanted to block emunands, so they implemented a stronger layer of encryption around NFIRM so that it became impossible to load an NFIRM for a newer firmware version after the console is already done booting. For some reason they weren't able to change the bootrom at all to account for the new layer of encryption, so they basically had to insert arm9loader into the boot sequence in place of NFIRM. The bootrom executes arm9loader blindly, thinking it's just NFIRM. Arm9loader loads the keys for decrypting the real NFIRM from the secret NAND sector using the OTP hash, then disables the OTP permanently until hard reboot and runs NFIRM. Note that arm9loader doesn't care at all if it's running on a N3DS or O3DS, it only needs access to the NAND keys.
So here you have a two stage bootloader, with the bootrom and then arm9loader.
Process9, the component of NFIRM which runs in the ARM9 userland, verifies the signatures of all the titles that are loaded. So this is the component which verifies the home menu along with whatever else you run on the system.
So, if this is true, and our code actually replaces ARM9Loader, and not the HomeMenu, wouldn't this A: Allow a setup like BootMii, and B: Still allow for easy upgrading by using Sighax to place a signed ARM9Loader replacement that acts as "BootMii", but also as Firm Protection, allowing it to, just as A9LH does, ignore system calls to write to Native_FIRM, so you could system upgrade as normal? While also allowing (if implemented) ARM9 Binary Launches through the custom ARM9 Loader?
Both a9lh and sighax allow a setup like Bootmii. You can gain total code execution before the system fully loads. Under a9lh arm9loader is tricked into running your own code instead of NFIRM, and under sighax the bootrom is tricked into running your own code instead of NFIRM/arm9loader. The difference is pretty minimal.
a9lh (and sighax) itself doesn't implement FIRM protection. That's a feature of your CFW, which patches Kernel9 and Process9. Sighax and a9lh would both get installed to the NFIRM partition on your NAND so the same FIRM protection would protect them both.
And your a9lh or sighax installation will likely work the same way, just loading an external arm9 payload from CTRNAND or the SD card so that you don't have to keep dangerously tampering with the NFIRM partition. Although you could embed whatever code you wanted to straight into the hack. Some novel stuff like 3DSafe or ShadowNAND has done that.
And if all this is possible, couldn't you also continue to allow custom home menus, as the chain of trust is now broken, and signatures related to Home Menus could also be ignored?
You could already have custom home menus with a9lh. Heck, you could even do it with earlier hacks like menuhax if you only wanted to have a custom emunand home menu. The system simply doesn't reach the point where it verifies the Home Menu signature before it gets hacked, and signature verification becomes disabled.
But, theoretically, if you could replace the first bootloader, as this seems to do, anything is possible. If you could dump ARM9Loader, you could theoretically understand how it calls the next stage 2, and boot normally after Sighax loads, right?
We already understood how arm9loader executes NFIRM, since this was exploited to enable a9lh. We've also known how NFIRM is launched in general since the very first CFWs were released.