I'm trying to learn more about hardware hacking and vulnerabilities and I've always wondered, how did people observe the code that the 3DS runs when it boots before they found an exploit?
If you mean the bootroms, there's no alternative to dumping them (which was done over a year ago by certain famous people)
If you mean the secret sector (required for decrypting N3DS kernels), the contents were effectively leaked by security deficiencies in, iirc, the kernels themselves
All installable titles are encrypted with keys coming from variable combinations of: the kernel, the bootrom, the OTP, the movable.sed, the nand CID, etc -- but while, without all of those sources, it's impossible to get the actual keys themselves, you can use a hacked console (that can normally decrypt your object of interest, for example you legally bought a digital game with) to decrypt that object without necessarily having direct access to the keys involved (for example, dumping titlekeys)
The above is how pretty much any encryption/decryption had to be done until most people got access to the bootrom = B9S release, since most of the keys come in part from the bootrom
While still comparatively unpopular, it's now possible to do most of those jobs on a PC, at an appropriately increased speed!
The usual way to realize an exploit even exists is to blindly feed an application invalid data (hacked saves, too long of a description in the case of the mset exploit, non-level QRs for Cubic Ninja...) and see how it reacts (ignores it, rejects it, crashes and quits, crashes and freezes or does weird stuff) - then you need to figure out if the crash is exploitable (ie if it can be controlled by changing the data you feed it and/or "external" factors)In order to develop an exploit you need to observe and study the code that is being run, right?