HMAC = Hash-based message authentication code. Basically, you use a standard hash function, but add in a secret key. Nintendo DSi uses SHA1-HMAC for banners, which is why DSi-compatible flash carts use banners from licensed games instead of their own games.
RSA signature = using RSA public key cryptography to create a signature that validates the contents of some file. Usually this is done by taking a hash of the contents and then encrypting the hash using the RSA private key. The signature can be verified by decrypting the hash with the public key and comparing that hash with the actual hash. RSA signatures are used by the DSi and 3DS for virtually everything.
As far as I know, there are no general-purpose cracks for RSA signatures other than brute-force. However, that's assuming the implementation is working. The 3DS Boot ROM signature hax was only possible because the Boot ROM's RSA verification function is broken. (The main FIRM RSA verification is correct, though.)
