Mods can be many things and require many different skills depending upon what you want to do.
Depending upon the console (if it is DS or newer or came on floppy disc or optical media) then the game can probably be broken up into its constituent files. The 3ds certainly can do this.
From where again it depends what you want to edit.
I tend to split things up into graphics, text, audio, levels and game logic (this including assembly programming where most of the rest is more abstract) but any of those can bleed into other areas (edit text and if you want new characters then probably some graphics, likewise text can inform game logic if it is a scripting based game) and there can be a nice difficulty spike depending upon the game and what you are doing or what the original devs did.
If things are file based (as opposed to being all packed into one file like older systems, and emulated versions thereof) then various developers (including Nintendo which provides example formats for other devs to use) will often reuse certain formats. This can allow you to do some things, recognise what you are dealing with (to either look further or eliminate -- want to edit text and there is a nice big sound file in this folder and you can ignore it) and possibly have some premade tools that will handle it.
I have a guide to GBA and DS but with the more generalised approach
https://gbatemp.net/threads/gbatemp-rom-hacking-documentation-project-new-2016-edition-out.73394/
http://www.romhacking.net/start/ also works for many.
http://gbatemp.net/topic/291274-the-ultimate-nintendo-ds-rom-hacking-guide/ has another approach.
I don't know what we are suggesting for people to explore 3ds ROM innards these days.
https://gbatemp.net/threads/release-every-file-explorer.373615/ has some stuff though for when you do.
https://github.com/dnasdw/3dstool is also what a lot of people will use. Don't know what the situation is with keys these days (3ds ROMs are encrypted but as we have total control over such things then keys are known, though some laws somewhere may say "don't share keys" so they are not always included).
https://gbatemp.net/threads/tutorial-how-to-decrypt-extract-rebuild-3ds-cia-cxi-files.383055/ also is reasonably up to date and will do stuff.
