Game modding/ROM hacking and exploit making are two quite different fields in the way people approach them, you take either to a high level and you could do the other (or at least follow along with what the other is doing) but they are still quite different.
ROM hacking then.
First question. Is the game you are hacking using a file system? Most older consoles do not, indeed the usual line is the DS for handhelds and things which use discs for home consoles.
If it does use a file system you have file names, directory names, extensions/magic stamps, file sizes and more besides to help narrow things down and figure out what is what and what you want to edit.
If it is all packed in one binary without a file system then all is not lost. Some arcade games might not have a file system but they might have something vaguely similar with split boards, not to mention if you are hacking an arcade game these days then go look at MAME/MESS source code as it will usually have someone that looked at it (most arcade games have 10 or so games for the board in question so it is reasonable to look at every one of them, compared to often thousands of games on a home console or handheld).
All is not lost if you have it all in a binary though as it does make certain things a bit easier. Most notably here would be tracing where you find something in a game and work back from there to figure out what put it there/changes it/uses it, this is a fair bit easier if you don't have a file system parsing and CD read command in the way.
Tracing is the advanced (ish, you could probably learn basic tracing in a couple of hours) technique though and there are many others like relative search (finds European languages text quite well), simple looking (get a tile editor to view graphics and press page down a lot), corruption (corrupt part of the ROM and see what changed/broke), compression searching (compression often has telltale signatures), backwards searching (find a colour palette in the game and search for that in the ROM) and the list goes on for a while. Guides to ROM hacking will cover this, I have one in my signature.
Device hacking. It is far harder today than it was back when and I don't envy those coming up now. Time was devs did not care and would provide specs, a means to code and allow it. Now device makers want only allow certain people do develop so they block things in certain ways. Originally if you got your code onto a chip that faked a ROM chip well enough you could run things, today there are likely many layers of protection and restriction so you get to cut through those, on top of figuring out what is there in the first place (Nintendo does not announce to the world it uses certain chips so you have to figure that out). For every restriction though there is a, usually more and more involved, workaround though. These days you will also want to know how computer operating systems work and the layers of one (games usually run in a restricted mode where the operating system of a console can do far more, you probably want that level of access rather than a game level).
http://gbatemp.net/threads/some-hacking-concepts-and-links.287721/ has a bit more.