Usually, you don't "search" for a vulnerability, you find it by seeing a crash, or by knowing an unpatched bug in the browser and trying it.
Nintendo doesn't use their own browser but Opera/webkit, which is open source so they NEED to release the source code for that program. Analyzing the sources can help find vulnerabilities, if you understand what would make the browser crash.
If you really want to "search" (and not find), then you need to first understand how the chipsets are working (PPC, ARM, ASM, registers, etc.)
then what's good or bad coding (like using string comparison instead of memory comparison to compare two strings --> it's the Wii vulnerability)
then either analyze the sources, or decompile existing programs to get ASM sources (machine language) and understand what's happening in low level and could be exploited.
it could be race attacks (registers not correctly verified before read, or random slowdown allowing another program to edit it between two accesses -> this is the current OSDriver exploit we are using on 5.3.2)
It could be encryption algorithm bug (like PS3), so you need to learn cryptology.
So, to search/find exploit, you need to understand low level coding and how the console is working internally (from power ON to program execution).
And you don't decide to search for it, usually you find an issue by understanding how everything is working. you look at the sources, decompile programs, etc. and you learn from it, see what happens, and if you are a good developer you notice something that the nintendo developers should have done better and you know why, so you see if you can use that vulnerability to do anything with that "bug" (or bad coding style from official devs). That's not always possible, not all errors can be exploited.