Personally, I like Majora's Mask over Ocarina of Time. Never got into Banjo Kazooie (but liked Yooka-Laylee). Definitely liked Star Fox 64 and Mario Kart 64. The controller was pretty terrible, IMO. The frame rate in most games was also pretty terrible. I'd say the NES, SNES, and GBA were all better systems. The Gamecube had a vastly superior controller and it really was the Gamecube/Xbox/PS2 generation where 3D wasn't so absolutely terrible*.
I admittedly have had a very Nintendo-heavy background, so it's hard for me to point at most non-Nintendo consoles and have a good frame of reference for them. Having said that, the original GB was pretty terrible for its screen. The Game Gear was terrible for the battery life. The XBox wasn't terrible, but it was way too FPS heavy. The PS3/Xbox 360 seem to me too PC and that's continued with modern consoles. The Wii has way too many useless motion control stuff, and the Wii U just doesn't have much of a library. The Neo Geo's games were horribly expensive.
Basically, there's a long list of weaknesses for most consoles. A lot of it can be overlooked because, of course, the focus is having enough fun games to play. It's still worth mentioning the many weaknesses many systems have precisely because part of being "best" is often at least in part "least worse". It's hard to put the N64 very high on my list precisely because decent 3D technology just wasn't ready--well, not for any reasonable price point.
* PS1 had horrible polygon warping, Saturn had questionable transparency support, and N64 had generally terrible frame rates in most games and really low quality textures. There was also the issue of terrible cameras in a lot of games, but that's more a symptom of 3D games being such a nascent thing. I will give Nintendo credit with their whole snapping to behind view camera controls, even if at best it's a hack.
