StarFox 64/Lylat wars. Maybe, turrican would be further up the list and I would probably look either side in the franchise and I also have to say Panzer Dragoon.
Doom 64 - Doom on a console... yeah.
Mario Kart 64. Already covered. A flawed game rendered pointless by its sequels in a lot of cases though even those could stand to be better. Going by your "need to own both" I would say many of the competitors worked about as well.
Paper Mario. Have not put enough time in, probably would look to later consoles though.
Star Wars Shadow of the Empire. The was a PC version, also this when Jedi Knight/Dark Forces is a thing?
Star Wars Rogue Squadron. Good, eclipsed by later entries in the franchise.
Banjo. Good, XBLA is better.
Conkers. Actually never got around to it,
will have to link this though.
Mario Party, would later go on to compete against Madden, Pokemon and Megaman
battle network in "least changes made to a franchise but still keep it popular" stakes". Credit where credit is due it did popularise the party game style.
Perfect Dark, XBLA version absolutely and options wise I would hold it up as one of the finest multiplayer games made. Borderline unplayable on the N64 today though.
Blast Corps
F-Zero X. Though I have never been a fan of high speed racing games like this Wipeout does have to be mentioned and was huge at the time.
Wave Race 64. Though I am sure the massive jet ski demographic remembers. Also how much of it is because it was a launch title and there was sweet fuck all for a not inconsiderable amount of time.
Killer Instinct Gold. When the arcade version exists?
Sin and Punishment
1080 Snowboarding. A weak control scheme (this was pre Tony Hawk mind you) and in a world where SSX exists? Sticking with the time I hold the Cool Boarders series was the better choice, not sure where Snowboard kids falls into this.
WWF No Mercy. A wrestling game, really?
[on the PS1]
Road Rash I played to death. Depending upon where you find yourself those might be fighting words.