Ho boy, this is a hard list to narrow down. Of course all the "must haves" should be on my list, but that doesn't leave a lot of room for other games. So I have chosen the best from both, or my favourites that I wouldn't go without:
Mario Kart 64
Super Mario 64
The Legend of Zelda: Majora's Mask
Banjo-Kazooie
GoldenEye 007
Super Smash Bros.
Mario Party (series)
Conker's Bad Fur Day
Mario Tennis
Pokémon Snap
Star Fox 64
Donkey Kong 64
1080 Snowboarding
Blast Corps
Harvest Moon 64
Hey You Pikachu!
Hybrid Heaven
Jet Force Gemini
Mario Golf
Mystical Ninja Starring Goemen
Ogre Battle 64
Pokémon Stadium
Star Wars: Rogue Squadron
Shadows of the Empire
Turok
Wave Race 64
San francisco Rush 1
Yoshi's Story
I would say that Hybrid Heaven is one of my favourite unsung heroes of the N64. It's aged terribly, but it has its own feel, its own style. So many memories on this list.