The Saturn was badly designed so it was pretty difficult to make games for it (many times developers had to use assembly rather than C)
And the N64 had the same issue - it was incredibly badly designed.
Specs-wise, it was better than the PlayStation, but this was capped by its overall built - for example, each texture was capped to 4kb, effectively meaning that to properly texture without blurring, you needed to use several layers of textures (Conker's Bad Day), which is why games opted to use shading instead.
Another issue was the shared RAM memory, which works fine and dandy on paper, but in reality can greatly delay access times due to high latency and using DMA copying from Main RAM was not an option, nearly nullifying the benefits of having a cartridge medium and causing graphical pop-in as shown in for example Turok and its despicable draw distance. Some designers were ingenious enough to include texture streaming straight from the cartridge to avoid the trainwreck of the built-in system (Indiana Jones and the Infernal Machine), but that's just crude busy work.
The cartridge size was another limit - even at that time, 64 Megabytes (512Mbit) just wasn't enough for a full-feature game, which is why N64 games often have to use crude codecs for FMV's or lack FMV's altogether, as they take up a lot of space - instead, in-game cutscenes were used. The forementioned textures more often than not, even with trilinear filtering, looked blurred due to space conservation. A game that takes up one CD would have to be on over 10 cartridges to be a 1 to 1 port, and cartridges were much more expensive than CD's.
It's also worth mentioning that at the time, programmers weren't used to coding for a 64-bit platform yet - it was a completely new experience, often resulting in buggy games.
Another bottleneck was the fillrate of the system - despite being capable of a higher polycount than the PlayStation or the Saturn, its memory management effectively crippled overall graphics and required severe work-arounds to achieve good results.
All these reasons and more made programming for the platform a complete nightmare - there are very good reasons as to why there are "so many games" for it. Every single "good" part of the design happens to have its "nightmarish" counterpart.
The system has its highlights, but one shouldn't be blinded by nostalgia - it has plenty of flaws.
//Completely off-topic, I know.