The PS1 had dedicated sound and video chips whereas the N64 was unified, I'm sure you have played a few games where the sound was cropped in various ways, particularly on split screen multiplayer..
More importantly whilst the N64 could indeed display more textured polygons on screen than the PS1, this rarely happened as it only had 4Kb of texture memory, compared to the 1Mb dedicated video memory on the PS1, of which a variable portion could be used for textures... the result, combined with the low resolution anti-alaising on the N64, was what gave many 3D games such a 'muddy' (or more cartoony) look: textures had to be very low quality because of such a lack of memory for them... Many games resorted to shading on polygon sides when there wasn't any memory available, giving it a light source too, but again this wasn't as good as the PS1.
Otherwise though, you're right, the biggest drawback of the N64 was of course the cartridge format... had it been cd based then yeah, even with the problems above, games would've been consistently better, and the ability to stream music off cd would've nullified the sound chip advantage the PS1 has anyway.
But, to say the PS1 was 'a lot weaker' than the N64 remains untrue....
, and why has the 3DS version of Retroarch not got the 4DO (3DO emulator) core, judging by your statement? The 3DO was a 32bit console, and also 'a lot weaker' than a PS1 - we should be smashing 60fps on all 3DO games on our 3DSes by now....
The PS1 is very simple architecture, which explains why emulation is so advanced compared to similar consoles, and that's almost as important as how powerful the machine is.
[/picky bastard mode]