I heard MAME 2003 works a wee bit better than MAME 2000 (it's not built in with the stable Retroarch pkg), you might want to give it a try. I couldn't download it since Download Cores doesn't work for me, but you could find them, even here on GBATemp.
But don't get your hopes up. The comptability is supposedly only slightly better. Still, it's worth a try.
The complex structure of the PS3 makes it hard to develop certain emulators. Not sure but I read that the N64 emulator is not possible since it uses OpenGL, which is not supported on PS3, I think? That's what I read, it might not be correct, but either way the N64 emulator is out of question. I use my Wii for that so it's fine for me.