Well one thing is that the Nintendo DS has two CPUs one at 67MHz and the other at 33MHz with 4MB of memory and seperate video memory. The Nintendo 64 has a 93.75MHz processor with a "Reality Co-Processor" at 62.5MHz and 4MB of RAM expandable to 8MB.
Most N64 games could virtually be ported to Nintendo DS. One thing is that the N64 supported resolutions of up to 640x480, Nintendo DS supports one resolution, 256x384 (counting both screens). Besides that, one of those screens is usually just a map or something without 3D graphics. So knock that down to 256x192. Nintendo DS cartridges, I assume, have a faster transfer speed than Nintendo 64 cartridges and the memory is probably faster. Not only that, ARM is a very efficient architecture which means that more tasks can be completed in less instructions and memory is more efficient.
Emulating N64 games on DS is 100% impossible, but most games could be relatively easily ported to DS. Other games would need a virtually perfect port with every optimization in the book to work, games like Majora's Mask may be able to handle it if the memory is more efficiently used. That separate video memory on the DS would certainly help wonders.
Here's how this discussion ends...
Emulation of N64 on NDS = impossible
Porting of some N64 games to NDS = easy
Porting of other N64 games to NDS = very difficult
Enjoy, thanks for listening!