I dont think movie recording is possible (too weak hardware for that), but there was a plugin for PSP called SV Capture that u could record small GIF's.
Unfortunately a RemoteJoy alternative is not possible for 3DS since the port was only designed to charge the 3ds battery.