I have the same problem. So far, I can't figure out which x264 setting is making the 3DS upset. I haven't given up, though.Thanks for the suggestions AleronIves, I’ve tried these myself with not great results, even at 240p Handbrake encoded videos struggled on the 3ds
Yeah, this is because 800x240 is 10:3, and the 3DS screen is 5:3, because it doesn't use square pixels. IIRC HandBrake allows you to add extra x264 options, so you can addI‘ve tried encoding at 800x240 and they looked all weird, when I turned on correct aspect ratio it was just a long, narrow vertical stripped video in the centre of the screen.
to your encoding options to signal that the video is supposed to be 400x240 (5:3) during playback, and then it should fill the screen on 3DS video player 1.3.3. This will also give you the correct aspect ratio when you play the video on a PC.