Video scaling is not currently available on RetroArch 3DS builds. The issue is indeed tearing, though, as in pictures not locked to vblank (even with VSYNC enabled in the settings). The tearing is vertical rather than horizontal because the 3DS screens are actually vertical-orientation screens turned sideways. It's not a scaling/filtering issue: currently the screen is just bilinear filtered to fill the entire screen, which looks bad but at least doesn't result in the pixel warping you'd get from doing the same without filtering the image.
EDIT: A great place to test this is anything that does flashing on a frame-by-frame basis. The only one I can think of off the top of my head is Alex Kidd in Miracle World for the Sega Master System; the title screen logo flickers seizure-inducingly by design. When running under RetroArch you can see the tearing interfere with this effect such that the second half of the screen will be updated. The tearing does seem intermittent, so you might not observe it on the first loop of the attract mode, but it should become clear eventually.
EDIT2: OK, I did some screwing around with this, the issue appears to be that RetroArch is currently not able to autodetect the refresh rate and just uses whatever its default is. Take a look at Settings > Video for "Refresh Rate". I've forgotten what the default was, but try reducing this manually to 59.834 or so. I don't know that this is precisely correct, but it seems to improve (eradicate?) the tearing issue. Knowing the exact refresh rate would be useful here, but this certainly seems (more) functional.