It most certainly can. Maybe not h264 but DivX/XviD should work very well.
h264 is more demanding than DivX/XviD but I'm not sure how much more, it might be able to handle 480p h264 just fine. I had an old linux based handheld with no hardware acceleration and it could play 480p XviD just fine. The 3DS is a lot more powerful than that, so h264 might work fine too.
Would be perfect if it could play 480p h264 then you could just copy scene releases onto it or stream them

I think 3D would probably have to be downscaled to 400x240x2 (not that it matters much) since it would require twice as much processing power as 480p in 2D otherwise.