Well, let me check what's going on (I suspect it's sar problem) so can you:
1. Open video player ->
2. Play the video ->
3. Stop the video ->
4. Go back to main menu where you see gear icon on right bottom ->
5. Tap on gear icon ->
6. Tap on Advanced settings ->
7. Tap on Dump logs ->
8. Upload the log (/3ds/Video_player/logs/YYYY_MM_DD_hh_mm_ss.txt) here
Edit: I mean if you are complaining "I encoded my video at 320x240 but it's displayed as if it was 320x225 in full-screen mode", then I think it's sar problem.
What I mean is, fullscreen playback works fine, but I believe that a film displayed pixel-perfectly on the upper screen should not be scaled just because of UI elements like the clock, WiFi icon, or the green leaf indicator. This is, I think, a natural rule of video playback software — pixel-perfect display should not be arbitrarily disrupted, let alone by non-essential interference like this. This is the issue I care about most. It is the only issue I care about most. Nobody wants a pixel-perfect video to be inexplicably scaled because of a green leaf or WiFi icon — the beautiful pixel-perfect image suddenly becomes blurry. And even if those UI elements overlap and cover part of the image, there's no need to scale it at all.
I also think that if touching the screen can enter UI mode, then touching it again should be able to exit UI mode as well.
There is another issue with this player, though I'm not sure whether you can solve it: it decodes RGB888 video but outputs RGB565, which causes a lot of unnatural color banding (color contour lines). I'd like this to be fixed — specifically, I want the New 3DS to hardware-decode H.264 video and output RGB888, rendered on a video player that also renders in RGB888. Perhaps this problem can be solved. I know that Nintendo apparently defaults to RGB565 output in some situations, including decoding and rendering software, but I think it might be possible to fix this. RGB888 video is far superior to RGB565.
However, I came up with a quirky workaround entirely on my own after thinking about it for a long time. The original goal was to eliminate the color banding I was seeing when playing video on the 3DS — I had assumed for a long time it was a compression issue on my end. My video encoding command is:
ffmpeg -nostdin -y -stats -stats_period 1 ^
-i "%FILE_IN%" ^
-vf "crop='min(iw,ih*400/240)':ih,subtitles='%SUB_ESC%',format=yuv420p16le,zscale=w=-2:h=240:m=bt709:min=bt709:filter=spline36,scale=sws_dither=bayer:in_color_matrix=bt709,format=rgb565le,scale=flags=neighbor:sws_dither=0:out_color_matrix=bt709,format=yuv420p"^
-c:v libx264 -preset slow -crf 15 -profile:v high -level 3.1 -fps_mode cfr ^
-x264-params "aq-mode=3:aq-strength=1.5:qcomp=0.65:ref=5:bframes=5:no-fast-pskip=1" ^
-color_primaries bt709 -color_trc bt709 -colorspace bt709 ^
-c:a aac -b:a 128k -ac 2 -ar 48000 ^
"%FILE_OUT%" || pause
As you can see, I first convert the color to 16-bit, then convert to RGB565 with Bayer dithering applied, and then map it back to the nearest RGB888 colors with no dithering, before encoding. The idea is that by doing this, the colors are already encoded in the RGB565 color space at compression time — and with dithering applied, more colors can be simulated. When played back, there will be no unexpected, unnatural color banding from an RGB888-to-RGB565 conversion, because those color contour lines that would have appeared are identified in advance during the 10-bit source → RGB565 conversion step, and Bayer dithering is used to simulate additional colors. Combined with encoding settings that preserve fine detail without dropping small features (to retain the dither pattern), the unnatural banding disappears entirely.
I'm including image comparisons — the right side is my optimized version with the RGB565 intermediate step to RGB888; the left is the standard version. My player is set to RGB565 pixel-perfect display, so what you see is exactly what the real 3DS shows.
I also recommend this video encoding script to others.
I also want to show you some of the videos I've encoded specifically for the New 3DS — all at 400×240. I've encoded quite a lot of them.
Respect, and keep going!

