Homebrew Homebrew app [Release] Video player for 3DS

[AleronIves]

Thanks for the suggestions AleronIves, I’ve tried these myself with not great results, even at 240p Handbrake encoded videos struggled on the 3ds

I have the same problem. So far, I can't figure out which x264 setting is making the 3DS upset. I haven't given up, though.

I‘ve tried encoding at 800x240 and they looked all weird, when I turned on correct aspect ratio it was just a long, narrow vertical stripped video in the centre of the screen.

Yeah, this is because 800x240 is 10:3, and the 3DS screen is 5:3, because it doesn't use square pixels. IIRC HandBrake allows you to add extra x264 options, so you can add

--sar 1:2

to your encoding options to signal that the video is supposed to be 400x240 (5:3) during playback, and then it should fill the screen on 3DS video player 1.3.3. This will also give you the correct aspect ratio when you play the video on a PC.

 

[P34ch]

Hmm it’s all a bit weird isn’t it, especially handbrake making videos laggy. The best option I’ve found is using my old walkman video encoder (which funnily enough has 400x240 resolution), I can encode videos on that at 800x480 and they play perfectly, and are close to full screen. That’s why I suggested the stretch option, without realising the strange resolution probably won’t allow this. Other than that I just think manually zooming in, with the black border and writing removed or faded out after a few seconds would work the best of all.
Thanks AleronIves for your help with seeing the difficulties of programming a full screen.
Core-2-Extreme I hope you can add an option or a fade out to get rid of the top border and icons in windowed mode, that would be amazing. Thank you again for your fantastic work, 1.3.3 is much improved from earlier versions.

 

[AleronIves]

Other than that I just think manually zooming in, with the black border and writing removed or faded out after a few seconds would work the best of all.

What do you mean? Press Select to enter full screen mode and get rid of all the UI stuff on the bottom screen.

 

[P34ch]

AleronIves, what I meant was when you press select and exit fullscreen, there is a black bar at the top of the top screen. With frame rate, date and time and icons up the corner.
I would like to see a way to turn this off, then when you manually zoom in with R1 you can set your own sized videos to suit, but without the black bar getting in the way at the top.
Hopefully there’s a way for Core-2-Extreme to add this option.

 

[AleronIves]

I think I solved the mystery of choppy playback in hardware mode. Core_2_Extreme used the baseline profile and the fast preset to encode test videos, and I looked at what these settings actually do:

      --profile <string>      Force the limits of an H.264 profile
                                  Overrides all settings.
                                  - baseline:
                                    --no-8x8dct --bframes 0 --no-cabac
                                    --cqm flat --weightp 0
                                    No interlaced.
                                    No lossless.

      --preset <string>       Use a preset to select encoding settings [medium]
                                  Overridden by user settings.
                                  - fast:
                                    --rc-lookahead 30 --ref 2 --subme 6
                                    --weightp 1

The baseline profile disables CABAC, 8x8DCT, and b-frames. The fast preset limits reference frames to 2 (from the default of 3). Using these settings makes videos play without stuttering in C2E's tests, so I ran some of my own to figure out which settings are important.

other = deblock, rc-lookahead, me, subme tunings

t01: baseline, fast (same as C2E's tests)
t02: baseline, other
t03: baseline, 3 ref, other
t04: 8x8dct, cabc, 2 bframes, other
t05: 8x8dct, cabc, 3 ref, 2 bframes, other
t06: 8x8dct, cabc, 3 ref, 3 bframes, other
t07: cabac, 3 ref, 3 bframes, other

Tests 1-3 played fine, but tests 4-7 have the stuttering effect; therefore, it appears that either Nintendo's hardware decoder doesn't like b-frames, or there is a bug in the video player when it comes to passing b-frames to MVD_Services.

With this knowledge, I ran a few more tests on the settings related to the baseline profile:

t08: 8x8dct, cabc, 3 ref, 0 bframes, other
t09: cabac, 3 ref, 0 bframes, other
t10: 8x8dct, 3 ref, 0 bframes, other
t11: 0 bframes, other

All 4 tests play fine, and the FPS for CAVLC (test 9) and CABAC (test 11) is almost identical. The averages were 223 and 222 for my clip, respectively. It seems MVD_Services handles CABAC just fine, as long as you don't use b-frames.

tl;dr

Using baseline profile is not required; just disable b-frames, and hardware decoding should work. Using 3 reference frames is also fine; using more might work, but it would also increase the memory required to decode the stream, and since the 3DS has very little RAM, using tons of references is probably a bad idea.

AleronIves, what I meant was when you press select and exit fullscreen, there is a black bar at the top of the top screen. With frame rate, date and time and icons up the corner.
I would like to see a way to turn this off, then when you manually zoom in with R1 you can set your own sized videos to suit

Oh, I see. I guess you could do that, but why not crop the video when you encode it so that it'll fill the screen in full screen mode, rather than manually zooming the video during playback?

 

[Core_2_Extreme]

Hmm it’s all a bit weird isn’t it, especially handbrake making videos laggy. The best option I’ve found is using my old walkman video encoder (which funnily enough has 400x240 resolution), I can encode videos on that at 800x480 and they play perfectly, and are close to full screen. That’s why I suggested the stretch option, without realising the strange resolution probably won’t allow this. Other than that I just think manually zooming in, with the black border and writing removed or faded out after a few seconds would work the best of all.
Thanks AleronIves for your help with seeing the difficulties of programming a full screen.
Core-2-Extreme I hope you can add an option or a fade out to get rid of the top border and icons in windowed mode, that would be amazing. Thank you again for your fantastic work, 1.3.3 is much improved from earlier versions.

ok, I'll just make video to topmost so you can overwrap the UI with video.

t01: baseline, fast (same as C2E's tests)
t02: baseline, other
t03: baseline, 3 ref, other
t04: 8x8dct, cabc, 2 bframes, other
t05: 8x8dct, cabc, 3 ref, 2 bframes, other
t06: 8x8dct, cabc, 3 ref, 3 bframes, other
t07: cabac, 3 ref, 3 bframes, other[/code]
Tests 1-3 played fine, but tests 4-7 have the stuttering effect; therefore, it appears that either Nintendo's hardware decoder doesn't like b-frames, or there is a bug in the video player when it comes to passing b-frames to MVD_Services.

With this knowledge, I ran a few more tests on the settings related to the baseline profile:

t08: 8x8dct, cabc, 3 ref, 0 bframes, other
t09: cabac, 3 ref, 0 bframes, other
t10: 8x8dct, 3 ref, 0 bframes, other
t11: 0 bframes, other

All 4 tests play fine, and the FPS for CAVLC (test 9) and CABAC (test 11) is almost identical. The averages were 223 and 222 for my clip, respectively. It seems MVD_Services handles CABAC just fine, as long as you don't use b-frames.

Thank you for your information, I'll also investigate it.

 

[Core_2_Extreme]

I think I solved the mystery of choppy playback in hardware mode. Core_2_Extreme used the baseline profile and the fast preset to encode test videos, and I looked at what these settings actually do:

      --profile <string>      Force the limits of an H.264 profile
                                  Overrides all settings.
                                  - baseline:
                                    --no-8x8dct --bframes 0 --no-cabac
                                    --cqm flat --weightp 0
                                    No interlaced.
                                    No lossless.

      --preset <string>       Use a preset to select encoding settings [medium]
                                  Overridden by user settings.
                                  - fast:
                                    --rc-lookahead 30 --ref 2 --subme 6
                                    --weightp 1

The baseline profile disables CABAC, 8x8DCT, and b-frames. The fast preset limits reference frames to 2 (from the default of 3). Using these settings makes videos play without stuttering in C2E's tests, so I ran some of my own to figure out which settings are important.

other = deblock, rc-lookahead, me, subme tunings

t01: baseline, fast (same as C2E's tests)
t02: baseline, other
t03: baseline, 3 ref, other
t04: 8x8dct, cabc, 2 bframes, other
t05: 8x8dct, cabc, 3 ref, 2 bframes, other
t06: 8x8dct, cabc, 3 ref, 3 bframes, other
t07: cabac, 3 ref, 3 bframes, other

Tests 1-3 played fine, but tests 4-7 have the stuttering effect; therefore, it appears that either Nintendo's hardware decoder doesn't like b-frames, or there is a bug in the video player when it comes to passing b-frames to MVD_Services.

With this knowledge, I ran a few more tests on the settings related to the baseline profile:

t08: 8x8dct, cabc, 3 ref, 0 bframes, other
t09: cabac, 3 ref, 0 bframes, other
t10: 8x8dct, 3 ref, 0 bframes, other
t11: 0 bframes, other

All 4 tests play fine, and the FPS for CAVLC (test 9) and CABAC (test 11) is almost identical. The averages were 223 and 222 for my clip, respectively. It seems MVD_Services handles CABAC just fine, as long as you don't use b-frames.

tl;dr

Using baseline profile is not required; just disable b-frames, and hardware decoding should work. Using 3 reference frames is also fine; using more might work, but it would also increase the memory required to decode the stream, and since the 3DS has very little RAM, using tons of references is probably a bad idea.


Oh, I see. I guess you could do that, but why not crop the video when you encode it so that it'll fill the screen in full screen mode, rather than manually zooming the video during playback?

According to my test, MVD service(hardware decoder) seems to can't handle B-frame at all (or my code is wrong).
As you said, no matter what preset, profile (baseline, main or high) and level you use hw decoder handle it fine if you don't use B-frame.
e.g.

ffmpeg -i {input} -acodec copy -vcodec h264 -crf 15 -profile:v high -preset veryslow -level:v 6.2 -x264-params bframes=0 {output}

Also if you use B-frame, no matter what preset, profile (main or high) and level you use hw decoder won't handle it fine.(video will be played at lower framerate maybe hw decoder is ignoring B-frame????? I need to inspect more)
e.g.

ffmpeg.exe -i {input} -acodec copy -vcodec h264 -crf 15 -profile:v main -preset ultrafast -level:v 1.0 -x264-params bframes=1 {output}

 

[P34ch]

That’s brilliant Core-2, on both points, I for one can’t wait for the next verson now. Many thanks in advance.

 

[AleronIves]

I can think of one other thing that might help... As you know, ThirdTube is using your video player to play YouTube videos. I haven't seen the stuttering effect on YT videos, so maybe you could examine how YT playback is working?

I downloaded a YT video with youtube-dl at 144p, 240p and 360p and looked at the streams. Google encoded them with x264, but it removed the x264 header with the encoder settings. I used ffmpeg to convert from DASH to regular MP4 and then opened the 144p video with Avidemux, and the video contains B-frames, so it appears MVD_Services does support B-frames under certain circumstances. The N3DS browser also lets you play YouTube videos, and I assume it uses MVD_Services for that...

 

[Core_2_Extreme]

I can think of one other thing that might help... As you know, ThirdTube is using your video player to play YouTube videos. I haven't seen the stuttering effect on YT videos, so maybe you could examine how YT playback is working?

I downloaded a YT video with youtube-dl at 144p, 240p and 360p and looked at the streams. Google encoded them with x264, but it removed the x264 header with the encoder settings. I used ffmpeg to convert from DASH to regular MP4 and then opened the 144p video with Avidemux, and the video contains B-frames, so it appears MVD_Services does support B-frames under certain circumstances. The N3DS browser also lets you play YouTube videos, and I assume it uses MVD_Services for that...

Youtube provide 2 types of video file in h264 format

youtube-dl.exe -F {url} | grep -i "avc"
160          mp4        256x144    144p   91k , mp4_dash container, avc1.4d400c@  91k, 30fps, video only, 3.56MiB
133          mp4        426x240    240p  198k , mp4_dash container, avc1.4d4015@ 198k, 30fps, video only, 7.67MiB
134          mp4        640x360    360p  579k , mp4_dash container, avc1.4d401e@ 579k, 30fps, video only, 22.43MiB
135          mp4        854x480    480p 1091k , mp4_dash container, avc1.4d401f@1091k, 30fps, video only, 42.25MiB
136          mp4        1280x720   720p 2206k , mp4_dash container, avc1.4d401f@2206k, 30fps, video only, 85.44MiB
298          mp4        1280x720   720p60 3410k , mp4_dash container, avc1.4d4020@3410k, 60fps, video only, 132.08MiB
299          mp4        1920x1080  1080p60 5755k , mp4_dash container, avc1.64002a@5755k, 60fps, video only, 222.87MiB
18           mp4        640x360    360p  735k , avc1.42001E, 30fps, mp4a.40.2 (44100Hz), 28.48MiB
22           mp4        1280x720   720p 2335k , avc1.64001F, 30fps, mp4a.40.2 (44100Hz) (best)

First one is video only, these files are encoded main or high profile so it contain B-frames.
Another one is normal mp4 file (video with audio), these files are encoded main profile except format 18.

Thirdtube uses format 18(360p baseline) if video duration <= 1h
I think that's why thirdtube plays youtube video witiout any problems.
I also patched format number to force it to use format 134(360p main) as a result video was played at lower framerate (same as my app).

New3DS browser also uses format 18 as well???? I'm not sure.
Edit : Actually, youtube provide format 18(360p baseline) and 22(720p main) for older devices.
So New3DS browser plays 360p baseline video.

 

[AleronIves]

Oops, I forgot ThirdTube uses 360p. I thought 144p/240p would be the most interesting, since the lowest quality would probably have limited settings, but I guess not.

I did find one other interesting thing. I remembered that Blu-ray has limits on B-frames, and x264 usually does not respect these limits. I tried using:

--b-pyramid strict 
--b-pyramid none

and the low framerate problem got maybe 50% better. It's still choppy, but it's not a slideshow anymore. It seems like B-frame support might work, if we could just figure out the specific way Nintendo expects them to be used. Based on this test, it seems MVD_Services is picky about how you use B-frames as references.

 

[AleronIves]

I made 6 more test encodes to probe MVD_Services a bit more. It appears 800x240 is limited to 4 reference frames, as using more than this causes the 3DS to crash. This would suggest 640x480 (for 4:3 content in 800px mode) is limited to 5 reference frames, and 400x240 is limited to 8 reference frames, assuming my calculations are correct.

If you can confirm this, you might want to add a check that will abort playback and show an error message that this video won't play in hardware mode, instead of crashing if the video requires a larger DPB than MVD_Services can provide.

Would it be possible to add a debug option to make the video player present itself as an O3DS game? If the N3DS underclocks itself to 268 MHz, disables the extra CPU cache, and threaded decoding is disabled, it should be possible to benchmark O3DS performance on a N3DS, right? I assume you'd have to restart the video player for the setting to take effect, since a game can't change between O3DS and N3DS modes while it's already running, but it would be a handy testing feature.

 

[MarioKartFan]

Assuming @Core_2_Extreme is able to get the HW decide issue sorted out would a port of Moonlight be feasible making use of this? Would be incredible to stream games to the N3DS.

 

[AleronIves]

Moonlight could use the decoding capabilities, but there's a lot more to it than that, since you also need support for button inputs and streaming over Wi-Fi. Since a Wii U client exists, somebody may decide to make a 3DS one some day, but it's way out of scope for this project.

In positive news, I tried converting some 4:3 DVD footage to 640x240 (SAR 1:2), and it looks great, although the Dolby Digital audio has some clicks and pops during playback, so I guess AC3 support isn't 100% right now.

Unfortunately, it seems my calculations for reference frames were wrong. I can play 640x240 with 4 references in HW mode just fine, but 5 references yields a crash, just as it does with 800x240.

 

[MarioKartFan]

I’m aware that it is out of scope. My question is whether the capabilities being developed here by @Core_2_Extreme could be leveraged in a streaming client.

 

[Core_2_Extreme]

I made 6 more test encodes to probe MVD_Services a bit more. It appears 800x240 is limited to 4 reference frames, as using more than this causes the 3DS to crash. This would suggest 640x480 (for 4:3 content in 800px mode) is limited to 5 reference frames, and 400x240 is limited to 8 reference frames, assuming my calculations are correct.

If you can confirm this, you might want to add a check that will abort playback and show an error message that this video won't play in hardware mode, instead of crashing if the video requires a larger DPB than MVD_Services can provide.

Would it be possible to add a debug option to make the video player present itself as an O3DS game? If the N3DS underclocks itself to 268 MHz, disables the extra CPU cache, and threaded decoding is disabled, it should be possible to benchmark O3DS performance on a N3DS, right? I assume you'd have to restart the video player for the setting to take effect, since a game can't change between O3DS and N3DS modes while it's already running, but it would be a handy testing feature.

It may be because MVD work buffer size.
try 1.3.4-dev (3dsx cia) it allocate (MVD_DEFAULT_WORKBUF_SIZE * 2) (about 18MB) instead of (width * height * 9) bytes (in 1.3.3).

Yes, it's possible except memory size.
If you want to simulate memory size I think you can't do it with same build you need to compile your .cia as legacy mode.

Assuming @Core_2_Extreme is able to get the HW decide issue sorted out would a port of Moonlight be feasible making use of this? Would be incredible to stream games to the N3DS.

Interesting.
Is it streaming client (and remote control), right?
And is there any documentation about communication protocol between server and client?

 

[AleronIves]

Yes, it's possible except memory size.

If you can add an option to run in Legacy mode, I'll run some encoding tests for O3DS, too. I don't think the memory size will be a problem, since even O3DS has 64MiB for games.

Moonlight is on GitHub. If you want to work on a 3DS version, you want Moonlight Embedded:

https://github.com/moonlight-stream/moonlight-embedded

Here's the Wii U version based on it:

https://github.com/GaryOderNichts/moonlight-wiiu

 

[Core_2_Extreme]

If you can add an option to run in Legacy mode, I'll run some encoding tests for O3DS, too. I don't think the memory size will be a problem, since even O3DS has 64MiB for games.

Moonlight is on GitHub. If you want to work on a 3DS version, you want Moonlight Embedded:

https://github.com/moonlight-stream/moonlight-embedded

Here's the Wii U version based on it:

https://github.com/GaryOderNichts/moonlight-wiiu

I added fake model function (settings -> advanced settings -> use fake model)
3dsx cia (same link as previous dev version)

thanks, I may do it someday.

 

[MarioKartFan]

That would be amazing.

 

[AleronIves]

Well, the fake model option is definitely working, because performance is terrible in O3DS mode.

I am a little confused though... I tried XviD 240x144p @ 24 fps, and the average fps is 51, and the recent average is always above 40, but the video is still playing in slow motion with skipping audio. If the decoding speed is > 40 fps on a 24 fps clip, shouldn't it play at full speed? Based on your benchmarks, H.261 is only ~6 fps faster than H.263+ (not quite the same as XviD, but close enough), so it seems like you'd need to use 160x96 or even 80x48 to get full speed on O3DS...

On the N3DS front, the larger buffer for MVD_Serivces is working. Clips with 5-9 reference frames no longer crash the system.