Homebrew Homebrew app [Release] Video player for 3DS

[AleronIves]

I think I'm starting to understand. It looks like MVD_Services needs both PTS and DTS. It's impossible to play the video properly without both of them.


First, I found this: https://web.archive.org/web/2018070...k:80/janos/2008/06/08/b-frames-in-directshow/


If the stream does not have B-frames (I and P only), you only need one set of timestamps, because all P-frames reference the I-frame or P-frame that came before them, and frames are stored in playback order.

I <-- P <-- P (decode order = playback order)

DTS and PTS are required for a stream that contains B-frames, because you need two pieces of information. DTS is used for decoding, because the frames must be stored out of playback order. Since B-frames refer to multiple references, the references must be decoded before you can decode the B-frames. This means a P-frame that is displayed after a B-frame during playback must be decoded before the B-frame, or else you can't decode the B-frame because one of its references isn't available.

Once you decode the frames, you need PTS to put them back into playback order so you can display them. Without PTS, you'll play the frames in the wrong order.

I <-- P <-- B (decode order)
I <-- B --> <-- P (playback order)

According to this: https://stackoverflow.com/questions/13595288/understanding-pts-and-dts-in-video-frames


raw DTS values are arbitrary, because they're used as offsets. The first DTS is converted to "wall clock" time (in this case, 3DS system clock time?), and all subsequent DTS values are an offset from the "wall clock" time.


This page: https://www.ramugedia.com/how-generate-dts-pts-from-elementary-stream


shows the difference between decoding I and P only vs I, P, and B; however, it doesn't cover the case when B-frames are used as references (--b-pyramid x264 setting).

This would indicate that you need to give frames to MVD_Services with DTS and PTS. The out of order frame bug is probably caused by giving DTS only, and giving PTS only will cause the frames to be corrupt, since you can't decode them in PTS order.

 

[Core_2_Extreme]

I think I'm starting to understand. It looks like MVD_Services needs both PTS and DTS. It's impossible to play the video properly without both of them.


First, I found this: https://web.archive.org/web/2018070...k:80/janos/2008/06/08/b-frames-in-directshow/


If the stream does not have B-frames (I and P only), you only need one set of timestamps, because all P-frames reference the I-frame or P-frame that came before them, and frames are stored in playback order.

I <-- P <-- P (decode order = playback order)

DTS and PTS are required for a stream that contains B-frames, because you need two pieces of information. DTS is used for decoding, because the frames must be stored out of playback order. Since B-frames refer to multiple references, the references must be decoded before you can decode the B-frames. This means a P-frame that is displayed after a B-frame during playback must be decoded before the B-frame, or else you can't decode the B-frame because one of its references isn't available.

Once you decode the frames, you need PTS to put them back into playback order so you can display them. Without PTS, you'll play the frames in the wrong order.

I <-- P <-- B (decode order)
I <-- B --> <-- P (playback order)

According to this: https://stackoverflow.com/questions/13595288/understanding-pts-and-dts-in-video-frames


raw DTS values are arbitrary, because they're used as offsets. The first DTS is converted to "wall clock" time (in this case, 3DS system clock time?), and all subsequent DTS values are an offset from the "wall clock" time.


This page: https://www.ramugedia.com/how-generate-dts-pts-from-elementary-stream


shows the difference between decoding I and P only vs I, P, and B; however, it doesn't cover the case when B-frames are used as references (--b-pyramid x264 setting).

This would indicate that you need to give frames to MVD_Services with DTS and PTS. The out of order frame bug is probably caused by giving DTS only, and giving PTS only will cause the frames to be corrupt, since you can't decode them in PTS order.

So, do I need to pass packet to decoder in dts order then I need to display decoded image in pts order, right?

 

[AleronIves]

Yes, it seems MVD_Services needs DTS for decoding, and then you'll need the PTS to shuffle the decoded frames into the correct order for playback. If the video doesn't have B-frames, then the PTS shuffle won't actually change the order of the frames, but if the video does have B-frames, the order will change and prevent the "out of order" frames bug.

 

[Core_2_Extreme]

Yes, it seems MVD_Services needs DTS for decoding, and then you'll need the PTS to shuffle the decoded frames into the correct order for playback. If the video doesn't have B-frames, then the PTS shuffle won't actually change the order of the frames, but if the video does have B-frames, the order will change and prevent the "out of order" frames bug.

If I pass in dts order, output image will corrupt and then,

1) MVD_Services requires more cleanup when you close a video to make sure there are no leftover frames in memory before you start a new video. This should fix the grey static problem, and it may fix some of the crashing, too.

I found that if video does contain B-frames, mvd service sometimes won't write anything to the output buffer.
I mean (let's say we have 3 output buffers and playing video that display character 1, 2, 3, 4, 5, 6... )
if video does not contain B-frames

//start
decode();
buffer[0] = get_frame();//buffer[0] contains raw image that displays character "1"
//next
decode();
buffer[1] = get_frame();//buffer[1] contains raw image that displays character "2"
//next
decode();
buffer[2] = get_frame();//buffer[2] contains raw image that displays character "3"
//next
decode();
buffer[0] = get_frame();//buffer[0] contains raw image that displays character "4"
//next
decode();
buffer[1] = get_frame();//buffer[1] contains raw image that displays character "5"
//next
decode();
buffer[2] = get_frame();//buffer[2] contains raw image that displays character "6"

so we see 1, 2, 3, 4, 5, 6.... on screen but if video does contain B-frames

//start
decode();
buffer[0] = get_frame();//buffer[0] contains raw image that displays character "1"
//next
decode();
buffer[1] = get_frame();//buffer[1] contains raw image that displays character "2"
//next
decode();
buffer[2] = get_frame();//buffer[2] contains raw image that displays character "3"
//next
decode();
buffer[0] = get_frame();//does not write anything to the output buffer so, buffer[0] contains raw image that displays character "1"
//next
decode();
buffer[1] = get_frame();//buffer[1] contains raw image that displays character "5"
//next
decode();
buffer[2] = get_frame();//buffer[2] contains raw image that displays character "6"

so we see 1, 2, 3, 1, 5, 6.... on screen so, you were right, I need to clean up output buffer (usually decoder always write new picture every time so it doesn't need or just waste time if do so)
And one more problem, even mvd service didn't write anything to the output buffer it returns "SUCCESS" code instead of "ERROR"!
Why? idk.

So, in this dev version I don't reorder anything, but I fill output buffer by one color(let's say blue) and check it before displaying it.
If it seems(I don't scan entire buffer because it cause massive slowdown) same color, meaning mvd service didn't write anything to the buffer, just display next picture.
by doing so out of order problem has been fixed, but it just displays next buffer so framerate stays low (however, I think it's better than v1.3.3).
Press ZR to toggle detect blue image mode.
3dsx cia

 

[AleronIves]

And one more problem, even mvd service didn't write anything to the output buffer it returns "SUCCESS" code instead of "ERROR"!

Hmmm... It definitely seems like MVD_Services has some bugs with B-frames that may not be fixable...

So, in this dev version I don't reorder anything, but I fill output buffer by one color(let's say blue) and check it before displaying it.

Well, when I disable "blue mode" with ZR, the video playback looks like it did before you added the raw image buffer: there are no out of order frames or glitches, but the playback is choppy. When I press ZR to enable "blue mode", I get a strong strobing effect, because the screen turns blue every time MVD_Services fails to draw a frame. If I press A to pause at the right time, my whole screen is blue, so it appears that you are drawing the blue frame to the screen, instead of drawing the next video frame, which makes it difficult to look at the screen without getting a headache.

On the bright side, it does appear that the FPS increases in "blue mode" vs regular mode, so it would probably be an improvement if you can stop the blue frames from being displayed.

If the video doesn't have B-frames, it doesn't matter whether I use "blue mode" or not, and playback is fine. Unfortunately, I played a few videos without B-frames, and then when I tried to play a video with B-frames again, my 3DS froze, and I had to hold the power button to turn it off. When I press B to stop a video, it says the raw image buffer still has like 50 frames in it. Do you need to erase the buffer when stopping a video? IDK...

 

[Core_2_Extreme]

Well, when I disable "blue mode" with ZR, the video playback looks like it did before you added the raw image buffer: there are no out of order frames or glitches, but the playback is choppy. When I press ZR to enable "blue mode", I get a strong strobing effect, because the screen turns blue every time MVD_Services fails to draw a frame. If I press A to pause at the right time, my whole screen is blue, so it appears that you are drawing the blue frame to the screen, instead of drawing the next video frame, which makes it difficult to look at the screen without getting a headache.

If you enable "detect blue image", you shouldn't see any blue image.
If you disable it, you'll see blue image if video contains B-frames that's expected behavior.
But you reported opposite, could you check it by pressing SELECT button to open log in home menu?
note : "detect blue image" is disabled by default.

 

[AleronIves]

You're right; I had it backwards. I thought it would only insert blue images when you press ZR, but it actually always inserts blue images, and pressing ZR determines whether to hide them or not.

I still don't understand why this is necessary, though. The blue image check restores the original B-frame playback from before you added the raw image buffer: there are no glitches or out of order frames, but the animation is not smooth.

The raw image buffer made playback much better for O3DS, but it made MVD_Services playback worse, right? Can you disable the raw image buffer in hardware mode only, so you don't need the blue image check to prevent out of order frames, as in v1.3.3?

 

[Core_2_Extreme]

You're right; I had it backwards. I thought it would only insert blue images when you press ZR, but it actually always inserts blue images, and pressing ZR determines whether to hide them or not.

I still don't understand why this is necessary, though. The blue image check restores the original B-frame playback from before you added the raw image buffer: there are no glitches or out of order frames, but the animation is not smooth.

The raw image buffer made playback much better for O3DS, but it made MVD_Services playback worse, right? Can you disable the raw image buffer in hardware mode only, so you don't need the blue image check to prevent out of order frames, as in v1.3.3?

Actually, decoding speed dropped however, number of decoded frames have been increased.
In v1.3.3 only 27/50 frames were displayed while in dev version (with blue image detection) 38/50 frames were displayed. (although it should display 50/50 frames....)
I think decoding speed dropped in this dev version is also because it process more frames.
I used 864*480@24fps video to test.

So I think it's better to use blue image detection even decoding speed will drop.
Or are you experiencing different result?

 

[AleronIves]

In v1.3.3 only 27/50 frames were displayed while in dev version (with blue image detection) 38/50 frames were displayed. (although it should display 50/50 frames....)

Oh, I see. I didn't realise the blue image version was displaying more frames, so I agree that's better. I guess my eyes are not good enough to see a difference between 27/50 and 38/50 frames. I wonder if it will ever be possible to reach 50/50 frames...

How will you control this in v1.4.0, though? If blue image detection is always enabled, videos that don't use B-frames will get reduced performance, even though blue image detection is not needed. Is there a way to auto-detect if a video uses B-frames, so you can enable or disable blue image mode automatically? I guess you could read the x264 settings in the H.264 stream to see if the string " bframes=0 " is present. If it is, you could disable blue image mode. If bframes>=1 or the string is missing, use blue image mode? It would be nice if there were a better way to tell the difference, since e.g. YouTube 360p baseline profile doesn't have B-frames, but there is no x264 header to read to detect that fact.

 

[Core_2_Extreme]

v1.4.0 release

Decoded image(raw) buffer has been added and it makes playback
much much much much better (especially on OLD3DS)
Spanish(español) translation has been added (by Cookiee)
Romanian(Română) translation has been added (by Tescu48)

https://github.com/Core-2-Extreme/Video_player_for_3DS/releases/tag/v1.4.0

 

[AleronIves]

Congratulations on a great release.

Maybe you're aware of this already, but it seems games can use up to 30% of the CPU time on the system core on O3DS?

http://3dbrew.org/wiki/APT:SetApplicationCpuTimeLimit

Maybe you could use the system core for audio decoding and/or video resizing, so you could get a little bit faster video decoding on the main core? I suppose 30% CPU time is not enough to do threaded video decoding on O3DS, but it might help for other tasks...

 

[Core_2_Extreme]

Congratulations on a great release.

Maybe you're aware of this already, but it seems games can use up to 30% of the CPU time on the system core on O3DS?

http://3dbrew.org/wiki/APT:SetApplicationCpuTimeLimit

Maybe you could use the system core for audio decoding and/or video resizing, so you could get a little bit faster video decoding on the main core? I suppose 30% CPU time is not enough to do threaded video decoding on O3DS, but it might help for other tasks...

Actually, I'm already using 80% of core #1 on O3DS, 20% on N3DS.
Core mask on O3DS :
Drawing #0
Decoding audio #0
Decoding video #0
Color conversion #1
Reading data from SD #1

on N3DS :
Drawing #0
Decoding audio #0
Decoding video #2 if core #2 isn't available, then #0
Color conversion #3 if core #3 isn't available, then #0
Reading data from SD #1

 

[Zeninari]

Can Someone Tell Me If It Can Decode HD Videos? Most Of My Videos Are Either 720p Or 1080p.

The Only File I Got Working So Far Was A Video That Was 480p, And Funner Than That It Was A Screen Capture Of The Intro To Avalon Code Ds xD

Any Response Would Be Appreciated, Thanks!

 

[Zeninari]

Hmm Maybe Not, I Was Able To Get A Wide Screen HD Video Working, Just A Bit Slow.

Most Of The Errors I Keep Getting Is Mostly "Out Of Linear Memory" Or Out Of Memory In Some Way.

I Dont Know How To Fix This, Do I Just Downgrade The Files To A Lower Resolution, Decrease File Size, Or Do I Need More Space

Im Sorry With These Idiotic Questions, I Just Want To Know How To Fix It Or Whats Happening. xD


EDIT: And Before I Froget, Some Videos Still Load, But I Get An Error Before It Disappears.
"Fsuser_Open File Failed" What Does This One Mean?

 

[Core_2_Extreme]

Hmm Maybe Not, I Was Able To Get A Wide Screen HD Video Working, Just A Bit Slow.

Most Of The Errors I Keep Getting Is Mostly "Out Of Linear Memory" Or Out Of Memory In Some Way.

I Dont Know How To Fix This, Do I Just Downgrade The Files To A Lower Resolution, Decrease File Size, Or Do I Need More Space

Im Sorry With These Idiotic Questions, I Just Want To Know How To Fix It Or Whats Happening. xD


EDIT: And Before I Froget, Some Videos Still Load, But I Get An Error Before It Disappears.
"Fsuser_Open File Failed" What Does This One Mean?

If you are using N3DS or N2DS it's possible to decode 720p video, but most likely it won't run at full speed.
If you are playing H.264 video, recommended resolution is 800x240(up to 60fps) on N3DS, 256x144(up to 30fps) on O3DS.

For "Out Of Linear Memory" error, you need to downscale your video.

For "Fsuser_Open File Failed", do you see that error during playback or before starting playback?

 

[Zeninari]

Hmm Maybe Not, I Was Able To Get A Wide Screen HD Video Working, Just A Bit Slow.

Most Of The Errors I Keep Getting Is Mostly "Out Of Linear Memory" Or Out Of Memory In Some Way.

I Dont Know How To Fix This, Do I Just Downgrade The Files To A Lower Resolution, Decrease File Size, Or Do I Need More Space

Im Sorry With These Idiotic Questions, I Just Want To Know How To Fix It Or Whats Happening. xD

If you are using N3DS or N2DS it's possible to decode 720p video, but most likely it won't run at full speed.
If you are playing H.264 video, recommended resolution is 800x240(up to 60fps) on N3DS, 256x144(up to 30fps) on O3DS.

For "Out Of Linear Memory" error, you need to downscale your video.

For "Fsuser_Open File Failed", do you see that error during playback or before starting playback?

For Fsuser Error, It Hoppens Before Playback Starts Then Goes Away, Mostly If Not Everytime With "Every Other" Video That Can Play. Almost Every Video With The Fsuser Error Freezes My System Upon Trying To Change The Video As Well, So Rip.

And Thanks For Letting Me Know, Ill Downscale My Videos Then.

still it playing an almost slightly above hd video was insane, though it was lagging quite a bit. still a shocker though . . . (Ripped Strait From A Ps4 (Opening For Sao vs AW))

Edit: I Havent Tested All Of My Videos For Fsuser So Ill Check Now. Ill Make Another Comment If Its Both. But For Now Still What Is Said Above "Before Playback"

Edit 2: Some Of The Fsuser Errors Just Freeze The System Immediately. Still Before Playback Though. Also I Was Suprised To See The First Few Frames Of A 9 Minute Video I Had In HD. Honestly Very Suprising xD

 

[Zeninari]

For Fsuser Error, It Hoppens Before Playback Starts Then Goes Away, Mostly If Not Everytime With "Every Other" Video That Can Play.

And Thanks For Letting Me Know, Ill Downscale My Videos Then.

still it playing an almost slightly above hd video was insane, though it was lagging quite a bit. still a shocker though . . . (Ripped Strait From A Ps4 (Opening For Sao vs AW))

OH I Frogot, Im Using A New Nintendo 2ds Xl With A 128Gb Sd Card Inserted

believe me or not but im actually almost out of space xD
dumping my collection was heftier than i thought . . .

Edit: Question, What About 480p Videos (Standard) Can It Run That Resolution Just Fine (I Want As Much Quality As Possible ;-; )

 

[AleronIves]

The top 3DS screen is 800x240. You can't get any extra quality by going above that resolution.

 

[Zeninari]

The top 3DS screen is 800x240. You can't get any extra quality by going above that resolution.

Intresting, I Was Just Preplexed, Because The One Video I Have That Plays Just Fine In Wide Screen Looks Way Higher Than The Default Resolution. My First Thought Was It Was Casting The Pixels Onto A 800x240 Screen, But I Dont Know.

But Anyway Thanks For The Response Aleronlves, Ill Keep That Info In Mind AGAIN xD

yes i know that the screen was that resoultion, as before this i was using the camera app for my videos, and it DID not support videos above the regular quality (800x400), and in the wrong codec.

Cheers!

 

[Zeninari]

Also I Am Currently Translating Files By Downgrading Them

And Im Still Amazed By The 800x240 Resolution And Amazing Resolution On The 3ds. Still Not Bad . . . Not Bad At All.

Edit: Im Impressed! The Aplication Was Able To Load A Movie(Thats INSANE!) This Application Is Something Else.

You May Ask "How Long Did The Conversion Take?"

well . . . 4 1/2 Hours

yes it hurt but it was worth it xD