Homebrew [Research] MPEG Video Codec Port

[sixtails]

Hey guys, I was thinking of trying to port some Video Codecs to the 3DS, considering people seem to be using MJPEG. In particular I was thinking of writing a more efficient video player, and some maybe a new screen capture program (SD/Wifi). While I have experience in signals and image processing/compression, I've never needed to deal with a video/audio codec before (though I'm planning just going to port it over, so I don't suspect this will be much of a problem).

Unfortunately between work, university and Pokemon it may take a few months.

So here's the preliminary planning:

1. Does anyone know if this has been attempted before? If anyone has a better starting point that would be awesome. I've looked around, but from what I can tell, we've only done MJPEG.
2. Codec Choice for Video: I want to start out with something that has very low CPU usage for both encoding and decoding video; in particular I was thinking MPEG-1 part 2 for video. Any thoughts on this? I don't readily know of anything else that does this.
   
3. Codec Choice for Audio: I'm not sure how much CPU we'll have to spare, but atleast for N3DS users, I think we can encode audio. Again, I want to choose something that has very low CPU usage. I'm not sure that the obvious choice, MPEG-1 part 3 Layer II, is best here.
4. Something I'll need to research is the implementations available for these codecs. The two prominent choices being FFMPEG and VLC. I could also look into smaller scale implementations such as http://www.uow.edu.au/~nabg/MPEG/mpeg1.html.
5. When choosing an implementation, I'll also want to keep in mind that the ARM11 should have DSP & SIMD instructions. Do any codecs take advantage of these already? Can any codecs take advantage of these?
6. Another possibility for optimization is using the GPU. I'm not sure how necessary this would be, but if anyone has an educated guess...
   
7. My 3DS specific knowledge is a bit limited here, but I assume for video capture, it's a matter of SVC backdoor to spawn a kernel11 thread and read from video memory. Anyone have any experience with this?
   
8. I don't know the the 3DS audio systems well enough to perform capture, but I assume it's a case of hooking dsp or something. Is there any documentation out there on this already?

If anyone can give some input, that would be a huge help, thanks!

 

[Roboman]

Moflex is a codec that runs very efficiently on 3ds, but its proprietary and closed source.
The 3ds development kit offers an encoder and player but videos encoded by said encoder count as warez.

There has been some work Reverse engineering the codec though.
https://gbatemp.net/threads/released-moflex-player.400053/
Most Mobiclip videos play and you can encode your videos to Mobiclip with this tool. But only I-frames are encoded, so it has... really really big file sizes.

 

[Mrrraou]

bink video was optimized for the 3ds and stuff so that should probably work too, as it worked on the ds too anyway

 

[sixtails]

Moflex is a codec that runs very efficiently on 3ds, but its proprietary and closed source.
The 3ds development kit offers an encoder and player but videos encoded by said encoder count as warez.

There has been some work Reverse engineering the codec though.
https://gbatemp.net/threads/released-moflex-player.400053/
Most Mobiclip videos play and you can encode your videos to Mobiclip with this tool. But only I-frames are encoded, so it has... really really big file sizes.

Yeah, I want to avoid proprietary formats where possible. It may be possible to write an encoder based on the decoder and just inject the videos, but I was under the impression there was some time restrictions or something on that. It also doesn't really solve the problem of a realtime encoder.

bink video was optimized for the 3ds and stuff so that should probably work too, as it worked on the ds too anyway

Again, it's proprietary, and I'm not sure that we have the power to encode in realtime. I think we also only have decoders for that as well, so I'd be on my own for the encoder.

Thanks for the help guys!

 

[uyjulian]

The n3DS has a hardware h264 decoder.

 

[sixtails]

The n3DS has a hardware h264 decoder.

This could be interesting, specifically the information from here. It's probably useful if we want to add a secondary codec to a video player, especially for battery life and space. However I'm unsure about trying to use this to encode videos as we haven't REd it to the point where we know what the hardware is.

 

[sixtails]

I've been busy this weekend with some tournaments, completing my Sun run-through and other stuff. But I got some time to look at some codecs online.

Upon examination, I'm not sure where VLC is getting it's MPEG codec from, but it seems to have some dependency, also it seems to only support NEON FPOs which the 3DS doesn't have access too. Next I looked at FFMPEG, which has it's codecs split into a library called libavcodec. This one actually looks pretty promising, and it seem to have pretty good support for ARM optimizations. Also the codebase for MPEG-1 is the same as MPEG-2, so it should be very easy to add MPEG-2 later on, as well as experimenting with MPEG-2 encoding (I'm thinking that a N3DS may be able to handle it). I also looked at gstreamers implementation, but for some reason the encoder is classified as "Bad". It's also coded in C++, ick, and doesn't seem to have a hardware implementation.

The tentative roadmap for the project is as follows:

1. Compile libavcodec for linux-x64.
2. Develop frontend.
   1. Figure out frame type and develop plan to build around that.
   2. Look into how multithreading and pipelining is handled.
      
   3. Write MEPG-1 part 2 to frame decoding, and frame to MPEG-1 part 2 encoding frontend.
   4. Write MPEG-1 part 3 layer II to PCM decoding, and PCM to MPEG-1 part 3 to layer II encoding frontend.
   5. Add container format to unify frontends. Probably just MPEG-1 Part 1.
3. Strip away the rest of ffmpeg (this can be added back later for the video player, but the encoder will need to minimize on RAM usage if we want live encoding to work on an O3DS).
4. Port over project for 3DS.
5. Develop AV player based on frontend decoding.
6. Develop live capture program to SD using frontend encoding.
   1. Figure out how to use SVC to spawn K11 Thread.
   2. Figure out how to efficiently capture frames.
   3. Figure out how to capture audio.