Video Codec for GBA?

Discussion in 'GBA - Game Development, ROM Hacks and Translations' started by DSLSC, Sep 13, 2013.

  1. DSLSC
    OP

    DSLSC Member

    Newcomer
    28
    3
    Aug 27, 2013
    United States
    I would like to make a wireless remote camera project for the GBA, and will need some compression to make the video framerate useable. The encoder will be a Parallax Propeller, an 8-core 32-bit CPU running at 80MHz (=20MIPs/core.)

    JPEG has been considered, although technically, that's not video (MJPEG, perhaps!) I did come across Tony Savon's Caimans Video Codec (also a version for the DS.) It looks very promising, and I sent Tony an e-mail on Monday. However, the last activity on both of those websites is 2005...I haven't received a response, and don't really expect to. Does anyone here have source code, a library, or just some really good documentation? Or is there another codec that I should be considering?
    Admittedly, I can reverse-compile a Caimans Video Codec example download, although that's disrespectful, and doesn't make the encoder part any easier. On second thought, the demos play very glitchy on VisualBoyAdvance, audio and video don't match on the GBA, and the video is nonfunctional on the DS (in GBA mode.) Some demo, if you ask me...but it's mostly there.

    BTW, this project isn't for the DS because it doesn't have a link port. I'm not sure whether I could interface an RF transceiver to a DS's slot 1 (definitely not slot 2.) RS-232 at 115,200 baud is the datarate I'm hoping to achieve. If needed, depending on the transceiver chip/module, I could probably get up to 256,000 baud. And no, WiFi on the DS won't do, as I'm hoping for a 600+ foot range. Thanks in advance for ideas.
     


  2. Lemmy Koopa

    Lemmy Koopa M3 Perfect fanboy

    Member
    347
    282
    Sep 17, 2012
    United States
    Ohio
    So is this like GBA camera?

    You know you could probably do wifi with the DS.
     
  3. FAST6191

    FAST6191 Techromancer

    pip Reporter
    23,154
    8,895
    Nov 21, 2005
    To the best of my knowledge there are only 3 video players for the GBA, the Rad video tools appeared first on the DS (and there it was not used so much but that is probably because of the VX/MODS stuff). No source code or anything like that and I would probably go in for reinventing the wheel if I had to do it.

    The caimans stuff you mention.

    Meteo ( http://wiki.pocketheaven.com/index.php?title=Meteo ) aka METEO Avi-2-GBA Video Codec.

    Whatever Nintendo used for the GBA videos (I believe some precursor to the mobiclip/actimagine stuff we see on the DS with the extensions vx and mods).

    I suppose there is also the GBA AV adapter/TV player -- http://www.superufo.com/review_html/GBA_SP_AV_Adaptor.html http://gbatemp.net/threads/1650-gba-sp-av-adaptor-j.20867/ though good luck finding one of those in 2013.

    There are a handful more for the DS but that is neither here nor there, not to mention fewer issues redrawing the screen in bitmap and the speed boosts and dual processor nature of things.
     
  4. DSLSC
    OP

    DSLSC Member

    Newcomer
    28
    3
    Aug 27, 2013
    United States
    Nope. As I mentioned in my initial post, that doesn't go close to far enough. I am really hoping to make a wildlife spycam, with an optional children's ride-on toy for a moveable base. And yes, I know that such things are for sale, but what's the fun in that? :)

    I suppose the real question is: Does someone have up-to-date contact info for Tony Savon? I looked at decompiling an example binary, but that's only half of the job, as I don't have an encoder. And, as I mentioned, I'd really like to have his approval, rather than appear to be taking advantage of someone else's work.
     
  5. cualquiercosa327

    cualquiercosa327 Member

    Newcomer
    46
    4
    Mar 24, 2009
  6. DSLSC
    OP

    DSLSC Member

    Newcomer
    28
    3
    Aug 27, 2013
    United States
    Nice find...some more rabbit trails to chase. OK, so Caiman's is based on the MPEG-4 video codec. Source code for that shouldn't be too hard to find. MPEG-1 might even be easier. I tried disassembling Caimans 1.0 a few weeks back, and was making some sense of it...but without getting a 100%-complete disassembly, bug-tracing would probably be nearly impossible. After all, program memory can be ARM instructions, THUMB instructions, a Data Table or Memory...

    Thanks a lot for the link.
     
  7. Tony Savon

    Tony Savon Member

    Newcomer
    10
    0
    Aug 25, 2003
    Hello guys,
    Sorry if I'm coming back to you after so long. I was cleaning up my secondary email accounts and I found recent inquiries about the GBA Caimans codec. A quick search took me here, so I thought I would spend some time to answer your questions.
    But first I have to say that, as most of you probably understood, I've been away from the GBA for quite some years, so please forgive me if some memories are blurred. Caimans codec was born out of an experiment and my will to learn how to develop for the GBA. It evolved into a commercial project, but the drive behind it never ceased to be the same one that brings you guys here: to have some fun. It was probably the last time I had so much fun developing something, and I have fond, albeit distant memories of it.
    I eventually sold my IP to Activision (in 2007, I think) and thus I cannot release the encoder (not that I would know where to find the source code any more). The websites are still up an running as a form of nice memory, but really the service is no more.
    That said, after all these years, I think it's fine to give a very high level description of the techniques that were used. I'm really grateful that you are still talking about this codec after so many years, but it really was nothing fancy and I'm sure that you guys can now do much better than that humble Italian bedroom coder.

    There really existed 3 versions of Caimans codec: The caimans codec, the caimans codec pro and the caimans codec for DS.

    Caimans Codec / Caimans codec 2: http://www.caimans.net/gbavideo/demos2.shtml
    This first version was extremely simple: it used fixed 4x4 pixels codebooks. On the encoder side, each frame is converted to the YUV12 colorspace, divided in static blocks of 8x8 pixels (so you get up to 30x20 blocks on a GBA screen). A first pass of the encoder finds out the "scene changes", or frames that differer for more than a certain % from the previous one. Scene changes are called I-frames. If more than a certain number of frames go by without any scene change, an I-frame is forced anyway.
    For all the frames in between two I-frames -we call then P-Frames- the encoder would then identify a set of codebooks that it could use to approximate all the blocks of all those frames. I think I used 20-30 kbytes sets, depending on the target quality. The search for the optimal set of codebook was done with k-mean algorithm and each block could be encoded with a single entry or a set of 4 entries in the codebook table (more on this later), or with a special code "zero", which means "do not update this block from the previous frame". Since codebooks are 4x4 pixels and blocks are 8x8 pixels, a block can be either a single codebook entry which is X and Y doubled at playtime, or a set of 4 codebooks. This simple tweak made it possible to encode sequences where some objects zooms in or out quite effectively.
    That's it for the encoder.
    At play time, the player simply gets an I-frame and a set of codebooks. It displays the I-frame then proceeds to decode the p-frames using a timer driven irq (I think). Each call just updates the blocks that need to be updated reading from the codebook table. This until a new I-frame is reached and a new codebook table is fetched.
    Seek was possible as each I-frame is completely independent. So the rom had a table with the offset of the I-frames and the player could simply jump to that offset.
    I remember that I frames, including all the codebooks, ended up being huge, so the codebooks were not physically stored together with the I-frame, but each of them was "streamed" together with the P-frame that was first using it.
    The first implementation was done in C and it was a mess but I managed to circulate a couple of proof of concepts around. I had no plans other than that. But one day Activision calls me to use it in Spiderman 2! They asked me if it was usable as a library. I said "yes!", but of course it wasn't :-). A few sleepless night followed that email (I had a proper job to do during the day) during which I rewrote a lot of the code in assembler, I made a library out of the player and I was able to send it to them. I remember that, because of the simplicity of the encoding technique, it was extremely fast. I remember I could run it at 60 fps if I wanted, and the whole player core was only few KB and would fit entirely in the on-chip memory, or whatever it was called (sorry, I forgot the GBA-terminology).
    I also developed a very simple audio codec based on ADPCM, but to be honest, although some of the demos on the website use it, it was never used in commercial videogames: the bitrate required for the video was so high (I think that it was 50-70Kbytes/sec) that saving few kbytes on a 16Khz 8-bit PCM mono track was totally pointless.

    Caimans Codec Pro: http://www.caimans.net/gbavideo/demos.shtml
    After the codec appeared in Spiderman 2 and in a number of other games, I got calls from other developers who wanted to store more video. I developed the caimans codec pro for the purpose, which further expanded the original idea. This time I used the YUV9 color space, but the main difference is that it added variable size and non-square codebooks. Again, a block is always 8x8, but codebooks can be one of the following sizes:

    8x8
    8x4
    4x4
    4x2

    A block is encoded and decoded recursively: if a good 8x8 approximation is found, then go with it. Otherwise break the block into two 8x4 and go down recursively. you could end up with a block that had the upper half encoded with a single entry, the lower half encoded with 4 entries, or 6 or whatever.

    On the top of that, I added a very basic form of motion compensation. A block could also be "shifted" (only entire pixels, no fancy qpel or halfpel) in the predicted frame. It really did something for the compression rate, but it killed the efficiency at playtime (more on this later)

    I kept the k-mean algorithm as the foundation of the encoder, but due to the complexity of the new algorithm, the codebook search was painfully slow. I remember that it took a good 30 minutes to encode less than 1 minute of video on my pentium 3. It would have been trivial to improved the encoder, but I never really liked to work on that part, I was so much more into optimizing the player :-).
    speaking of it, I did any trick I could come up with, I optimized my code as much as I could, but I never managed to push it beyond 15fps. Which was enough though. The video quality was only ok imho, but the bitrate could go down to few kbytes per second. It was only used commercially in a couple of products. One of them, A Lizzy Mcguire title, contained a full episode of the show (I think it was 24 minutes). I remember the producers only gave me 10 megabytes to store the entire episode. It was painful to hand tweak sequences to save up to the last bit, and the result didn't look very nice, but kids liked it. Go figure.

    Caimans Codec DS: www.ds-video.com
    This started as an experiment after I had given my GBA IP to Activision (they never used the encoder after that though... go figure). I started toying with the DS to see if I could repeat that experience. The GBA was fading away anyway...
    At first I ported the GBA caimans codec pro, which was quite easy to do and would play at much higher frame rate due to the faster CPU (there might still be some demo on the website). But I wanted to do better so I started thinking of a full featured mpeg4 decoder. I remember I started downloading the source code of the reference mpeg4 Simple Profile player and hammered it until I had it compile for the DS (which took a long time). Finally it does, and plays video. Yeah, B/W footage at 0.5 fps :-).
    But that was a starting point, so I started profiling the player, rewriting everything from scratch and moving to assembler for the the critical parts. I added a lot of tables to speed up computation, I got rid completely of Floating Point math and so on. I got some 8-12 fps on regular mpeg SP streams and I felt I could not do better than that. So what I did was I started tweaking the mpeg4 encoder until it would bias the encoding towards those elements of the mpeg4 framework that I knew I could decode faster (codebooks) and using less and less of what I could decode very slowly (half pel, all kind of motion vectors etc). So, the player is technically playing mpeg4, but the encoder is encoding a stream that is meant to be a bit easier to decode. This had a minimal impact on the quality, but allowed me go past the 15 fps barrier that I considered the minimum acceptable quality.
    But everyday life, the job etc were starting to be more demanding, so I never really pursued that interest any more. I remember I released a dragon's lair prototype, but that was it.

    Thanks for bringing back those epic times, and good luck with your efforts!
     
  8. FAST6191

    FAST6191 Techromancer

    pip Reporter
    23,154
    8,895
    Nov 21, 2005
    The EA people seemed to have an odd approach to doing stuff on the GBA ( http://knzl.de/krawall/ ).

    Thanks for the read though, sounds like you pushed the GBA a bit harder than I would have imagined without veering into story of mel territory. When you say on chip memory do you meant the 32 kilobytes of stuff on the ARM itself? Impressive either way but in on chip ram is good stuff.
     
  9. Tony Savon

    Tony Savon Member

    Newcomer
    10
    0
    Aug 25, 2003

    Yeah, I mean the on-chip ram. TO be honest I was not even using all of it for the first version of the codec. In fact, I think the code was half of that. But if you think of it, all it had to do was to read the codebook entries from cart and copy them to ram, read the index entries from cart an use them to print the blocks on screen using codebooks as tiles. Color space conversion from YUV to RGB and a the dithering RGB888->RGB555 were probably the only time consuming parts, but still a fairly small piece of code. A very similar approach to mine(although even simpler, for obvious reasons) Vector-Quantization compression technique has been used more recently on the Commodore 64. Now, that's something worth dropping the jaw for if you ask me :-)
     
  10. Coto

    Coto GBAtemp Addict

    Member
    2,342
    392
    Jun 4, 2010
    Chile

    Yes, i've heard a lot of that one before. Wii pair single-precision integers do some sort of vector quantization (through the paired floating point pipeline hardware on the cpu)
     
  11. DSLSC
    OP

    DSLSC Member

    Newcomer
    28
    3
    Aug 27, 2013
    United States
    Hey Tony! Apologies for my shortsighted comments starting this thread, though! Just came looking through GBATemp, and surprise. (Maybe I should go back and edit that first post...) Anyway, I understand that you sold the IP to someone else in 2007. The GBA is practically history by now, and nobody's doing much with it. Does it constitute an exception to the IP rules if the IP is not being used, and has no future for being used on the GBA? I am not thinking of anything commercial, but if I can get it to work well enough, I'd certainly consider writing an article for Nuts & Volts or some other similar magazine. That is, as long as the rules would not prohibit me from sharing a grand accomplishment with other fellow electronics hobbyists.

    My project got waylaid by the side when I couldn't get JPEG to go very fast (almost 2fps decode with 100% pure ASM), and disassembling Caimans 1.0 looked like a total and complete wreck. (The bitrate on codec 2.0 is too high...did you only sell Caimans 2.0? If so, please can you spare the source for 1.0?) The DS version might be handy, but I haven't done any serious DS programming, and am not sure whether I could interface a long-range wireless chip to the Slot-1 on the DS Lite.

    Below is a photo of a camera experiment I did in 2010 with a VV6410 CMOS image sensor, a Parallax Propeller, and a Nintendo GBA. At the moment, I was writing for the GBA in C, but soon after went to ASM, which provides much greater control. The data was being transferred in bitmap form.

    GBA Cartridge Video test (overview, 07-05-2010).jpg
    In the picture on the GBA screen, you can see a clock by the bookshelf, and an Epson Stylus Photo R800 printer below. Notice that I was using a 16-bit bus to communicate between the Parallax Propeller (in the breadboard) and the GBA. I also tried a version using RS-232 (@115Kbits) that went slightly slower due to the bitmap nature. (If you're wondering, the chips on the top right of the breadboard comprise a parallel-port programmer for a GBC cartridge hack.)

    The main reason I didn't pursue JPEG is because I couldn't get decode to run fast enough on the GBA. The Propeller has 8 32-bit CPUs running at 20MIPS, so compression would not be a speed problem. (The 32K of shared RAM in the Propeller would be the challenge!)

    Only a "bedroom coder"? Lots of amazing creations have come from humble places. Look forward to hearing from you soon. And again, apologies for my comments :grog:
     
  12. Tony Savon

    Tony Savon Member

    Newcomer
    10
    0
    Aug 25, 2003

    Hi DSLC. No offence taken, I don't really see what the apologies should be for (or perhaps I just didn't read the thread completely :-) ).

    Anyway, as I said I'm afraid I don't have the code for the codec any more. When I gave my stuff to the guys at Activision I had to sign some intimidating paperwork so I just stopped the development and I kinda lost track of the code. I might still have some old backups somewhere but I moved to London since then... not sure where (and when) I could look for them...

    You have an Impressive setup there! If you want to "stream" video, I think that even if you had this codec available, it would not suit your needs very well, as the vector search at the encoder side would add latencies. You have to have a number of frames available before you can create the codebooks, and only after you have them (in my case all the frames between two I-frames), you can encode them against the codebooks. MJPEG is perhaps still your best guess, being i-frames only.
    I did something with jpegs too http://www.caimans.net/gba/ . That too was used in spiderman 2, now that I think of it. The cut scenes are compressed with that.
    The trick to make jpeg decode faster, at the cost of some loss in quality, is to use a custom color space conversion matrix. Not sure which one you are using for your jpeg decoder but If it's the "proper"one descrived here: http://en.wikipedia.org/wiki/JPEG#Color_space_transformation
    Than it's very likely to be a bottleneck. I got the idea of using an approximated one studig codecs like Indeo, CVID and other prehistoric formats that were designed to run on very simple architectures. In order to speed up YUV->RGB they use the following matrix

    | R | | 1.0 0.0 2.0 | | Y |
    | G | = | 1.0 -0.5 -1.0 | | Cb|
    | B | | 1.0 2.0 0.0 | | Cr|

    Which works surprisingly well, and on the GBA screen you won't notice the color shift anyway. Unlike the proper matrix, this conversion only requires integer math (1/2 and 2x can be computed with shifts)
    I did some experiments with motion jpeg too and I could definitely step in the 5-6 fps domain, as far as I remember, especially using only a portion of the screen, which seems to be the case for you. Also, you can implement a very simple lossy layer on the top of the vector quantization at the encoder end: just send, say, half of the quantized blocks of each frame: those that differ the most from the same blocks in the previous frame. This halves the data streamed as well as the CPU required to decode it, and as a metter of fact doubles the frame rate...
    Sorry I can't be of more help.
     
  13. Foxi4

    Foxi4 On the hunt...

    pip Reporter
    23,528
    21,456
    Sep 13, 2009
    Poland
    Gaming Grotto
    If you are looking for a homebrew solution, look into Eponasoft's codec for the DS. It's heavily outdated source-wise, but could be a valuable source of information nonetheless.
     
  14. DSLSC
    OP

    DSLSC Member

    Newcomer
    28
    3
    Aug 27, 2013
    United States
    Hmm. . .well, I don't have any experience at all with any sort of video codec, and was just hoping that someone who does have experience could help out! Actually, I was intending to go full-screen, and was just starting with a 160x120 bitmap because that's all I could cram into the Propeller's RAM without a realtime encoder of some sort. (On top of that, I wasn't debayering the sensor's data, so that's really an 80x60 bitmap.)

    I'm assuming that you did your programming on a PC, and not on paper, so I guess I don't quite understand. Latencies don't bother me in this case. 5-6fps begins to reach acceptable, but the Caimans codec had audio crammed on top of a higher framerate. Sorry if I sound jealous. . .but I just think it would be great to put to use something that's been forgotten! However, I do understand the legalistic challenges possibly involved. Maybe they'd be agreeable to releasing the IP back to you, especially as they probably aren't making a dime off of it anyway.

    Attached is my JPEG decoder source, and it displays a JPEG as fast as possible. Transparent on-screen text is accomplished by using all of the sprites in a bitmap form. . .this was the actual start of my "camera project", but halted due to the lack of framerate speed. The top number is the CPU cycle count for the decode operation; the lower text is the start of communications progress. Yes, I can probably tweak it a little more, but again. . .:wacko:

    Attached is a ZIP file containing what I've accomplished, namely the GBA file (which decodes a 13KB JPG—a worst-case scenario to make it as fast as possible before reducing quality and filesize), and the 55KB ASM code decoder. I used a combination of Jeff Frohwein's JPEG decoder, a BASIC decoder, and a Windows C++ decoder.

    Thanks for your time, and especially for showing back up!
     

    Attached Files:

  15. GHANMI

    GHANMI GBAtemp Advanced Fan

    Member
    966
    491
    Jun 10, 2012
    Kingdom Hearts Chain of Memories did have videos on the GBA.
    I don't think it's the same as Nintendo's demos because of the size difference, but that's a wild guess.
     
  16. DSLSC
    OP

    DSLSC Member

    Newcomer
    28
    3
    Aug 27, 2013
    United States
    Foxi4: that might help. However, I do understand that the DS has 2 CPUs (running faster), and more RAM. The VB6 code could help a lot, though, especially as I haven't seen any source code for a video codec encoder. (It does have to be something that I can port to 100% ASM in the memory-limited Parallax Propeller. . .extensive use of external object libraries could render it unusable.) My biggest challenge with the JPEG encode/decode routines that I've found is the extreme memory requirements. Extreme in the sense of ~32K+, as when you pop another 4GB RAM stick in your computer, 512K suddenly means nothing at all!

    Will look into it. Thanks.

    GHANMI: If you're running video from the cartridge, the bandwidth there is probably adequate for uncompressed video! I'm trying to keep this under 250Kbit/s (~25KByte/s), as that's the fastest synchronous speed possible on the GBA link port. (The RFM69HW RF transceiver module goes to 300Kbit/s (~30KByte/s), but that's in best conditions. Speed and signal strength sort of go together.) Besides. . .where's the source code ?!
     
  17. Foxi4

    Foxi4 On the hunt...

    pip Reporter
    23,528
    21,456
    Sep 13, 2009
    Poland
    Gaming Grotto
    There are some crafty ways to cut down on the decoding process, but don't expect going beyond Thunderdome with the framerate without buffering - real-time playback is a bitch even on the DS and aside from commercial solutions I have not seen many successful video codecs on either platform aside from DPG which probably works on fairy dust.

    I'm pretty sure both the GBA and the DS have some hardware decoders you could use to off-load the CPU, but you'll have to check GBAtek for the details. Good luck!
     
  18. Coto

    Coto GBAtemp Addict

    Member
    2,342
    392
    Jun 4, 2010
    Chile
    Well the vector quantization approach is useful under very tight memory requirements and the bandwidth is halved because you transfer groups of them only:

    - You divide n elements (coordinate vectors, pixels) which will result in m "groups". called centroid points (k-means teory)
    - Accounting that a centroid point has a fairly similar density to each other (N:1 centroid point)
    - (1: ) centroid point "distance" seeks to define a "run" for every stored element on it:

    Y:X^p -> (N: ) K

    where p linear is a dimension (assymetric) of: Y variable of scalar or vector X , existing over field (N: ) K
     
  19. Tony Savon

    Tony Savon Member

    Newcomer
    10
    0
    Aug 25, 2003

    Hi,
    I remember that Jeff Frohwein's decoder was quite "off the book" and definitely slow. Burton Radons' implementation, although C only, was at least twice as fast, and definitely faster than mine. Why don't you give it a try? you'll be amazed at how much faster it is
    This looks like it:
    https://github.com/ricbit/Oldies/tree/master/2005-11-graphosds/arm9/source/jpeg
    Of course, this C implementation screams for a porting to asm, for running it in IWRAM and all these things I've long forgotten. But even in its current state it looks like something you can try with minimal effort.
     
  20. DSLSC
    OP

    DSLSC Member

    Newcomer
    28
    3
    Aug 27, 2013
    United States
    I've seen that library before. I managed to build a test project with Burton Radon's C-based library (not something I do very well), and gasp. . .it decodes the same JPEG just a tiny bit slower than my pure ASM one! (To be fair, I ran both from CARTROM, as there were some devKitPro errors that disabled IWRAM.) However, my only real problem is that I don't understand C very well, which is why I prefer to program in ASM. Translating the C code with all those structs and typedefs, pointer-to-member. . .let's say that the end result better be worth it! Which is why I based my library on Frohwein's library: I could follow that C code. Perhaps the less understandable C code is, the more efficient it is. (Because CPUs don't speak English.)

    Probably the routine costing the most in CPU cycles in my version is the "Decode" routine, where the CPU is searching through the Huffman tables for a match. Expanding the table for a straight-match would require 16KB of memory (if I recall correctly).

    Thanks for the tips. I do get what you're saying about the Caimans' encoder, although I can add 2MB of DRAM to the Propeller externally...