Accessing the DRC/TV video buffer

Discussion in 'Wii U - Homebrew' started by ::Phoenix::, Dec 16, 2016.

  1. ::Phoenix::
    OP

    ::Phoenix:: GBAtemp Regular

    Member
    184
    170
    May 11, 2010
    Italy
    Hi there, I am writing an homebrew which would hopefully allow to take screenshots at anytime while the Wii U executes. I already managed to get VPADRead hooked thanks to function patcher from loadiine's code and I am also able to hook such function whenever a title gets loaded.

    The only thing I am missing is a way of accessing the raw video buffer of the DRC and/or TV in order to be saved to sd card after a specific button press (which wil be checked thanks to the VPADRead hook). Is accessing the video buffer something that can be done with kernel privileges or do I need IOSU access?

    It seems the browser is able to access this data when sharing screens on miiverse, so there must be a way even without an exploit, but don't know how...

    -EDIT- Maybe I found the right function, should be GX2SetTV/DRCBuffer.
     
    Last edited by ::Phoenix::, Dec 16, 2016
  2. shinyquagsire23

    shinyquagsire23 SALT/Sm4sh Leak Guy

    Member
    1,962
    3,231
    Nov 18, 2012
    United States
    Las Vegas
    I'd disassemble GX2CopyColorBufferToScanBuffer, that's what actually ends up copying buffers to the displays. For some games there might only be one color buffer copied to both screens. There's probably an official function used somewhere else though, I've never really looked.
     
  3. ::Phoenix::
    OP

    ::Phoenix:: GBAtemp Regular

    Member
    184
    170
    May 11, 2010
    Italy
    Thank you! I will check it out. As for know I am looking for something in gx2.rpl which allows to copy buffers from one format to another. It seems the TV/DRC buffer can be initialized to many formats (with the most frequent one being R8G8B8A8, I think), but it would be a pain in the ass to implement the conversion routine from scratch.
     
  4. Maschell

    Maschell GBAtemp Advanced Fan

    Member
    900
    1,314
    Jun 14, 2008
    Gambia, The
    In HID to VPAD I'm also hooking into GX2CopyColorBufferToScanBuffer to draw a cursror, but it doesn't work with all games. (demo)

    https://github.com/Maschell/hid_to_vpad/blob/master/src/patcher/function_hooks.cpp#L49

    To hook easily into functions, you can use
    https://github.com/Maschell/function-patcher-example

    I started to make screeenshot tool serval month ago, if you're interested in it, I can share the source code. My goal was to "record" the gamepad screen by making 30 screenshots/second.
    http://gbatemp.net/threads/wii-u-hacking-homebrew-discussion.367489/page-936#post-6255919
    https://vid.me/WIkU
     
    Last edited by Maschell, Dec 16, 2016
    I pwned U! and shinyquagsire23 like this.
  5. DeslotlCL

    DeslotlCL GBAtemp's Saint Holy Sword Dragon

    Member
    1,884
    2,069
    Oct 28, 2015
    Chile
    under your bed
    Can't believe i didn't see that shovel knight footage you captured with screenshots only, that is really amazing!

    Does it slowdown the gameplay while capturing screenshots for the video? It would be cool to record online gameplay footage with it.
     
  6. Maschell

    Maschell GBAtemp Advanced Fan

    Member
    900
    1,314
    Jun 14, 2008
    Gambia, The
    Yes, thats the reason why I tested it with Shovel Knight. This was the only game I found that could keep 25-30fps :D
    I also spent a few days on porting a mp4 codec to the wiiu (from the wii), but gave up. Everything should still sleep somewhere on my HDD, if somebody is really interested in continuing it, I can search/share it.

    btw. Don't try to write many small screenshots to your SD Card, the nintendo API is killing your sd card (Won't recognize it until you format everything)
     
    Last edited by Maschell, Dec 16, 2016
  7. DeslotlCL

    DeslotlCL GBAtemp's Saint Holy Sword Dragon

    Member
    1,884
    2,069
    Oct 28, 2015
    Chile
    under your bed
    Still, you made an impressive job dude!
     
  8. ::Phoenix::
    OP

    ::Phoenix:: GBAtemp Regular

    Member
    184
    170
    May 11, 2010
    Italy
    I am already using your function patcher. But started from scratch instead of that example though. My goal is less ambitious, I would like to take single screenshots in png format after a button combination has been pressed and do that from anywhere in the Wi U. This seems to be working thanks to the EXIT_RELAUNCH_ON_lOAD feature of HBL, which allows me to let the hooks work within the currently running title. What I am insterested to know is if you spent some time implementing conversion routines for all the possible buffer formats in order to save them to jpeg, or you found tha all games use plain simple RGBA8?
     
    Last edited by ::Phoenix::, Dec 16, 2016
  9. Maschell

    Maschell GBAtemp Advanced Fan

    Member
    900
    1,314
    Jun 14, 2008
    Gambia, The
    I only implemented the conversion for like 2 or 3 texture formats. I can send you my compression/conversion code when I'm home)
    Maybe @AboodXD has conversions for all textures formats?
     
  10. ::Phoenix::
    OP

    ::Phoenix:: GBAtemp Regular

    Member
    184
    170
    May 11, 2010
    Italy
    As far as I can see only some of the formats defined in the enumeration are meant for the screen buffer and they are simple variations of uncompressed rgba. It should not be difficult to write down some on the fly conversion.

    Your source code would be very helpful!
     
    Last edited by ::Phoenix::, Dec 16, 2016
  11. ::Phoenix::
    OP

    ::Phoenix:: GBAtemp Regular

    Member
    184
    170
    May 11, 2010
    Italy
    @Maschell I suppose that if I want to access the SD card on any title launch I will need iosuhax, unless I write a loader a LA loadiine for running the title inside mii maker, right?
     
  12. gudenau

    gudenau Never a unique idea

    Member
    3,257
    1,224
    Jul 7, 2010
    United States
    /dev/random
    It is not hard to access the FB, just replace the buffer call and read from there. I've done it.
     
  13. ::Phoenix::
    OP

    ::Phoenix:: GBAtemp Regular

    Member
    184
    170
    May 11, 2010
    Italy
    Thay's what I am doing right know. I am also able to save the whole raw buffer to SD. Didn't have much time to continue further. I just need to convert the raw data to PNG and move the saving process on a separate thread for not making the game stuck while saving.

    Strangely the video buffer seems scrambled, with some swizzle-like algorithm, did nt expect it for the raw video buffer. Any tips about that?
     
    Last edited by ::Phoenix::, Dec 20, 2016
  14. gudenau

    gudenau Never a unique idea

    Member
    3,257
    1,224
    Jul 7, 2010
    United States
    /dev/random
    I had a raw buffer, and PNG will take a decent amount of processing because of it's format. BMP would just require a header.
     
  15. starlander

    starlander Newbie

    Newcomer
    6
    2
    Jan 13, 2017
    United States
    Phoenix, are you still working on this project?


    Sent from my iPhone using Tapatalk
     
    I pwned U! likes this.
  16. BullyWiiPlaza

    BullyWiiPlaza Nintendo Hacking <3

    Member
    1,727
    1,409
    Aug 2, 2014
    Germany
    I also still need this for the TCP Gecko Installer... :mellow:
     
  17. ::Phoenix::
    OP

    ::Phoenix:: GBAtemp Regular

    Member
    184
    170
    May 11, 2010
    Italy
    My Job takes much of my time. Unfortunately, I am not able towork on homebrew recently. I hope I will be able to come back to at some point.
     
    I pwned U! likes this.
  18. BullyWiiPlaza

    BullyWiiPlaza Nintendo Hacking <3

    Member
    1,727
    1,409
    Aug 2, 2014
    Germany
    Alright so I did some work on the screenshot thing and I managed to hook the GX2CopyColorBufferToScanBuffer function and print to the UDP log using the following hook:
    Code:
    static volatile int executionCounter = 0;
    
    DECL(void, GX2CopyColorBufferToScanBuffer, const GX2ColorBuffer *colorBuffer, s32 scan_target) {
        if (executionCounter > 120) {
            GX2Surface surface = colorBuffer->surface;
            log_printf("GX2CopyColorBufferToScanBuffer {surface width:%d, height:%d, image size:%d, image data:%x}\n", surface.width, surface.height, surface.image_size, surface.image_data);
    
            executionCounter = 0;
        }
    
        executionCounter++;
        real_GX2CopyColorBufferToScanBuffer(colorBuffer, scan_target);
    }
    This runs the hook every 120 executions since the function is called like 120 times per second to avoid spam in the log file. It dumps out some fields from the GX2ColorBuffer's GX2Surface for information. The log file looks like this:
    Code:
    GX2CopyColorBufferToScanBuffer {surface width:1280, height:720, image size:3686400, image data:f5c7b800}
    GX2CopyColorBufferToScanBuffer {surface width:854, height:480, image size:1658880, image data:f5ae6000}
    GX2CopyColorBufferToScanBuffer {surface width:1280, height:720, image size:3686400, image data:f5c7b800}
    GX2CopyColorBufferToScanBuffer {surface width:854, height:480, image size:1658880, image data:f5ae6000}
    ...
    As you can see, the entries oscillate between two image resolutions and the image size/image data address stays the same. I went ahead and dumped the image data using JGecko U but it mainly contains 0xFF and 0x00 bytes which doesn't seem useful. How would you go and convert that to an actual JPEG or PNG now so you have a viewable screenshot on your computer?
     
  19. MarcusD

    MarcusD nobody's hero

    Member
    1,169
    2,298
    Oct 16, 2015
    Hungary
    a pixel between life and death
    it must be raw image data. you must first get the framebuffer mode (you can guess the pixel size with (image_size / (width * height))) then you can use ImageMagick or a handwritten program to make a png for you from it.