Hacking [Development] Using the GX2 engine under Browserhax?

[QuarkTheAwesome]

Hey all!

So I've been trying to get GX2 (the graphics engine, not loadiine!) running under 5.5 browserhax and, well, I've gotten stuck and need some help F:

I've been following Dimok's coloured cube example basically to a T. With the help of my funky memory library thing with a terrible name (shameless link) I've successfully allocated a command buffer (0x400000 big at 0x102CF780), ran GX2Init, gotten the current scanmode, calculated the size of the TV's buffer (0xFF0000 @ 1080p) and even allocated it from bucket memory (at 0xE0001000.) However, when I try to run GX2SetTvBuffer it crashes. Every time. I can't figure out why

It's ran like so:

GX2SetTvBuffer(tvScanBuffer, scanBufferSize, tvRenderMode, GX2_SURFACE_FORMAT_TCS_R8_G8_B8_A8_UNORM, GX2_BUFFERING_DOUBLE);

Where tvScanBuffer is an unsigned char pointer (0xE0001000, allocated as 0xFF0000 big), scanBufferSize is an unsigned int set to 0xFF0000, and tvRenderMode is set to GX2_TV_RENDER_1080 (0x5). All the other constants are exactly the same as the coloured cube example. It's also worth noting that I have OSScreen up and running to help with logging (but removing it doesn't seem to affect anything).

I have an exception handler installed so I can tell you all the registers if required, but I'd rather not type them all out.

Anyway, I hope you guys have some ideas. As I said, I'm a bit stuck.

 

[Exzap]

Your memory library trashes all existing heaps, chances are you kill one that is needed by GX2.

You could try to set a custom memory allocator via GX2SetDefaultAllocator() - Although I don't think GX2SetTvBuffer() uses it.

void GX2SetDefaultAllocator(void* (*GX2DefaultAllocateFunc)(uint32_t userParameter, uint32_t size, sint32_t alignment), void (*GX2DefaultFreeFunc)(uint32_t userParameter, void* mem));

 

[QuarkTheAwesome]

Your memory library trashes all existing heaps, chances are you kill one that is needed by GX2.

You could try to set a custom memory allocator via GX2SetDefaultAllocator() - Although I don't think GX2SetTvBuffer() uses it.

Just tried that - didn't change anything. As you suspected, it never gets called.

 

[dimok]

Hmm I am not sure if your understanding of userspace and kernel space is correct because you use the term "userspace" for all browser exploit applications and "kernel space" for all applications that require some kernel pre-configuration before the start. That would be not correct and is really confusing. Every application I did so far for the WiiU was a userspace application (loadiine gx2 is an exception a bit because it has kernel space functions too but only 2 or 3).

Now to your problem:
Do you stop GX2 of browser with GX2Shutdown() before you launch your GX2Init()? You can get the main core of GX2 with the following function "int GX2GetMainCoreId(void)". As already said you have to stop GX2 on the correct core before you can re-use it.

Anyway just to make things clear again:
I have to repeat what I said already before but the issue is always the same. Your entrypoint is basically not going to work with most of the hardware functions. You are hijacking an already running application which has most of the hardware initialized for custom usage and already uses them. Additionally to that the exploit environment is even more "special". For example you cant even create a simple callback function for core 0 or core 2 threads to run because your code is in a memory area where core 0 and core 2 dont have access to execute stuff. You can only use existing code in RAM from other stuff (e.g. coreinit) to run on those cores using a ROP chain. I am not even sure if the FPU is enabled at that state of environment you are in (I didnt try it but you will definately need it for GX2). If you really want to do some decent homebrew application with GX2 graphics and other nice features, you should talk to MN1 and ask him to give you the exploit. Then you can create a proper hook at the beginning of some application (int main(int argc, char *argv[])) and have a proper userspace entrypoint. This is what HBL and other application, that can be launched through it, do. They hook the main() call of a title. This gives you a proper and clean environment to do your own handlings of all hardware and memory. It won't get much better even with IOSU exploit, except that you could get additional hardware access such as USB storage for example or SD card access everywhere or even NAND access. Doing investigation on the GX2 with the browser exploit as you try is a waste of time in my opinion and will be obsolete as soon as there is a public kernel exploit because with a proper entry point you don't have all those issues from the browser exploit and you dont have to do hardcoded ram addresses or hijack the bucket heap handle to get some free memory space in it. It just works as intended to.

 

[QuarkTheAwesome]

Hmm I am not sure if your understanding of userspace and kernel space is correct because you use the term "userspace" for all browser exploit applications and "kernel space" for all applications that require some kernel pre-configuration before the start. That would be not correct and is really confusing. Every application I did so far for the WiiU was a userspace application (loadiine gx2 is an exception a bit because it has kernel space functions too but only 2 or 3).

Now to your problem:
Do you stop GX2 of browser with GX2Shutdown() before you launch your GX2Init()? You can get the main core of GX2 with the following function "int GX2GetMainCoreId(void)". As already said you have to stop GX2 on the correct core before you can re-use it.

Anyway just to make things clear again:
I have to repeat what I said already before but the issue is always the same. Your entrypoint is basically not going to work with most of the hardware functions. You are hijacking an already running application which has most of the hardware initialized for custom usage and already uses them. Additionally to that the exploit environment is even more "special". For example you cant even create a simple callback function for core 0 or core 2 threads to run because your code is in a memory area where core 0 and core 2 dont have access to execute stuff. You can only use existing code in RAM from other stuff (e.g. coreinit) to run on those cores using a ROP chain. I am not even sure if the FPU is enabled at that state of environment you are in (I didnt try it but you will definately need it for GX2). If you really want to do some decent homebrew application with GX2 graphics and other nice features, you should talk to MN1 and ask him to give you the exploit. Then you can create a proper hook at the beginning of some application (int main(int argc, char *argv[])) and have a proper userspace entrypoint. This is what HBL and other application, that can be launched through it, do. They hook the main() call of a title. This gives you a proper and clean environment to do your own handlings of all hardware and memory. It won't get much better even with IOSU exploit, except that you could get additional hardware access such as USB storage for example or SD card access everywhere or even NAND access. Doing investigation on the GX2 with the browser exploit as you try is a waste of time in my opinion and will be obsolete as soon as there is a public kernel exploit because with a proper entry point you don't have all those issues from the browser exploit and you dont have to do hardcoded ram addresses or hijack the bucket heap handle to get some free memory space in it. It just works as intended to.

I understand where you're coming from (and I'm sorry for making you repeat yourself) but I'll keep on trying. I don't want to have to just sit around and wait for a public kernel exploit for an indefinite and extended amount of time, especially if I've made some cool homebrew or whatever for it. Yes, I'm stubborn. Yes, it's a waste of my time, but what is a good use for it? I just want to work with the tools available now to create something that everyone can use. You don't have to bother with me if that philosophy doesn't work for you, but I do appreciate your help. ;D

(Just a quick note since this seems needed these days: This is NOT a thread on the politics of the kernel exploit, please don't start talking about it)

As for the term 'userspace', I know its true meaning but it's just what everyone seems to use for browserhax stuff. I'll try to start using the proper term.

Back on topic, it seems I was a bit silly and assumed that since GX2Init succeeded, I didn't need to shut it down. Oops ;3
I'll try that when I can.

 

[QuarkTheAwesome]

Well, I've changed the program around to follow the gx2thread example on libwiiu a little bit.

Once it calls GX2Shutdown on the correct core (0 in my case), it swaps to a new thread on core 1 where we run GX2Init, allocate memory, etc. etc. etc. We still crash at GX2SetTVBuffer though when using Coloured Cube's method of setting buffers. I tried gx2thread's buffer method and it crashes at GX2Invalidate. I suspect I'm doing it wrong though.

 

[brienj]

Well, I've changed the program around to follow the gx2thread example on libwiiu a little bit.

Once it calls GX2Shutdown on the correct core (0 in my case), it swaps to a new thread on core 1 where we run GX2Init, allocate memory, etc. etc. etc. We still crash at GX2SetTVBuffer though when using Coloured Cube's method of setting buffers. I tried gx2thread's buffer method and it crashes at GX2Invalidate. I suspect I'm doing it wrong though.

I don't want to say it isn't possible or anything, but I worked forever trying to get the AX library to work for sound and nothing would ever work, I would imagine that you'll have the same luck with GX2 unfortunately, but I hope I'm proven wrong.

 

[QuarkTheAwesome]

but I hope I'm proven wrong.

That's why I'm trying ;3