Hacking [Development] Many, many memory questions

[QuarkTheAwesome]

Hey all!

So I've been poking around in the documentation a lot recently and I've ended up with quite a few questions surrounding memory where everything's a but unclear and it seems we've only got vague forum posts to go off. So, I figured I'd throw a list at you all and we'll see how many we can answer ;3
Everything's in relation to 5.5.1 userspace.
Well, here we go!

1. Exactly how much memory is available to us in userspace? How does the ELF loader affect this?
2. If the amount of RAM we have is so limited, why can programs like LiteNES declare variables (check out the rom variable) that are so much bigger than what we're meant to have (something like 32MiB from what I read?)
   
   (Q1/2) General consensus so far: There aren't any hard limits defined, just try it and see what sticks
   Some of my preliminary investigations seem to point to us having something like 441MB of MEM2 available, please see this post
   In regards to the ELF loader; see here
   I'm also working on a program that should be able to get us 160ish MB of MEM2. Watch this space.
   
3. Do we get access to MEM2? The Internet Browser has RAMPID 4, so don't we have it aswell?
   
   (Q3) So far, the answer is yes. Use OSAllocFromSystem or set up a memory heap.
   
   
4. If I just declare a variable with no special allocations or whatever, where does that get put in relation to MEM1/MEM2 etc?
   
   (Q4) Prelim: MEM1.
   
   
5. How do you use OSAllocFromSystem? What's up with the align variable?
   
   (Q5) See this post.
   
   
6. Where does OSAllocFromSystem get its RAM from (MEM1/MEM2 etc)?
   
   (Q6) MEM2. Also look at using a heap instead.
   
   
7. Why should I use OSAllocFromSystem over just a variable declaration?
   
   (Q7) Depends on the variable, normal variable declarations go into MEM1 (quick but small) while proper allocations go into MEM2 (slower but huge)
   
   
8. Could I theoretically mess with all RAM through poking at its address?
9. Why can't I poke at RAM directly even if it's in a section that I am meant to have access to?
10. What's up with all the other memory functions? How and why do they work?

I'm sorry for the list of noob questions but I really need an answer. Thanks!

EDIT: The discussion below has kinda shifted off the topic of these memory questions and onto far more interesting things around 5.5.x homebrew. Here's a funky list.

Spoiler: Discussion Summary

Keep in mind that I've stripped out all the technical discussion. X means either 1 or 2 in relation to sndcore libraries.

• Me: We actually have a whole bunch of RAM available [post]
• Dimok: That's not really what matters, the issues surrounding sndcoreX and gx2 are because the browser's already using them [post]
• Me: Since the browser is killed at the start of most homebrew, couldn't we find its old library handles and release them, thus giving us access to said libraries? [post]
• NWPlayer123: Sound1 and Sound2 can't be used at the same time, likely why everyone's having Sound2 issues. Also TCPGecko is far better than RPC since the browser is still running. [post]
• Dimok: Yeah, I suppose releasing the old libs could work. The issue is that the ones we care about have special shutdown functions that MUST be called, and it's unlikely the browser is actually calling them. [post]
• brienj: Since our homebrew is launched through the video player, could that be interfering with soundX? [post]
• Me/NWPlayer123: Probably.
• NWPlayer123: Well, I took apart the code and can confirm that you need to be on the correct CPU in order to shutdown the sound. [post]
• Me: Ooh, that's cool. Where'd you get the code? [post]
• NWPlayer123: Just download it from the NUS and find a plugin for IDA. [post]
• Me: So I managed to call the sound shutdown function correctly, but I still can't make a new instance. [post]
• NWPlayer123: Are you calling it properly? [post]
• Me: Just realized I was calling the wrong sound library. Oops. [post]
• Me: No luck. [post]
• brienj: At least I know I'm not the only one ;3 [post]

 

[rw-r-r_0644]

Hey all!

So I've been poking around in the documentation a lot recently and I've ended up with quite a few questions surrounding memory where everything's a but unclear and it seems we've only got vague forum posts to go off. So, I figured I'd throw a list at you all and we'll see how many we can answer ;3
Everything's in relation to 5.5.1 userspace.
Well, here we go!

1. Exactly how much memory is available to us in userspace? How does the ELF loader affect this?
2. If the amount of RAM we have is so limited, why can programs like LiteNES declare variables (check out the rom variable) that are so much bigger than what we're meant to have (something like 32MiB from what I read?)
   
3. Do we get access to MEM2? The Internet Browser has RAMPID 4, so don't we have it aswell?
4. If I just declare a variable with no special allocations or whatever, where does that get put in relation to MEM1/MEM2 etc?
5. How do you use OSAllocFromSystem? What's up with the align variable?
6. Where does OSAllocFromSystem get its RAM from (MEM1/MEM2 etc)?
7. Why should I use OSAllocFromSystem over just a variable declaration?
   
8. Could I theoretically mess with all RAM through poking at its address?
9. Why can't I poke at RAM directly even if it's in a section that I am meant to have access to?
10. What's up with all the other memory functions? How and why do they work?

I'm sorry for the list of noob questions but I really need an answer. Thanks!

For the first questions, from what i have thought (but I may be wrong) we have 32MB for the code of the program and an unspecified RAM limit (so maybe there's a part of ram of 32Mb where the code is loaded and a part of ram for the variables)

My opinion is that if there is a RAM limit, it won't be so small because the browser has also to execute php and other scripts with number and size of variables that change on each script

Not sure thought, I would like to know that too

 

[brienj]

Hey all!

So I've been poking around in the documentation a lot recently and I've ended up with quite a few questions surrounding memory where everything's a but unclear and it seems we've only got vague forum posts to go off. So, I figured I'd throw a list at you all and we'll see how many we can answer ;3
Everything's in relation to 5.5.1 userspace.
Well, here we go!

1. Exactly how much memory is available to us in userspace? How does the ELF loader affect this?
2. If the amount of RAM we have is so limited, why can programs like LiteNES declare variables (check out the rom variable) that are so much bigger than what we're meant to have (something like 32MiB from what I read?)
   
3. Do we get access to MEM2? The Internet Browser has RAMPID 4, so don't we have it aswell?
4. If I just declare a variable with no special allocations or whatever, where does that get put in relation to MEM1/MEM2 etc?
5. How do you use OSAllocFromSystem? What's up with the align variable?
6. Where does OSAllocFromSystem get its RAM from (MEM1/MEM2 etc)?
7. Why should I use OSAllocFromSystem over just a variable declaration?
   
8. Could I theoretically mess with all RAM through poking at its address?
9. Why can't I poke at RAM directly even if it's in a section that I am meant to have access to?
10. What's up with all the other memory functions? How and why do they work?

I'm sorry for the list of noob questions but I really need an answer. Thanks!

I've found better luck by declaring a huge array from the start, than to try and allocate it later. I would assume that the RAM would be fairly big, considering the Internet Browser can stream video. I don't know for sure though, and if it doesn't work one way, I try another until it works, or just determine that it probably isn't going to happen. I wish there were hard limits defined, because you can't compare the specs of a game running to using the area for the Internet Browser you are locked into, so reading the documentation hasn't been much help.

 

[QuarkTheAwesome]

Answered 3, 6 and 7 on my own: OSAllocFromSystem seems to get its RAM from MEM2. This is kinda important as it gives me a proper idea of how much RAM I have (lots!)

What I did: (Please post if I've done anything wrong or misinterpreted anything!)

Code ran through the 5.5.x ELF loader.

    int testvar = 123;
    char print[255];
    __os_snprintf(print, 255, "Address of testvar: %X", &testvar);
    printstr(1, print);
 
    void* allocated;
    allocated = OSAllocFromSystem(0xFF, 64);
    __os_snprintf(print, 255, "Address of allocated: %X", allocated);
    printstr(2, print);
 
    __os_snprintf(print, 255, "Address of fancy allocated: %X", &allocated);
    printstr(3, print);
 
    while(1) {printstr(4, "All done and good!");}

This returns:

Address of testvar: F4C07EE8 //in MEM1, q4 answered
Address of allocated: 106DB680 //!!!!!!
Address of fancy allocated: F4C07FEC //Just the address of the pointer to allocated. Not sure what I was thinking with this one.
All done and good!

According to WiiUBrew, the area between 0x10000000 and 0x50000000 is mapped to MEM2 [source] and for background apps (us/Internet Browser, RAMPID 4) that should be 0x1C000000 big [source].
This gives me between 0x10000000 and 0x2C000000 as my userspace RAM. Since the first variable I allocated was not at 0x10000000 but a bit further along, I can deduce that on an average launch, I'm likely to have a bit more than 0x1B000000 available (0x1B924980 in this case) to me through OSAllocFromSystem.
That comes out at around 441MB (that may not sound like much, but that's only ~100MB less than the major (non-Nintendo) last-gen consoles). With further hacking, we'd probably be able to reuse the RAM before that 1B spot, up until a potential 469MB.

This means that we actually have a pretty decent amount of RAM available to userspace!

Please, please let me know if I'm wrong. If I'm not, we may have just stumbled upon something pretty big.

Update: Just ran the program a few more times and it keeps coming back with the same address (106DB680). Looks like whatever the browser was doing before we loaded has no effect.

 

[QuarkTheAwesome]

Just one more thing: I just allocated a few more variables and the address of each seems to be increasing by 0x140 each time rather than 0xFF (the size of what I allocated). I suspect that the printing code may be leaky or that it's something to do with the align variable on OSAllocFromSystem (which I still don't really understand ;3)

If not, I should probably delete the lot of this. Unfortunately I can't really check right now (time for sleep ) but I will double-check as soon as I can.

 

[eliboa]

How do you use OSAllocFromSystem? What's up with the align variable?!

"Data structure alignment is the way data is arranged and accessed in computer memory. It consists of two separate but related issues: data alignment and data structure padding. When a modern computer reads from or writes to a memory address, it will do this in word sized chunks (e.g. 4 byte chunks on a 32-bit system) or larger. Data alignment means putting the data at a memory address equal to some multiple of the word size, which increases the system's performance due to the way the CPU handles memory"
https://en.wikipedia.org/wiki/Data_structure_alignment

Why should I use OSAllocFromSystem over just a variable declaration?

You don't need to allocate memory for a simple variable. You need allocation for arrays or bigger structures i guess. Or for dynamic allocation.

Btw, you can also use MEMAllocFromDefaultHeapEx(unsigned int size, int align)

 

[NWPlayer123]

Okay, so basically, normal applications get 1GB of memory in MEM2 for both code and data, code area is at 0x02000000 to 0x10000000 (defined per application, but since we have to use the browser's JIT area at 0x01800000 to I think 0x02000000) and data is in 0x10000000 to 0x42000000 (1GB is 0x40000000 plus our starting offset of 0x02). Since we're hijacking the current application, I'm not sure how we get access to the 0x10000000 area, I ran MEMGetBaseHeapHandle and it gave me an address in there that in my memdump is an EXPH (Expanded Heap) header, then theoretically you should be able to use that handle to allocate memory. Every foreground application gets access to MEM2, and foreground applications also get access to MEM1 which is 32MB of super-fast memory at 0xF40000000, should be used for "high use" data. The foreground also has its own area at 0xE0000000, and it's 40MB (to 0xE4000000).
Basically,
0x01000000 to 0x01800000 - used by the system for OS modules/libraries
0x01800000 to 0x02000000(?) - Just In Time area used by the browser
0x02000000 to 0x10000000 - theoretically code for RPX applications but that we can't use without kernel
0x10000000 to 0x42000000(? - depends, can go to 0x50000000)
0xA0000000 to 0xD2000000(? - depends, can go to 0xE0000000) - mirror used by OSCodegenCopy which apparently the browser can't access
0xE0000000 to 0xE4000000 - Foreground area
0xF4000000 to 0xF6000000 - 32MB for MEM1
0xF6000000 to 0xFFFFFFFF - used by the system

I'm not at all sure how RPX applications are set to allocate, from what I can tell it depends, since ghs has a malloc builtin for Wii U, and when they're built, each section goes where it's supposed to, .text starts at 0x02000000, .rodata is at 0x10000000 (not sure if the loader modifies that offset since all my memdumps have library strings at the very start of it), .data is right after, .bss is right after that, and then all remaining memory can be allocated somehow. I assume you get a base heap handle like I tried, and then you can allocate a large chunk, then make a specialized heap (there are four available, expanded heap, frame heap, block heap, and unit heap).
Unit heap is used for allocating in predecided chunks (EG give it a giant amount and tell it to allocate in units of 32KB, each time you make a new allocation it'll just allocate 32KB directly, doesn't need a size param)
Expanded heap is used for general purpose stuff, its purpose is to work like malloc, lets you freely allocate memory. If you pass a negative alignment to MEMAllocFromExpHeapEx, it'll allocate it at the end of the expanded heap, so you can copy in compressed data and uncompress it into the start of the heap so you have less fragmentation.
Frame heap and block heap I'm not sure about :\

Here's a list of all available functions sorted into classes (you'll need to document what parameters they need tho, I started on that but lost my notes :\)

Spoiler

=== Expanded Heap ===
MEMAdjustExpHeap
MEMAllocFromExpHeapEx
MEMCheckExpHeap
MEMCheckForMBlockExpHeap
MEMCreateExpHeapEx
MEMDestroyExpHeap
MEMFreeToExpHeap
MEMGetAllocDirForMBlockExpHeap
MEMGetAllocModeForExpHeap
MEMGetAllocatableSizeForExpHeapEx
MEMGetGroupIDForExpHeap
MEMGetGroupIDForMBlockExpHeap
MEMGetSizeForMBlockExpHeap
MEMGetTotalFreeSizeForExpHeap
MEMInitAllocatorForExpHeap
MEMResizeForMBlockExpHeap
MEMSetAllocModeForExpHeap
MEMSetGroupIDForExpHeap
MEMUseMarginOfAlignmentForExpHeap
MEMVisitAllocatedForExpHeap
MEMiIsEmptyExpHeap

=== Block Heap ===
MEMAddBlockHeapTracking
MEMAllocFromBlockHeapAt
MEMAllocFromBlockHeapEx
MEMDestroyBlockHeap
MEMFreeToBlockHeap
MEMGetAllocatableSizeForBlockHeapEx
MEMGetTotalFreeSizeForBlockHeap
MEMGetTrackingLeftInBlockHeap
MEMInitAllocatorForBlockHeap
MEMInitBlockHeap

=== Frame Heap ===
MEMAdjustFrmHeap
MEMAllocFromFrmHeapEx
MEMCreateFrmHeapEx
MEMDestroyFrmHeap
MEMFreeByStateToFrmHeap
MEMFreeToFrmHeap
MEMGetAllocatableSizeForFrmHeapEx
MEMInitAllocatorForFrmHeap
MEMRecordStateForFrmHeap
MEMResizeForMBlockFrmHeap
MEMiGetFreeEndForFrmHeap
MEMiGetFreeStartForFrmHeap

=== Unit Heap ===
MEMAllocFromUnitHeap
MEMCalcHeapSizeForUnitHeap
MEMCountFreeBlockForUnitHeap
MEMCreateUnitHeapEx
MEMDestroyUnitHeap
MEMFreeToUnitHeap
MEMInitAllocatorForUnitHeap

=== Memory Lists ===
MEMAppendListObject
MEMGetNextListObject
MEMGetNthListObject
MEMGetPrevListObject
MEMInitList
MEMInsertListObject
MEMPrependListObject
MEMRemoveListObject

=== Misc. functions ===
MEMAllocFromAllocator
MEMCheckHeap
MEMCreateUserHeapHandle
MEMDumpHeap
MEMFindContainHeap
MEMFindParentHeap
MEMFreeToAllocator
MEMGetArena
MEMGetBaseHeapHandle
MEMGetFillValForHeap
MEMInitAllocatorForDefaultHeap
MEMSetBaseHeapHandle
MEMSetFillValForHeap

This took a lot longer to make than I was expecting

 

[QuarkTheAwesome]

Here's a list of all available functions sorted into classes (you'll need to document what parameters they need tho, I started on that but lost my notes :\)

Spoiler

=== Expanded Heap ===
MEMAdjustExpHeap
MEMAllocFromExpHeapEx
MEMCheckExpHeap
MEMCheckForMBlockExpHeap
MEMCreateExpHeapEx
MEMDestroyExpHeap
MEMFreeToExpHeap
MEMGetAllocDirForMBlockExpHeap
MEMGetAllocModeForExpHeap
MEMGetAllocatableSizeForExpHeapEx
MEMGetGroupIDForExpHeap
MEMGetGroupIDForMBlockExpHeap
MEMGetSizeForMBlockExpHeap
MEMGetTotalFreeSizeForExpHeap
MEMInitAllocatorForExpHeap
MEMResizeForMBlockExpHeap
MEMSetAllocModeForExpHeap
MEMSetGroupIDForExpHeap
MEMUseMarginOfAlignmentForExpHeap
MEMVisitAllocatedForExpHeap
MEMiIsEmptyExpHeap

=== Block Heap ===
MEMAddBlockHeapTracking
MEMAllocFromBlockHeapAt
MEMAllocFromBlockHeapEx
MEMDestroyBlockHeap
MEMFreeToBlockHeap
MEMGetAllocatableSizeForBlockHeapEx
MEMGetTotalFreeSizeForBlockHeap
MEMGetTrackingLeftInBlockHeap
MEMInitAllocatorForBlockHeap
MEMInitBlockHeap

=== Frame Heap ===
MEMAdjustFrmHeap
MEMAllocFromFrmHeapEx
MEMCreateFrmHeapEx
MEMDestroyFrmHeap
MEMFreeByStateToFrmHeap
MEMFreeToFrmHeap
MEMGetAllocatableSizeForFrmHeapEx
MEMInitAllocatorForFrmHeap
MEMRecordStateForFrmHeap
MEMResizeForMBlockFrmHeap
MEMiGetFreeEndForFrmHeap
MEMiGetFreeStartForFrmHeap

=== Unit Heap ===
MEMAllocFromUnitHeap
MEMCalcHeapSizeForUnitHeap
MEMCountFreeBlockForUnitHeap
MEMCreateUnitHeapEx
MEMDestroyUnitHeap
MEMFreeToUnitHeap
MEMInitAllocatorForUnitHeap

=== Memory Lists ===
MEMAppendListObject
MEMGetNextListObject
MEMGetNthListObject
MEMGetPrevListObject
MEMInitList
MEMInsertListObject
MEMPrependListObject
MEMRemoveListObject

=== Misc. functions ===
MEMAllocFromAllocator
MEMCheckHeap
MEMCreateUserHeapHandle
MEMDumpHeap
MEMFindContainHeap
MEMFindParentHeap
MEMFreeToAllocator
MEMGetArena
MEMGetBaseHeapHandle
MEMGetFillValForHeap
MEMInitAllocatorForDefaultHeap
MEMSetBaseHeapHandle
MEMSetFillValForHeap

Dimok has a bunch of files on most of his repos that contain basically all functions (useful for getting signatures). Unfortunately it doesn't have all the memory functions, but it's a start. (Link)

 

[dimok]

Hey the problem on this whole thing is not how much heap memory you have that you can still use. All the heap memory is not very usefull if you can't get the hardware to play with it.

As NWPlayer123 already pointed out the whole problem on the 5.5.1 exploit is that it hijacks a running environment. That environment already has gained several hardware accesses such as GX2 or the sndcore which are one time run only and uses them blocking you off from using them. brienj already ran into that problem with sndcore too. The problem here is not that it is userspace. Almost all my application run in userspace. The difference is that they are hooked into the very beginning of the hijacked application (e.g. MiiMaker). You could say they hook the "main" function (I know it is not 100% accurate). At that position you still have everything untouched. But to hook yourself at that position you need to modify the coreinit code which requires write access to the code area from userspace and that write access needs to be first setup in the kernel. Since unfortunately there is no kernel exploit available for public for your firmware version you can't do it.

Anyway here is one addition to NWPlayer123's text (which already explains it in theory) on how you could setup yourself a memory area:
https://github.com/dimok789/homebrew_launcher/blob/master/src/system/memory.c#L53

You can fragment that area into several parts by allocating less than the maximum available memory too and split the memory up. In that code part I gain the full MEM1 range for GX2 color-, depth-, hi-Z- and auxilary-buffers (AA only) as that is very fast memory and bucket memory for the scanbuffers. Additionally you have the full MEM2 memory (default heap) for your usage. The size depends on how much is setup for the application (example mii maker) which usually is around 1GB. RPX applications sometimes gain full default heap memory on start and setup own heap handles like I did for MEM1 and bucket memory and then they dont use the default heap handle anymore. Well this all works if you start from a fresh and good entry point (HBL offers that to the homebrews) but I can't tell you if and how good that will work from the browser exploit environment. It's really too bad that some developers still have to fight with that environment.

 

[brienj]

I had one question too, and may as well throw it in here, as it's kind of related to memory. I was playing around with the RPC, and I was dumping the memory from the framebuffer, which I figured I could open in Photoshop as a raw image, but I noticed that when dumping it, instead of dumping the memory as RGBA, it was dumping it as RRRRGGGGBBBBAAAA, repeating each byte of color four times, which made for an interesting looking picture. I haven't messed around much more with it, but does anyone know why it is doing that for the framebuffer, but it dumps all other memory just fine?

 

[QuarkTheAwesome]

As NWPlayer123 already pointed out the whole problem on the 5.5.1 exploit is that it hijacks a running environment. That environment already has gained several hardware accesses such as GX2 or the sndcore which are one time run only and uses them blocking you off from using them.

I'm interpreting that as "the Internet browser links with GX2 and sndcore, and you can't use them at the same time as the browser." If that's the case, then shouldn't we be able to release the libraries? For instance, at the start of almost every userspace (when I say userspace, I'm referring to 5.5.x browserhax) homebrew out there there's a few lines of code involving IM_ methods (IM_Open, IM_Close and IM_SetDeviceState) that, as far as I know, kill the browser/restart the system? (no documentation, kinda guessing based off comments in the libwiiu examples) If that's not what it's doing, sorry. I'm sure we could come up with code to kill the browser properly anyway.
Back on topic, if we've killed the browser, then it's not using those libraries anymore. If we can get the handles it had allocated (probably still in RAM somewhere) and release them (or just reuse them, whatever) we should, theoretically, get proper lib access.
Of course, to do that, we need to know when and where those handles are allocated. And that is where RPC comes in.

I'll be messing around with my WiiU and I'll try to get some info out of it, but I'd appreciate it if you guys have a poke around too. It'd be really useful.

(I'm also thinking of maybe moving this discussion to a new thread (we've kinda gotten off the original topic on this one). Opinions? Or should we just keep rambling here?)

 

[NWPlayer123]

I'm interpreting that as "the Internet browser links with GX2 and sndcore, and you can't use them at the same time as the browser." If that's the case, then shouldn't we be able to release the libraries? For instance, at the start of almost every userspace (when I say userspace, I'm referring to 5.5.x browserhax) homebrew out there there's a few lines of code involving IM_ methods (IM_Open, IM_Close and IM_SetDeviceState) that, as far as I know, kill the browser/restart the system? (no documentation, kinda guessing based off comments in the libwiiu examples) If that's not what it's doing, sorry. I'm sure we could come up with code to kill the browser properly anyway.
Back on topic, if we've killed the browser, then it's not using those libraries anymore. If we can get the handles it had allocated (probably still in RAM somewhere) and release them (or just reuse them, whatever) we should, theoretically, get proper lib access.
Of course, to do that, we need to know when and where those handles are allocated. And that is where RPC comes in.

I'll be messing around with my WiiU and I'll try to get some info out of it, but I'd appreciate it if you guys have a poke around too. It'd be really useful.

(I'm also thinking of maybe moving this discussion to a new thread (we've kinda gotten off the original topic on this one). Opinions? Or should we just keep rambling here?)

Eh, here's fine. I wish I knew what IM_Close did, AFAIK it'll shut down all running processes and threads, not sure if it frees the libraries it's using, though. I poked around with sound a while ago with TCPGecko (better than RPC since browser's still running, all apps need IM_Close to get OSScreen to function properly), which is how I found out it's still using the old Sound-1 libraries. I think I remember testing that Sound-1 and Sound-2 can't be acquired at the same time.

 

[QuarkTheAwesome]

I think I remember testing that Sound-1 and Sound-2 can't be acquired at the same time.

That explains why everyone (including me) can't get Sound2 working - the browser's Sound1 hasn't been released yet. (the issues with Sound1 are probably just RAM-related)

(sigh) I wish I had TCP Gecko..... (single tear)

 

[dimok]

I had one question too, and may as well throw it in here, as it's kind of related to memory. I was playing around with the RPC, and I was dumping the memory from the framebuffer, which I figured I could open in Photoshop as a raw image, but I noticed that when dumping it, instead of dumping the memory as RGBA, it was dumping it as RRRRGGGGBBBBAAAA, repeating each byte of color four times, which made for an interesting looking picture. I haven't messed around much more with it, but does anyone know why it is doing that for the framebuffer, but it dumps all other memory just fine?

Interessting. I never actually took a look at the framebuffer internals. My first though was 4x MSAA as every pixel is 4 times in it but then I remembered that cant be it since MSAA is done completely in the colorbuffer and auxilary buffer you have to allocate. You could write yourself a dump of every 4th byte i guess to get a proper image but I don't know the reason why it is 4 times in there at this point.

I'm interpreting that as "the Internet browser links with GX2 and sndcore, and you can't use them at the same time as the browser." If that's the case, then shouldn't we be able to release the libraries? For instance, at the start of almost every userspace (when I say userspace, I'm referring to 5.5.x browserhax) homebrew out there there's a few lines of code involving IM_ methods (IM_Open, IM_Close and IM_SetDeviceState) that, as far as I know, kill the browser/restart the system? (no documentation, kinda guessing based off comments in the libwiiu examples) If that's not what it's doing, sorry. I'm sure we could come up with code to kill the browser properly anyway.
Back on topic, if we've killed the browser, then it's not using those libraries anymore. If we can get the handles it had allocated (probably still in RAM somewhere) and release them (or just reuse them, whatever) we should, theoretically, get proper lib access.
Of course, to do that, we need to know when and where those handles are allocated. And that is where RPC comes in.

I'll be messing around with my WiiU and I'll try to get some info out of it, but I'd appreciate it if you guys have a poke around too. It'd be really useful.

(I'm also thinking of maybe moving this discussion to a new thread (we've kinda gotten off the original topic on this one). Opinions? Or should we just keep rambling here?)

Yeah you might have a success if you shutdown all the subsystems correctly. The IM_ methods that you are refering to shutdown all running threads from what I understand. I don't know exactly what it does since it has a call to IOS_Ioctlv and its IOSU code that does some action from that point on but it seems like an event that tells every process to shutdown and whether they do it is up to the implementation of the application i guess. This function is called with parameter 4 (not 3 as on homebrew codes) from OSShutdown btw.

I think the problem is that you need to call the correct shutdown functions. For example you need to call GX2Shutdown for GX2 and AXQuit for the sndcore. Problem is that you have to call them from the same core and maybe even from the same thread that the init function of those were called from. For example for GX2 you have to do every call to almost all GX2 functions from the same thread as it was initiated with and definately the same core as those functions are not thread safe at all. So maybe you are lucky and if you guess the correct core you might be able to stop AX and GX2 correctly. Additionally you might also have to call functions like AXRegisterFrameCallback with a NULL pointer to remove already registered callback function pointers from the browser and if needed register your own functions. Anyway I dont think the issue is not that the browser has acquired or linked the libraries as that is only a matter of memory but that it already started the usage of those and that before re-using them you need to shutdown everything of it properly. Theoretically it might be possible. To the last point of NWPlayer123, I am not sure if you cant acquire sound1 and sound2 at the same time (from library point of view), I never tried that. You could test it and if you get NULL pointer functions it doesnt work but if you get correct function pointers from the find export function for both libraries then the only problem is probably that sound1 has to be shutdown properly first to use sound2. Let's just hope you don't have to free all voices correct before the shutdown as that would be a pain in the ass to find the correct pointers.

 

[brienj]

Since you start a "video" to initiate the hack, could that be causing the issues with audio too? The "video" basically crashes the video player, right? And calling AXQuit to shutdown the audio core would lock up the console, it was one thing I thought to try. I could only call it for sndcore too, because as I said before, getting a handle to sndcore2 resulted in an instant lock up.

 

[NWPlayer123]

Since you start a "video" to initiate the hack, could that be causing the issues with audio too? The "video" basically crashes the video player, right? And calling AXQuit to shutdown the audio core would lock up the console, it was one thing I thought to try. I could only call it for sndcore too, because as I said before, getting a handle to sndcore2 resulted in an instant lock up.

Yes, I was going to post that you could just not run IM_Close when running RPC to get the same effect but it'll freeze, you HAVE to run IM_Close or it'll crash :\ You gotta somehow shutdown all threads with it and then reacquire handles and shut everything down, some stuff you don't need to even call since they're deprecated

 

[QuarkTheAwesome]

Yes, I was going to post that you could just not run IM_Close when running RPC to get the same effect but it'll freeze, you HAVE to run IM_Close or it'll crash :\ You gotta somehow shutdown all threads with it and then reacquire handles and shut everything down, some stuff you don't need to even call since they're deprecated

Funnily enough, the RPC system from libwiiu doesn't call IM_Close at all. This, of course, doesn't work on 5.5 but it's interesting that they didn't feel the need to add it in.

 

[QuarkTheAwesome]

Since you start a "video" to initiate the hack, could that be causing the issues with audio too? The "video" basically crashes the video player, right? And calling AXQuit to shutdown the audio core would lock up the console, it was one thing I thought to try. I could only call it for sndcore too, because as I said before, getting a handle to sndcore2 resulted in an instant lock up.

The Sound2 issue is probably because of what NWPlayer123 mentioned above (you can't use Sound1 and Sound2 at the same time, and the browser technically hasn't stopped using Sound1) and the Sound1 issue is likely because you're not on the same core the browser was (or as Dimok said, not even being on the same thread could cause an issue.) We could (also, as Dimok hinted at), theoretically, guess at which core the browser was on (I suspect it'd probably be the same each time anyway), start a thread on that core, and call AXQuit to free up Sound1 for our main thread. That is dodging the possible issue of being on the wrong thread, however. It also doesn't do much about how the WiiU could recover from an exception (us guessing wrong (if the core changes each time.)) It just needs a lot of investigation, that's all.

 

[NWPlayer123]

The Sound2 issue is probably because of what NWPlayer123 mentioned above (you can't use Sound1 and Sound2 at the same time, and the browser technically hasn't stopped using Sound1) and the Sound1 issue is likely because you're not on the same core the browser was (or as Dimok said, not even being on the same thread could cause an issue.) We could (also, as Dimok hinted at), theoretically, guess at which core the browser was on (I suspect it'd probably be the same each time anyway), start a thread on that core, and call AXQuit to free up Sound1 for our main thread. That is dodging the possible issue of being on the wrong thread, however. It also doesn't do much about how the WiiU could recover from an exception (us guessing wrong (if the core changes each time.)) It just needs a lot of investigation, that's all.

Apparently AXQuit just needs the same core which we can find with a variable (memdump, find the code and get the "real" address since data doesn't start right at 0x10000000, loader translates it). From what I can tell, this is all we need to do before getting the handle and calling OSDynLoad_Release, but the way that works is it keeps track of the number of references/acquires and only deallocates once it hits zero so we'll have to do some fancy loop with OSDynLoad_IsModuleLoaded

Spoiler: Assembly for snd_core

 

[QuarkTheAwesome]

Apparently AXQuit just needs the same core which we can find with a variable (memdump, find the code and get the "real" address since data doesn't start right at 0x10000000, loader translates it). From what I can tell, this is all we need to do before getting the handle and calling OSDynLoad_Release, but the way that works is it keeps track of the number of references/acquires and only deallocates once it hits zero so we'll have to do some fancy loop with OSDynLoad_IsModuleLoaded

Spoiler: Assembly for snd_core

View attachment 42983

Where'd you get the assembly? That's pretty darn useful to have.
It definitely wouldn't be that hard to do something like that. One quick question though: How do you figure out the function signatures and the like for stuff that isn't in wiiubrew or anything like that?