[Development] Many, many memory questions

Discussion in 'Wii U - Hacking & Backup Loaders' started by QuarkTheAwesome, Mar 17, 2016.

  1. QuarkTheAwesome
    OP

    QuarkTheAwesome Working for Hugs

    Member
    787
    1,928
    Apr 19, 2015
    Australia
    Stuck in the PowerPC
    Hey all!

    So I've been poking around in the documentation a lot recently and I've ended up with quite a few questions surrounding memory where everything's a but unclear and it seems we've only got vague forum posts to go off. So, I figured I'd throw a list at you all and we'll see how many we can answer ;3
    Everything's in relation to 5.5.1 userspace.
    Well, here we go!
    1. Exactly how much memory is available to us in userspace? How does the ELF loader affect this?
    2. If the amount of RAM we have is so limited, why can programs like LiteNES declare variables (check out the rom variable) that are so much bigger than what we're meant to have (something like 32MiB from what I read?)

      (Q1/2) General consensus so far: There aren't any hard limits defined, just try it and see what sticks
      Some of my preliminary investigations seem to point to us having something like 441MB of MEM2 available, please see this post
      In regards to the ELF loader; see here
      I'm also working on a program that should be able to get us 160ish MB of MEM2. Watch this space.
    3. Do we get access to MEM2? The Internet Browser has RAMPID 4, so don't we have it aswell?

      (Q3) So far, the answer is yes. Use OSAllocFromSystem or set up a memory heap.

    4. If I just declare a variable with no special allocations or whatever, where does that get put in relation to MEM1/MEM2 etc?

      (Q4) Prelim: MEM1.

    5. How do you use OSAllocFromSystem? What's up with the align variable?

      (Q5) See this post.

    6. Where does OSAllocFromSystem get its RAM from (MEM1/MEM2 etc)?

      (Q6) MEM2. Also look at using a heap instead.

    7. Why should I use OSAllocFromSystem over just a variable declaration?

      (Q7) Depends on the variable, normal variable declarations go into MEM1 (quick but small) while proper allocations go into MEM2 (slower but huge)

    8. Could I theoretically mess with all RAM through poking at its address?
    9. Why can't I poke at RAM directly even if it's in a section that I am meant to have access to?
    10. What's up with all the other memory functions? How and why do they work?
    I'm sorry for the list of noob questions but I really need an answer. Thanks!

    EDIT: The discussion below has kinda shifted off the topic of these memory questions and onto far more interesting things around 5.5.x homebrew. Here's a funky list.
    Discussion Summary
     
    Last edited by QuarkTheAwesome, Apr 7, 2016
    ajd4096 and Masterwin like this.


  2. rw-r-r_0644

    rw-r-r_0644 GBAtemp Fan

    Member
    333
    380
    Jan 13, 2016
    Italy
    For the first questions, from what i have thought (but I may be wrong) we have 32MB for the code of the program and an unspecified RAM limit (so maybe there's a part of ram of 32Mb where the code is loaded and a part of ram for the variables)

    My opinion is that if there is a RAM limit, it won't be so small because the browser has also to execute php and other scripts with number and size of variables that change on each script

    Not sure thought, I would like to know that too
     
    Last edited by rw-r-r_0644, Mar 17, 2016
  3. brienj

    brienj Trying to avoid getting cancer

    Member
    1,232
    1,703
    Jan 3, 2016
    United States
    I've found better luck by declaring a huge array from the start, than to try and allocate it later. I would assume that the RAM would be fairly big, considering the Internet Browser can stream video. I don't know for sure though, and if it doesn't work one way, I try another until it works, or just determine that it probably isn't going to happen. I wish there were hard limits defined, because you can't compare the specs of a game running to using the area for the Internet Browser you are locked into, so reading the documentation hasn't been much help.
     
  4. QuarkTheAwesome
    OP

    QuarkTheAwesome Working for Hugs

    Member
    787
    1,928
    Apr 19, 2015
    Australia
    Stuck in the PowerPC
    Answered 3, 6 and 7 on my own: OSAllocFromSystem seems to get its RAM from MEM2. This is kinda important as it gives me a proper idea of how much RAM I have (lots!)

    What I did: (Please post if I've done anything wrong or misinterpreted anything!)

    Code ran through the 5.5.x ELF loader.
    Code:
        int testvar = 123;
        char print[255];
        __os_snprintf(print, 255, "Address of testvar: %X", &testvar);
        printstr(1, print);
     
        void* allocated;
        allocated = OSAllocFromSystem(0xFF, 64);
        __os_snprintf(print, 255, "Address of allocated: %X", allocated);
        printstr(2, print);
     
        __os_snprintf(print, 255, "Address of fancy allocated: %X", &allocated);
        printstr(3, print);
     
        while(1) {printstr(4, "All done and good!");}
    This returns:

    Code:
    Address of testvar: F4C07EE8 //in MEM1, q4 answered
    Address of allocated: 106DB680 //!!!!!!
    Address of fancy allocated: F4C07FEC //Just the address of the pointer to allocated. Not sure what I was thinking with this one.
    All done and good!
    According to WiiUBrew, the area between 0x10000000 and 0x50000000 is mapped to MEM2 [source] and for background apps (us/Internet Browser, RAMPID 4) that should be 0x1C000000 big [source].
    This gives me between 0x10000000 and 0x2C000000 as my userspace RAM. Since the first variable I allocated was not at 0x10000000 but a bit further along, I can deduce that on an average launch, I'm likely to have a bit more than 0x1B000000 available (0x1B924980 in this case) to me through OSAllocFromSystem.
    That comes out at around 441MB (that may not sound like much, but that's only ~100MB less than the major (non-Nintendo) last-gen consoles). With further hacking, we'd probably be able to reuse the RAM before that 1B spot, up until a potential 469MB.

    This means that we actually have a pretty decent amount of RAM available to userspace!

    Please, please let me know if I'm wrong. If I'm not, we may have just stumbled upon something pretty big.

    Update: Just ran the program a few more times and it keeps coming back with the same address (106DB680). Looks like whatever the browser was doing before we loaded has no effect.
     
    Last edited by QuarkTheAwesome, Mar 18, 2016
    rw-r-r_0644 likes this.
  5. QuarkTheAwesome
    OP

    QuarkTheAwesome Working for Hugs

    Member
    787
    1,928
    Apr 19, 2015
    Australia
    Stuck in the PowerPC
    Just one more thing: I just allocated a few more variables and the address of each seems to be increasing by 0x140 each time rather than 0xFF (the size of what I allocated). I suspect that the printing code may be leaky or that it's something to do with the align variable on OSAllocFromSystem (which I still don't really understand ;3)

    If not, I should probably delete the lot of this. Unfortunately I can't really check right now (time for sleep :P) but I will double-check as soon as I can.
     
  6. eliboa

    eliboa Member

    Newcomer
    49
    56
    Jan 13, 2016
    France
    "Data structure alignment is the way data is arranged and accessed in computer memory. It consists of two separate but related issues: data alignment and data structure padding. When a modern computer reads from or writes to a memory address, it will do this in word sized chunks (e.g. 4 byte chunks on a 32-bit system) or larger. Data alignment means putting the data at a memory address equal to some multiple of the word size, which increases the system's performance due to the way the CPU handles memory"
    https://en.wikipedia.org/wiki/Data_structure_alignment

    You don't need to allocate memory for a simple variable. You need allocation for arrays or bigger structures i guess. Or for dynamic allocation.

    Btw, you can also use MEMAllocFromDefaultHeapEx(unsigned int size, int align)
     
    Last edited by eliboa, Mar 18, 2016
    QuarkTheAwesome likes this.
  7. NWPlayer123

    NWPlayer123 GBAtemp Addict

    Member
    2,629
    6,226
    Feb 17, 2012
    United States
    The Everfree Forest
    Okay, so basically, normal applications get 1GB of memory in MEM2 for both code and data, code area is at 0x02000000 to 0x10000000 (defined per application, but since we have to use the browser's JIT area at 0x01800000 to I think 0x02000000) and data is in 0x10000000 to 0x42000000 (1GB is 0x40000000 plus our starting offset of 0x02). Since we're hijacking the current application, I'm not sure how we get access to the 0x10000000 area, I ran MEMGetBaseHeapHandle and it gave me an address in there that in my memdump is an EXPH (Expanded Heap) header, then theoretically you should be able to use that handle to allocate memory. Every foreground application gets access to MEM2, and foreground applications also get access to MEM1 which is 32MB of super-fast memory at 0xF40000000, should be used for "high use" data. The foreground also has its own area at 0xE0000000, and it's 40MB (to 0xE4000000).
    Basically,
    0x01000000 to 0x01800000 - used by the system for OS modules/libraries
    0x01800000 to 0x02000000(?) - Just In Time area used by the browser
    0x02000000 to 0x10000000 - theoretically code for RPX applications but that we can't use without kernel
    0x10000000 to 0x42000000(? - depends, can go to 0x50000000)
    0xA0000000 to 0xD2000000(? - depends, can go to 0xE0000000) - mirror used by OSCodegenCopy which apparently the browser can't access
    0xE0000000 to 0xE4000000 - Foreground area
    0xF4000000 to 0xF6000000 - 32MB for MEM1
    0xF6000000 to 0xFFFFFFFF - used by the system

    I'm not at all sure how RPX applications are set to allocate, from what I can tell it depends, since ghs has a malloc builtin for Wii U, and when they're built, each section goes where it's supposed to, .text starts at 0x02000000, .rodata is at 0x10000000 (not sure if the loader modifies that offset since all my memdumps have library strings at the very start of it), .data is right after, .bss is right after that, and then all remaining memory can be allocated somehow. I assume you get a base heap handle like I tried, and then you can allocate a large chunk, then make a specialized heap (there are four available, expanded heap, frame heap, block heap, and unit heap).
    Unit heap is used for allocating in predecided chunks (EG give it a giant amount and tell it to allocate in units of 32KB, each time you make a new allocation it'll just allocate 32KB directly, doesn't need a size param)
    Expanded heap is used for general purpose stuff, its purpose is to work like malloc, lets you freely allocate memory. If you pass a negative alignment to MEMAllocFromExpHeapEx, it'll allocate it at the end of the expanded heap, so you can copy in compressed data and uncompress it into the start of the heap so you have less fragmentation.
    Frame heap and block heap I'm not sure about :\

    Here's a list of all available functions sorted into classes (you'll need to document what parameters they need tho, I started on that but lost my notes :\)
    Warning: Spoilers inside!
    This took a lot longer to make than I was expecting
     
  8. QuarkTheAwesome
    OP

    QuarkTheAwesome Working for Hugs

    Member
    787
    1,928
    Apr 19, 2015
    Australia
    Stuck in the PowerPC
    Dimok has a bunch of files on most of his repos that contain basically all functions (useful for getting signatures). Unfortunately it doesn't have all the memory functions, but it's a start. (Link)
     
  9. dimok

    dimok GBAtemp Advanced Fan

    Member
    728
    2,174
    Jan 11, 2009
    United States
    Hey the problem on this whole thing is not how much heap memory you have that you can still use. All the heap memory is not very usefull if you can't get the hardware to play with it.

    As NWPlayer123 already pointed out the whole problem on the 5.5.1 exploit is that it hijacks a running environment. That environment already has gained several hardware accesses such as GX2 or the sndcore which are one time run only and uses them blocking you off from using them. brienj already ran into that problem with sndcore too. The problem here is not that it is userspace. Almost all my application run in userspace. The difference is that they are hooked into the very beginning of the hijacked application (e.g. MiiMaker). You could say they hook the "main" function (I know it is not 100% accurate). At that position you still have everything untouched. But to hook yourself at that position you need to modify the coreinit code which requires write access to the code area from userspace and that write access needs to be first setup in the kernel. Since unfortunately there is no kernel exploit available for public for your firmware version you can't do it.

    Anyway here is one addition to NWPlayer123's text (which already explains it in theory) on how you could setup yourself a memory area:
    https://github.com/dimok789/homebrew_launcher/blob/master/src/system/memory.c#L53

    You can fragment that area into several parts by allocating less than the maximum available memory too and split the memory up. In that code part I gain the full MEM1 range for GX2 color-, depth-, hi-Z- and auxilary-buffers (AA only) as that is very fast memory and bucket memory for the scanbuffers. Additionally you have the full MEM2 memory (default heap) for your usage. The size depends on how much is setup for the application (example mii maker) which usually is around 1GB. RPX applications sometimes gain full default heap memory on start and setup own heap handles like I did for MEM1 and bucket memory and then they dont use the default heap handle anymore. Well this all works if you start from a fresh and good entry point (HBL offers that to the homebrews) but I can't tell you if and how good that will work from the browser exploit environment. It's really too bad that some developers still have to fight with that environment.
     
  10. brienj

    brienj Trying to avoid getting cancer

    Member
    1,232
    1,703
    Jan 3, 2016
    United States
    I had one question too, and may as well throw it in here, as it's kind of related to memory. I was playing around with the RPC, and I was dumping the memory from the framebuffer, which I figured I could open in Photoshop as a raw image, but I noticed that when dumping it, instead of dumping the memory as RGBA, it was dumping it as RRRRGGGGBBBBAAAA, repeating each byte of color four times, which made for an interesting looking picture. I haven't messed around much more with it, but does anyone know why it is doing that for the framebuffer, but it dumps all other memory just fine?
     
  11. QuarkTheAwesome
    OP

    QuarkTheAwesome Working for Hugs

    Member
    787
    1,928
    Apr 19, 2015
    Australia
    Stuck in the PowerPC
    I'm interpreting that as "the Internet browser links with GX2 and sndcore, and you can't use them at the same time as the browser." If that's the case, then shouldn't we be able to release the libraries? For instance, at the start of almost every userspace (when I say userspace, I'm referring to 5.5.x browserhax) homebrew out there there's a few lines of code involving IM_ methods (IM_Open, IM_Close and IM_SetDeviceState) that, as far as I know, kill the browser/restart the system? (no documentation, kinda guessing based off comments in the libwiiu examples) If that's not what it's doing, sorry. I'm sure we could come up with code to kill the browser properly anyway.
    Back on topic, if we've killed the browser, then it's not using those libraries anymore. If we can get the handles it had allocated (probably still in RAM somewhere) and release them (or just reuse them, whatever) we should, theoretically, get proper lib access.
    Of course, to do that, we need to know when and where those handles are allocated. And that is where RPC comes in.

    I'll be messing around with my WiiU and I'll try to get some info out of it, but I'd appreciate it if you guys have a poke around too. It'd be really useful.

    (I'm also thinking of maybe moving this discussion to a new thread (we've kinda gotten off the original topic on this one). Opinions? Or should we just keep rambling here?)
     
  12. NWPlayer123

    NWPlayer123 GBAtemp Addict

    Member
    2,629
    6,226
    Feb 17, 2012
    United States
    The Everfree Forest
    Eh, here's fine. I wish I knew what IM_Close did, AFAIK it'll shut down all running processes and threads, not sure if it frees the libraries it's using, though. I poked around with sound a while ago with TCPGecko (better than RPC since browser's still running, all apps need IM_Close to get OSScreen to function properly), which is how I found out it's still using the old Sound-1 libraries. I think I remember testing that Sound-1 and Sound-2 can't be acquired at the same time.
     
  13. QuarkTheAwesome
    OP

    QuarkTheAwesome Working for Hugs

    Member
    787
    1,928
    Apr 19, 2015
    Australia
    Stuck in the PowerPC
    That explains why everyone (including me) can't get Sound2 working - the browser's Sound1 hasn't been released yet. (the issues with Sound1 are probably just RAM-related)

    (sigh) I wish I had TCP Gecko..... (single tear)
     
    brienj likes this.
  14. dimok

    dimok GBAtemp Advanced Fan

    Member
    728
    2,174
    Jan 11, 2009
    United States
    Interessting. I never actually took a look at the framebuffer internals. My first though was 4x MSAA as every pixel is 4 times in it but then I remembered that cant be it since MSAA is done completely in the colorbuffer and auxilary buffer you have to allocate. You could write yourself a dump of every 4th byte i guess to get a proper image ;) but I don't know the reason why it is 4 times in there at this point.


    Yeah you might have a success if you shutdown all the subsystems correctly. The IM_ methods that you are refering to shutdown all running threads from what I understand. I don't know exactly what it does since it has a call to IOS_Ioctlv and its IOSU code that does some action from that point on but it seems like an event that tells every process to shutdown and whether they do it is up to the implementation of the application i guess. This function is called with parameter 4 (not 3 as on homebrew codes) from OSShutdown btw.

    I think the problem is that you need to call the correct shutdown functions. For example you need to call GX2Shutdown for GX2 and AXQuit for the sndcore. Problem is that you have to call them from the same core and maybe even from the same thread that the init function of those were called from. For example for GX2 you have to do every call to almost all GX2 functions from the same thread as it was initiated with and definately the same core as those functions are not thread safe at all. So maybe you are lucky and if you guess the correct core you might be able to stop AX and GX2 correctly. Additionally you might also have to call functions like AXRegisterFrameCallback with a NULL pointer to remove already registered callback function pointers from the browser and if needed register your own functions. Anyway I dont think the issue is not that the browser has acquired or linked the libraries as that is only a matter of memory but that it already started the usage of those and that before re-using them you need to shutdown everything of it properly. Theoretically it might be possible. To the last point of NWPlayer123, I am not sure if you cant acquire sound1 and sound2 at the same time (from library point of view), I never tried that. You could test it and if you get NULL pointer functions it doesnt work but if you get correct function pointers from the find export function for both libraries then the only problem is probably that sound1 has to be shutdown properly first to use sound2. Let's just hope you don't have to free all voices correct before the shutdown as that would be a pain in the ass to find the correct pointers.
     
    rw-r-r_0644, KiiWii, cmdj13 and 2 others like this.
  15. brienj

    brienj Trying to avoid getting cancer

    Member
    1,232
    1,703
    Jan 3, 2016
    United States
    Since you start a "video" to initiate the hack, could that be causing the issues with audio too? The "video" basically crashes the video player, right? And calling AXQuit to shutdown the audio core would lock up the console, it was one thing I thought to try. I could only call it for sndcore too, because as I said before, getting a handle to sndcore2 resulted in an instant lock up.
     
  16. NWPlayer123

    NWPlayer123 GBAtemp Addict

    Member
    2,629
    6,226
    Feb 17, 2012
    United States
    The Everfree Forest
    Yes, I was going to post that you could just not run IM_Close when running RPC to get the same effect but it'll freeze, you HAVE to run IM_Close or it'll crash :\ You gotta somehow shutdown all threads with it and then reacquire handles and shut everything down, some stuff you don't need to even call since they're deprecated
     
  17. QuarkTheAwesome
    OP

    QuarkTheAwesome Working for Hugs

    Member
    787
    1,928
    Apr 19, 2015
    Australia
    Stuck in the PowerPC
    Funnily enough, the RPC system from libwiiu doesn't call IM_Close at all. This, of course, doesn't work on 5.5 but it's interesting that they didn't feel the need to add it in.
     
  18. QuarkTheAwesome
    OP

    QuarkTheAwesome Working for Hugs

    Member
    787
    1,928
    Apr 19, 2015
    Australia
    Stuck in the PowerPC
    The Sound2 issue is probably because of what NWPlayer123 mentioned above (you can't use Sound1 and Sound2 at the same time, and the browser technically hasn't stopped using Sound1) and the Sound1 issue is likely because you're not on the same core the browser was (or as Dimok said, not even being on the same thread could cause an issue.) We could (also, as Dimok hinted at), theoretically, guess at which core the browser was on (I suspect it'd probably be the same each time anyway), start a thread on that core, and call AXQuit to free up Sound1 for our main thread. That is dodging the possible issue of being on the wrong thread, however. It also doesn't do much about how the WiiU could recover from an exception (us guessing wrong (if the core changes each time.)) It just needs a lot of investigation, that's all.
     
  19. NWPlayer123

    NWPlayer123 GBAtemp Addict

    Member
    2,629
    6,226
    Feb 17, 2012
    United States
    The Everfree Forest
    Apparently AXQuit just needs the same core which we can find with a variable (memdump, find the code and get the "real" address since data doesn't start right at 0x10000000, loader translates it). From what I can tell, this is all we need to do before getting the handle and calling OSDynLoad_Release, but the way that works is it keeps track of the number of references/acquires and only deallocates once it hits zero so we'll have to do some fancy loop with OSDynLoad_IsModuleLoaded
    Assembly for snd_core
     
  20. QuarkTheAwesome
    OP

    QuarkTheAwesome Working for Hugs

    Member
    787
    1,928
    Apr 19, 2015
    Australia
    Stuck in the PowerPC
    Where'd you get the assembly? That's pretty darn useful to have.
    It definitely wouldn't be that hard to do something like that. One quick question though: How do you figure out the function signatures and the like for stuff that isn't in wiiubrew or anything like that?