I'm writing this since I see a lot of speculation and incorrect information floating around about how gateway works. I'm going to post what I've learned from the smart guys in #3dsdev in a hopefully more understandable manner. The information I have is definitely not complete, and unfortunately may not be 100% accurate. I do not take any credit in this information, it's mostly learning from people like megazig, plutooo, and yellows8 in the chatroom. I just hope to make it more understandable.
First, for people who don't understand what a ROP payload is, here's a quick explanation. Usually for buffer overflow exploits, you overflow some buffer to overwrite a return address in memory. Typically, the hacker will write "shellcode" which is compiled ARM code and inject it using the same buffer overflow. The return address overwritten will be directed to that memory location and now you're running your own code to do whatever. With the introduction of XN (execute never) in modern computers, this is not possible because memory locations that can be executed (game code for example) are read-only and only the kernel can write to it. This leads to ROP, which, instead of injecting one return address to point to shellcode, you inject a bunch of addresses (anywhere from 20 to 100 or more) that point of bits and pieces of code that already exists in the game (or in this case the System Settings application) and effectively do what you want. A (overly) simple example, say an existing function sums 100 numbers. You can use the existing code to add two numbers by pointing to somewhere in the middle of the function.
So here's an overview of how Gateway works.
* Quick layout of the DS mode NVRAM: there are two "profiles" stored in the DS NVRAM chip, these profiles store the message, profile name, birthday, etc. You can read it on the nocash docs. Here's an excerpt on why there's two of them: "When changing settings, the older area is overwritten with new data (and incremented Update Counter). The two areas allow to recover previous settings in case of a write-error (eg. on a battery failure during write)."
1) The DS gateway installer overwrites the message length field of the second profile to be 110.
2) The installer writes an ROP payload to 110 offset from the message field of the second profile. This happens to be in an not-used area of the profile, so the DS mode still boots and doesn't detect corruption. However, this not-used area is very small (0x86 bytes) and is not enough for code to decrypt and load Loader.dat. Instead this payload is in charge of loading the entirely of profile 1 as another ROP payload, and jumps to that.
3) It then writes another ROP payload to the first profile. This second stage loader is responsible for 1) decrypting Loader.dat, 2) loading Loader.dat into memory, 3) jumping to Loader.dat
4) It then fixes some CRC16 redundancy checks for the modified settings, and changes the update counter so profile 2 will be used (remember, profile 1 is completely "corrupted" with the ROP data. this is why you need to rerun the installer after using DS mode because the 3DS will detect this corruption and reset the profile).
Now, the user reboots into 3DS mode and visits the DS profile settings. When this happens, the message will be loaded into memory; overflow, and the first stage ROP payload runs loading the second stage payload, which then decrypts Loader.dat (another ROP payload) which does all the cool stuff.
Here's some things yellows8 found out that explains why the DS payload is so "small":
1) Loader.dat is encrypted using AES-128-CBC with an IV value of all 00s.
2) The encryption key for Loader.dat is actually the first 16 bytes of the system menu code found at 0x00100100. This means that the Gateway people don't need to store the decryption key anywhere in memory, it already exists when the system menu launches.
3) Same thing with the file name "Launcher.dat". Any programmer would question how they're loading "Loader.dat" into memory and not have that string appear in the installer. The answer is that that string exists as "SYS:/Launcher.dat," an unrelated file that exists in the 3DS system, but the string was modified to point to a file in the SD card to open. Not really interesting exploit-wise, but interesting programming-wise.
Now the Launcher.dat payload is NOT ARM code, it's still a very large ROP chain. According to yellows8, most of the ROP chain is obfuscation.
Speculation
Those are the facts. Now, some commentary from me.
First, although I despise piracy and disapprove of groups like Gateway that hack for profit, I still admire the work they did which most likely involves:
1) Physically dumping the RAM (as neimod did; if you saw those flickr pictures, you'll see how much skills it takes to build such a contraption)
2) Analyzing the dumped code for user level exploits and kernel level exploits. They might have cheated with the DS profile exploit since that was on 3dbrew for months, but still a feat to get it working.
3) Getting ROP gadgets (they might have had a tool or built a tool for this) and creating the ROP chain. I admire their obfuscation techniques like using existing system code as a decryption key. This is also the first time I've ever heard of code obfuscation in an ROP chain.
4) Creating the kernel patches.
5) Creating a physical card and patching the kernel in a way that hinders the cloning of their physical card.
The rush to market may also explain their promises of more firmware support in the future. Remember, ROP can only use existing code, and this code changes from firmware version to version. For something like 4.1 - 4.2, the code may not change that much; but for something like 5.0, even if the UI is the same, the code could be very different. In order to make sure all their obfuscation works, they can't simply find new gadgets and rebuild the ROP chain. neimod also said that the kernel exploit has been fixed after 4.5; but if that is the case, the GW team should not make promises.
The other thing I both admire and curse the GW team is that it is impossible to analyze their work without hacking the 3DS ourselves. In order to see what code is being run, you need to dump the 3DS memory (or have a decrypted NAND dump, which usually requires dumping the memory to get keys). For experts like neimod and yellows8, who already have such dumps, it would be a matter of deobfuscating the ROP chains to figure out the exploits and such, and I suspect they already know; but they are unwilling to share and help additional piracy and I respect them for that. For enthusiasts like me, or clone companies, it makes our lives harder as the time and investment spent decoding the GW might be close to the time and effort spent cracking the 3DS from scratch, since the hard part is dumping the RAM and decoding/figuring out the code. By using ROP, even if they didn't obfuscate it, there is no easy way to figure out what is happening. Unlike stuff like the R4 or the PSJailbreak, where the security is in the shellcode, most of the security is based on the 3DS security itself. Effectively, the 3DS security that prevents hackers from seeing the memory is also preventing people like me from seeing how GW works.
On what the GW is theoretically capable of (on 4.5 at least since neimod said the kernel exploit is patched above that). Since it has kernel access, you can think of it like what the PSP lite custom firmwares are able to do. Although you have to rerun the exploit on each reboot, you should be able to do anything from loading unsigned code (homebrew) to theming the system menu to cheats and etc. Unfortunately, as it currently stands the only people who knows how GW works are the GW group themselves and #3dsdev hackers like neimod and yellows8. We can only hope that GW will work on getting homebrew to run and other nice stuff instead of just focusing on getting pirated ROMs with better and better compatibility to work.
First, for people who don't understand what a ROP payload is, here's a quick explanation. Usually for buffer overflow exploits, you overflow some buffer to overwrite a return address in memory. Typically, the hacker will write "shellcode" which is compiled ARM code and inject it using the same buffer overflow. The return address overwritten will be directed to that memory location and now you're running your own code to do whatever. With the introduction of XN (execute never) in modern computers, this is not possible because memory locations that can be executed (game code for example) are read-only and only the kernel can write to it. This leads to ROP, which, instead of injecting one return address to point to shellcode, you inject a bunch of addresses (anywhere from 20 to 100 or more) that point of bits and pieces of code that already exists in the game (or in this case the System Settings application) and effectively do what you want. A (overly) simple example, say an existing function sums 100 numbers. You can use the existing code to add two numbers by pointing to somewhere in the middle of the function.
So here's an overview of how Gateway works.
* Quick layout of the DS mode NVRAM: there are two "profiles" stored in the DS NVRAM chip, these profiles store the message, profile name, birthday, etc. You can read it on the nocash docs. Here's an excerpt on why there's two of them: "When changing settings, the older area is overwritten with new data (and incremented Update Counter). The two areas allow to recover previous settings in case of a write-error (eg. on a battery failure during write)."
1) The DS gateway installer overwrites the message length field of the second profile to be 110.
2) The installer writes an ROP payload to 110 offset from the message field of the second profile. This happens to be in an not-used area of the profile, so the DS mode still boots and doesn't detect corruption. However, this not-used area is very small (0x86 bytes) and is not enough for code to decrypt and load Loader.dat. Instead this payload is in charge of loading the entirely of profile 1 as another ROP payload, and jumps to that.
3) It then writes another ROP payload to the first profile. This second stage loader is responsible for 1) decrypting Loader.dat, 2) loading Loader.dat into memory, 3) jumping to Loader.dat
4) It then fixes some CRC16 redundancy checks for the modified settings, and changes the update counter so profile 2 will be used (remember, profile 1 is completely "corrupted" with the ROP data. this is why you need to rerun the installer after using DS mode because the 3DS will detect this corruption and reset the profile).
Now, the user reboots into 3DS mode and visits the DS profile settings. When this happens, the message will be loaded into memory; overflow, and the first stage ROP payload runs loading the second stage payload, which then decrypts Loader.dat (another ROP payload) which does all the cool stuff.
Here's some things yellows8 found out that explains why the DS payload is so "small":
1) Loader.dat is encrypted using AES-128-CBC with an IV value of all 00s.
2) The encryption key for Loader.dat is actually the first 16 bytes of the system menu code found at 0x00100100. This means that the Gateway people don't need to store the decryption key anywhere in memory, it already exists when the system menu launches.
3) Same thing with the file name "Launcher.dat". Any programmer would question how they're loading "Loader.dat" into memory and not have that string appear in the installer. The answer is that that string exists as "SYS:/Launcher.dat," an unrelated file that exists in the 3DS system, but the string was modified to point to a file in the SD card to open. Not really interesting exploit-wise, but interesting programming-wise.
Now the Launcher.dat payload is NOT ARM code, it's still a very large ROP chain. According to yellows8, most of the ROP chain is obfuscation.
Speculation
Those are the facts. Now, some commentary from me.
First, although I despise piracy and disapprove of groups like Gateway that hack for profit, I still admire the work they did which most likely involves:
1) Physically dumping the RAM (as neimod did; if you saw those flickr pictures, you'll see how much skills it takes to build such a contraption)
2) Analyzing the dumped code for user level exploits and kernel level exploits. They might have cheated with the DS profile exploit since that was on 3dbrew for months, but still a feat to get it working.
3) Getting ROP gadgets (they might have had a tool or built a tool for this) and creating the ROP chain. I admire their obfuscation techniques like using existing system code as a decryption key. This is also the first time I've ever heard of code obfuscation in an ROP chain.
4) Creating the kernel patches.
5) Creating a physical card and patching the kernel in a way that hinders the cloning of their physical card.
The rush to market may also explain their promises of more firmware support in the future. Remember, ROP can only use existing code, and this code changes from firmware version to version. For something like 4.1 - 4.2, the code may not change that much; but for something like 5.0, even if the UI is the same, the code could be very different. In order to make sure all their obfuscation works, they can't simply find new gadgets and rebuild the ROP chain. neimod also said that the kernel exploit has been fixed after 4.5; but if that is the case, the GW team should not make promises.
The other thing I both admire and curse the GW team is that it is impossible to analyze their work without hacking the 3DS ourselves. In order to see what code is being run, you need to dump the 3DS memory (or have a decrypted NAND dump, which usually requires dumping the memory to get keys). For experts like neimod and yellows8, who already have such dumps, it would be a matter of deobfuscating the ROP chains to figure out the exploits and such, and I suspect they already know; but they are unwilling to share and help additional piracy and I respect them for that. For enthusiasts like me, or clone companies, it makes our lives harder as the time and investment spent decoding the GW might be close to the time and effort spent cracking the 3DS from scratch, since the hard part is dumping the RAM and decoding/figuring out the code. By using ROP, even if they didn't obfuscate it, there is no easy way to figure out what is happening. Unlike stuff like the R4 or the PSJailbreak, where the security is in the shellcode, most of the security is based on the 3DS security itself. Effectively, the 3DS security that prevents hackers from seeing the memory is also preventing people like me from seeing how GW works.
On what the GW is theoretically capable of (on 4.5 at least since neimod said the kernel exploit is patched above that). Since it has kernel access, you can think of it like what the PSP lite custom firmwares are able to do. Although you have to rerun the exploit on each reboot, you should be able to do anything from loading unsigned code (homebrew) to theming the system menu to cheats and etc. Unfortunately, as it currently stands the only people who knows how GW works are the GW group themselves and #3dsdev hackers like neimod and yellows8. We can only hope that GW will work on getting homebrew to run and other nice stuff instead of just focusing on getting pirated ROMs with better and better compatibility to work.
Please elaborate why you posted the picture in response to my concerns.....
