Hacking [Devs] Automated crash reports for vulnerability discovery?

[DrDaxxy]

Dunno where to put this since it's more of a general console hacking idea than anything totally 3DS-specific, and I guess the right people would better be reached on IRC instead of here, but I don't feel like going there right now so here goes:

Say we already have code execution on a normally locked-down platform, like the 3DS. And we're developing system software (like a CFW) that's destined to be used by the general public, not just privately shared among a few console hackers.

Why not put in an opt-in feature to inject exception handlers everywhere we can and automatically send crash dumps to us? Not for catching bugs in homebrew - but among the countless CFW/homebrew/etc. end users, surely some of them have gotten random crashes due to browser or savegame corruption bugs that could be exploited to gain userland code execution. And hey, we might even hit the jackpot and have someone get a kernel panic, save a dump to disk, then submit it on next boot.

Of course, on the 3DS it's not that big of a deal anymore since we have no shortage of very useful exploits, but I shouldn't have to tell you it's always good to have alternatives (or spares for when something we use right now gets patched).

Possible problems I see with this: For one, it would require a bunch of effort. There's also the same problems you get with regular error reporting: generates lots of data/traffic, and that data may contain sensitive private information. That said, we should be able to keep traffic down at least a little by shipping an updated blacklist of already investigated bugs that we don't want to get more reports of, and we should of course let users know that what kind of data these reports may contain when asking for their permission to send them.

So, can anyone tell me whether this is a terrible idea and why?

 

[NintenDavid]

Seems awesome, but it might take a while to develop something like that.

 

[pokemoner2500]

Doesn't LUMA (Dev version) already have something like this?

 

[NintenDavid]

I don't think so, but I never use dev version.

 

[shinyquagsire23]

This would probably just end with a bunch of people looking for crashes and finding a bunch of useless ones. Anyone capable of finding exploitable crashes probably also is able to know if it's exploitable, and is possibly able to actually exploit it.

 

[DrDaxxy]

Doesn't LUMA (Dev version) already have something like this?

Sort of. It can take crash dumps. It doesn't have functionality for automatically posting them on the Internet. The feature's intended for homebrew developers debugging their own code on their own hardware, not detecting bugs in Nintendo software or commercial games.
Of course, that's already exactly the low-level functionality one would need to implement this; the rest is fairly easy.

This would probably just end with a bunch of people looking for crashes and finding a bunch of useless ones. Anyone capable of finding exploitable crashes probably also is able to know if it's exploitable, and is possibly able to actually exploit it.

For what it's worth, my intention isn't to have users actively look for bugs, but to collect data from a massive amount of them (as with regular crash reporters included in popular software). Sure, this doesn't make examining crashes any easier, but it does show you where to look.

As per Sturgeon's law, I'd expect 90% of (raw) reports to be garbage caused by either user error, unstable hax and two dozen boring issues in the browser and some popular games. Filtering them is a challenge. In addition to the aforementioned blacklist I'd suggest ratelimiting (if one person gets 50 crashes a day, it's probably nothing interesting), ignoring what can be identified as homebrew, and ignoring (or at least flagging) patched code. Happy to hear other ideas.

That said: Let's assume we build this and a large enough number of people use it but not a single interesting crash is submitted. That could still be useful. A game with an abnormally high crash rate (in relation to global playtime - obviously you'll get more reports about Pokémon than Face Racers: Photo Finish) is probably rather poorly written and I'd suspect you'd have better than average chances of finding some silly buffer overflow in the savegame handling code.

 

[NintenDavid]

Sort of. It can take crash dumps. It doesn't have functionality for automatically posting them on the Internet. The feature's intended for homebrew developers debugging their own code on their own hardware, not detecting bugs in Nintendo software or commercial games.
Of course, that's already exactly the low-level functionality one would need to implement this; the rest is fairly easy.



For what it's worth, my intention isn't to have users actively look for bugs, but to collect data from a massive amount of them (as with regular crash reporters included in popular software). Sure, this doesn't make examining crashes any easier, but it does show you where to look.

As per Sturgeon's law, I'd expect 90% of (raw) reports to be garbage caused by either user error, unstable hax and two dozen boring issues in the browser and some popular games. Filtering them is a challenge. In addition to the aforementioned blacklist I'd suggest ratelimiting (if one person gets 50 crashes a day, it's probably nothing interesting), ignoring what can be identified as homebrew, and ignoring (or at least flagging) patched code. Happy to hear other ideas.

That said: Let's assume we build this and a large enough number of people use it but not a single interesting crash is submitted. That could still be useful. A game with an abnormally high crash rate (in relation to global playtime - obviously you'll get more reports about Pokémon than Face Racers: Photo Finish) is probably rather poorly written and I'd suspect you'd have better than average chances of finding some silly buffer overflow in the savegame handling code.

Now I think about this, it seems awesome, but who would we send it to?

 

[DrDaxxy]

Someone would have to run a service (with HTTP API or whatever) for consoles to send data to. Could be me, could be a CFW author, could be someone else entirely. Or multiple people, sharing their data, so that if someone's service goes down (whether intentionally or because of technical issues) things keep working.

...I'm not sure why I ever suggested uploading full dumps, that's totally impractical. Keeping it to stack traces (and other similarly small data), this should be cheap enough to host too. Unless I seriously underestimate the amount of CFW users (who would turn this on). Or the bugginess of the average 3DS game/modded system

Also mostly solves the privacy concerns. It might even be okay to publicly post the data. Admittedly that would also allow Nintendo to access it; then again, they could just build error reporting into the firmware and get reports from most if not all consoles on their own.