RELEASE Twili - Homebrew Debug Monitor

Discussion in 'Switch - Emulation, Homebrew & Software Projects' started by misson20000, Nov 7, 2018.

  1. misson20000

    misson20000 Newbie

    Nov 7, 2018
    United States
    Near Seattle
    This one's been in the works for a bit of a while. Presenting Twili, my homebrew debug monitor for the Nintendo Switch:

    Twili - Homebrew Debug Monitor (now with GDB stub)
    [GitHub] [Latest Release]

    What is it?: It's a debug monitor/development tool/nx-hbloader replacement.

    What does it let me do?:
    • It lets you send and launch homebrew over USB (or TCP, I don't judge).
    • If the homebrew is compatible, it lets you have stdin/stdout/stderr over USB, too, through the command line tool. This allows for a very fast development cycle.
    • If you don't need your homebrew to be an applet (you're developing a background sysmodule), it can reduce iteration time even farther by preventing fatal errors when your sysmodule inevitably crashes.
    • You can use GDB to debug your homebrew (applets, sysmodules, etc.) or even official software.
    • When something crashes, you can generate an ELF core dump that you can load in a real man's debugger (aarch64-linux-gnu-elf) to help you figure out what went wrong instead of having to turn your console off, remove the SD card, find the crash log, and use addr2line to turn it into something meaningful.
    • Did I mention that you can core dump any process, and not just homebrew?
    • It lets you query overall RAM usage on your console.
    • It lets you get a list of all the processes running on your console, and...
    • It lets you kill them, too.
    • If low-level debugging is your style, it lets you debug them too. As of 1.2.0, there's even a GDB stub.
    How do I use Twili's standard IO?: If you're using libtransistor, you get it for free. If you're using libnx, you can link against libtwili (found in my twili-libnx repo). If you launch your application through twib run, stdout/stderr will go to your console and stdin will come from that console. If you launch any other way, stdout/stderr will go to twibd logs and stdin will block.

    How is it different from nx-hbloader/nxlink?:
    Aside from having way more features, it runs each homebrew in its own process, which is a clean slate except for a small stub providing HBABI keys. This means that homebrew can manage its own memory and doesn't have to worry about cleaning up entirely behind itself for the next application.

    Does it have a GDB stub?: As of 1.2.0, yes! See the README on the github repo for instructions.

    What firmware versions does it work on?: To my knowledge, all of them. 1.0.0 might behave a bit differently regarding sysmodule crash fatal errors.

    What CFWs does it work on?: The only CFW that I can confirm functionality on and actively support is Atmosphère, and only somewhat recent versions of it (0.8.3+). Twili has been tested on Atmosphère 0.8.3. Twili might work on other CFWs with some tweaks. Don't run Twili on piracy firmware because you'll make me very sad. :(

    What should I know before I install it?: It replaces nx-hbloader. That means it replaces hbl.nsp. If you don't like Twili and want to remove it, don't forget to put back nx-hbloader's hbl.nsp.

    How do I install it?: See the installation instructions on the GitHub repo.

    Where do I download it?: Releases page on the GitHub repo.

    What is the most recent version?: The most recent version is Twili 1.2.0. I recommend you also update Twib, for GDB stub support.

    Where is the source code?:

    How is it licensed?: Twili is licensed under the GPLv3.

    Where do I report bugs?: If you're not 100% convinced it's a bug, it's best to bring it up either here or on Discord first. You can find me in the ReSwitched discord. If you're convinced it's a bug, I keep track of those on GitHub.

    Where do I get support?: Either here on on Discord. You can find me in the ReSwitched discord.
    Last edited by misson20000, Mar 31, 2019
    jakibaki, DSwizzy, Cubuss and 20 others like this.
  2. hippy dave

    hippy dave BBMB

    Apr 30, 2012
    United Kingdom
    Sounds very useful, thanks!
  3. Cyan

    Cyan GBATemp's lurking knight

    Global Moderator
    Oct 27, 2002
    Engine room, learning
    thank you for this tool.
    as you need 5 posts to get link rights, here is a direct link :

    can we get stderr/stdout over tcp too or it's usb only ?
    I guess I need to install the zadig/libusbk driver. I didn't install that yet, that's why I'd prefer tcp for now but I'd understand if it's not a good idea as it can't log if network is not init in the homebrew? or it can log over tcp even before launching the homebrew to debug?

    you said we need to replace hbloader.nsp, can't we have both ? can't we set that nsp to another titleID using the loader.ini file? I guess it would require an update to atmosphere to support more button combo or more titleID in the ini.

    If I need a simple stdout logger, it's nxlink I should use? I don't know all the existing tools yet.
    Last edited by Cyan, Nov 7, 2018
    lordelan and Subtle Demise like this.
  4. filfat

    filfat Musician, Developer & Entrepreneur

    GBAtemp Patron
    filfat is a Patron of GBAtemp and is helping us stay independent!

    Our Patreon
    Nov 24, 2012
    Well, I’m going to put my hate towards GPL aside because this is really a great project.

    Honestly, amazing work!
  5. misson20000

    misson20000 Newbie

    Nov 7, 2018
    United States
    Near Seattle
    Everything you can do over USB you can do over TCP too, but in my experience, the USB bridge is more stable. When I say that the homebrew needs to be compatible, I mean that it needs to be written to send stdio over to Twili's IPC service, which hides the details of USB or TCP from the application. I've done this for libtransistor, and it probably wouldn't be too hard to do for libnx.

    If you're not interested in launching applets through Twili, you can use it without replacing HBL.

    If you're looking for a just simple stdout logger and don't mind that the application has to make its own network connection, that logging may not necessarily be entirely synchronous (log messages may get queued for a short while, which is not good if you're debugging crashes), or that your application is living in a process that has been/will be used by other homebrew applications, nxlink is probably a better tool for you. If you're looking for separate stdout/stderr streams and stdin support, you want your homebrew to be explicitly blocked until you're sure log messages have been delivered (useful for crash debugging), you want your own process (to run in the background), or your network is configured in such a way that makes nxlink impractical, Twili is the better tool.
    Last edited by misson20000, Nov 7, 2018
    Cyan likes this.
  6. Member

    May 1, 2009
    Aside from the obvious usage of debugging, is this also the first step for a cheat engine running on switch itself?:unsure:
  7. Reisyukaku

    Reisyukaku Onii-sama~

    Feb 11, 2014
    United States
    >What CFWs does it work on?:
    >The only CFW that I acknowledge and support..
    That doesnt even answer the question though... I'll add this to my cringe comp though. [​IMG]
    Last edited by Reisyukaku, Nov 7, 2018
  8. misson20000

    misson20000 Newbie

    Nov 7, 2018
    United States
    Near Seattle
    No, not really. This is a development tool intended to be used with the corresponding desktop tool (twib). Using debug SVCs to poke process's memory spaces isn't new (it's what everyone else does), and this particular usage of them is geared specifically towards remote debugging.
 likes this.
  9. klock

    klock Member

    Dec 24, 2010
    United States
    This is excellent, super useful, thanks!
  10. misson20000

    misson20000 Newbie

    Nov 7, 2018
    United States
    Near Seattle
    I've released Twili 1.0.1, including some fixes for recent versions of Atmosphère and an issue that would prevent games from launching on 5.0.0+.

    Both the Twili sysmodule and twib/twibd have been updated. Old versions of twib/twibd should remain compatible, but I recommend you update them.

    I've also published a library allowing libnx applications to use Twili's stdio. You can find it and its usage instructions on GitHub at misson20000/twili-libnx.

    • Added `PrintDebugInfo` command
    • AppletHost has more SVC permissions now
    • Twili is now distributed as exefs.nsp
    • We use the new npdm-json format now.
    • Twib should no longer be quite as `exit()` happy
    • USBBridge catches `std::bad_alloc` and returns `LIBTRANSISTOR_ERR_OUT_OF_MEMORY` responses instead of blowing up.
    • Use upstream switch-tools (don't need my elf2kip fork anymore)
    • Managed processes now register service access control (rip sm:h on new versions of Atmosphere)
    • AppletTracker now clears created process queue when control applet is lost
    • Use applet memory instead of application memory so games can launch on 5.0.0+ with Twili running in background.
    Notable Known Issues
    • #55: Crash with 2000-000 while sitting on home menu: This crash doesn't seem to happen when twibd is running and is connected.
    • #56: LibusbK (Windows) driver association is bad: Will be addressed in the next release.
    the_leg and hippy dave like this.
  11. misson20000

    misson20000 Newbie

    Nov 7, 2018
    United States
    Near Seattle
    Twili 1.2.0 is out, notably including a GDB stub (finally). Usage instructions can still be found in the README.

    Changelog since 1.0.1
    • Added gdb stub (linux and osx only for now)
    • Added push and pull commands
    • Added launch command for pm:shell#LaunchProcess (won't launch games, but will launch sysmodules)
    • Added libusbK-based contrib driver package
    • Added buffering for twib pipes (speeds up throughput on small writes from ~315 KiB/s to ~2.35 MiB/s)
    • Made twib pipes faster (max speed I've been able to measure is ~20 MiB/s throughput)
    • Changed process launch to store executable on SD card instead of in memory (allows larger applications and fixes romfs for libnx)
    • Added configuration file
    • Fixed NextLoadPath for non-hbmenu (like RetroArch)
    • Fixed for Stratosphere compiled with newer libnx
    • Fixed thread handles for libnx
    • Fixed IAppletController
    • Fixed applet launch when first try fails
    • Fixed broken device ID calculation for TCP backend
    • Fixed 2000-0000 fatals
    • Fixed race condition in initial device probing (twib list-devices could report before scan was complete)
    • Refactored request handling
    • Refactored twib event loops and platform code
    • Reduced default buffer size (drastically improves memory usage: ~29 MiB -> 9 MiB)
    • Made twibd exit cleanly on SIGINT
    • Made twibd stop complaining about thread termination
    • Various improvements for Windows
  12. hippy dave

    hippy dave BBMB

    Apr 30, 2012
    United Kingdom
    This is sounding good. I haven't used gdb in years, maybe time to get back into it.