Homebrew RELEASE Twili - Homebrew Debug Monitor

[misson20000]

This one's been in the works for a bit of a while. Presenting Twili, my homebrew debug monitor for the Nintendo Switch:

Twili - Homebrew Debug Monitor (now with GDB stub)
[GitHub] [Latest Release]

What is it?: It's a debug monitor/development tool/nx-hbloader replacement.

What does it let me do?:

• It lets you send and launch homebrew over USB (or TCP, I don't judge).
• If the homebrew is compatible, it lets you have stdin/stdout/stderr over USB, too, through the command line tool. This allows for a very fast development cycle.
• If you don't need your homebrew to be an applet (you're developing a background sysmodule), it can reduce iteration time even farther by preventing fatal errors when your sysmodule inevitably crashes.
• You can use GDB to debug your homebrew (applets, sysmodules, etc.) or even official software.
  
• When something crashes, you can generate an ELF core dump that you can load in a real man's debugger (aarch64-linux-gnu-elf) to help you figure out what went wrong instead of having to turn your console off, remove the SD card, find the crash log, and use addr2line to turn it into something meaningful.
• Did I mention that you can core dump any process, and not just homebrew?
• It lets you query overall RAM usage on your console.
  
• It lets you get a list of all the processes running on your console, and...
• It lets you kill them, too.
• If low-level debugging is your style, it lets you debug them too. As of 1.2.0, there's even a GDB stub.

How do I use Twili's standard IO?: If you're using libtransistor, you get it for free. If you're using libnx, you can link against libtwili (found in my twili-libnx repo). If you launch your application through twib run, stdout/stderr will go to your console and stdin will come from that console. If you launch any other way, stdout/stderr will go to twibd logs and stdin will block.

How is it different from nx-hbloader/nxlink?: Aside from having way more features, it runs each homebrew in its own process, which is a clean slate except for a small stub providing HBABI keys. This means that homebrew can manage its own memory and doesn't have to worry about cleaning up entirely behind itself for the next application.

Does it have a GDB stub?: As of 1.2.0, yes! See the README on the github repo for instructions.

What firmware versions does it work on?: To my knowledge, all of them. 1.0.0 might behave a bit differently regarding sysmodule crash fatal errors.

What CFWs does it work on?: The only CFW that I can confirm functionality on and actively support is Atmosphère, and only somewhat recent versions of it (0.10.2+). Twili has been tested on Atmosphère 0.10.2. Twili might work on other CFWs with some tweaks. Don't run Twili on piracy firmware because you'll make me very sad.

What should I know before I install it?: It replaces nx-hbloader. That means it replaces hbl.nsp. If you don't like Twili and want to remove it, don't forget to put back nx-hbloader's hbl.nsp.

How do I install it?: See the installation instructions on the GitHub repo.

Where do I download it?: Releases page on the GitHub repo.

What is the most recent version?: The most recent version is Twili 1.3.0. Also update Twib on each release, since bugfixes are included.

Where is the source code?: https://github.com/misson20000/twili

How is it licensed?: Twili is licensed under the GPLv3.

Where do I report bugs?: If you're not 100% convinced it's a bug, it's best to bring it up either here or on Discord first. You can find me in the ReSwitched discord. If you're convinced it's a bug, I keep track of those on GitHub.

Where do I get support?: Either here on on Discord. You can find me in the ReSwitched discord.

 

[hippy dave]

Sounds very useful, thanks!

 

[Cyan]

thank you for this tool.
as you need 5 posts to get link rights, here is a direct link : https://github.com/misson20000/twili


can we get stderr/stdout over tcp too or it's usb only ?
I guess I need to install the zadig/libusbk driver. I didn't install that yet, that's why I'd prefer tcp for now but I'd understand if it's not a good idea as it can't log if network is not init in the homebrew? or it can log over tcp even before launching the homebrew to debug?

you said we need to replace hbloader.nsp, can't we have both ? can't we set that nsp to another titleID using the loader.ini file? I guess it would require an update to atmosphere to support more button combo or more titleID in the ini.

If I need a simple stdout logger, it's nxlink I should use? I don't know all the existing tools yet.

 

[filfat]

Well, I’m going to put my hate towards GPL aside because this is really a great project.

Honestly, amazing work!

 

[misson20000]

can we get stderr/stdout over tcp too or it's usb only?

Everything you can do over USB you can do over TCP too, but in my experience, the USB bridge is more stable. When I say that the homebrew needs to be compatible, I mean that it needs to be written to send stdio over to Twili's IPC service, which hides the details of USB or TCP from the application. I've done this for libtransistor, and it probably wouldn't be too hard to do for libnx.

you said we need to replace hbloader.nsp, can't we have both ? can't we set that nsp to another titleID using the loader.ini file? I guess it would require an update to atmosphere to support more button combo or more titleID in the ini.

If you're not interested in launching applets through Twili, you can use it without replacing HBL.

If I need a simple stdout logger, it's nxlink I should use? I don't know all the existing tools yet.

If you're looking for a just simple stdout logger and don't mind that the application has to make its own network connection, that logging may not necessarily be entirely synchronous (log messages may get queued for a short while, which is not good if you're debugging crashes), or that your application is living in a process that has been/will be used by other homebrew applications, nxlink is probably a better tool for you. If you're looking for separate stdout/stderr streams and stdin support, you want your homebrew to be explicitly blocked until you're sure log messages have been delivered (useful for crash debugging), you want your own process (to run in the background), or your network is configured in such a way that makes nxlink impractical, Twili is the better tool.

 

[ali.ee]

Aside from the obvious usage of debugging, is this also the first step for a cheat engine running on switch itself?

 

[Reisyukaku]

>What CFWs does it work on?:
>The only CFW that I acknowledge and support..
That doesnt even answer the question though... I'll add this to my cringe comp though.

 

[misson20000]

Aside from the obvious usage of debugging, is this also the first step for a cheat engine running on switch itself?

No, not really. This is a development tool intended to be used with the corresponding desktop tool (twib). Using debug SVCs to poke process's memory spaces isn't new (it's what everyone else does), and this particular usage of them is geared specifically towards remote debugging.

 

[klock]

This is excellent, super useful, thanks!

 

[misson20000]

I've released Twili 1.0.1, including some fixes for recent versions of Atmosphère and an issue that would prevent games from launching on 5.0.0+.

Both the Twili sysmodule and twib/twibd have been updated. Old versions of twib/twibd should remain compatible, but I recommend you update them.

I've also published a library allowing libnx applications to use Twili's stdio. You can find it and its usage instructions on GitHub at misson20000/twili-libnx.

Additions

• Added `PrintDebugInfo` command
• AppletHost has more SVC permissions now

Changes

• Twili is now distributed as exefs.nsp
• We use the new npdm-json format now.
• Twib should no longer be quite as `exit()` happy
• USBBridge catches `std::bad_alloc` and returns `LIBTRANSISTOR_ERR_OUT_OF_MEMORY` responses instead of blowing up.
• Use upstream switch-tools (don't need my elf2kip fork anymore)

Fixes

• Managed processes now register service access control (rip sm:h on new versions of Atmosphere)
• AppletTracker now clears created process queue when control applet is lost
• Use applet memory instead of application memory so games can launch on 5.0.0+ with Twili running in background.

Notable Known Issues

• #55: Crash with 2000-000 while sitting on home menu: This crash doesn't seem to happen when twibd is running and is connected.
• #56: LibusbK (Windows) driver association is bad: Will be addressed in the next release.

 

[misson20000]

Twili 1.2.0 is out, notably including a GDB stub (finally). Usage instructions can still be found in the README.

Changelog since 1.0.1

• Added gdb stub (linux and osx only for now)
• Added push and pull commands
  
• Added launch command for pm:shell#LaunchProcess (won't launch games, but will launch sysmodules)
  
• Added libusbK-based contrib driver package
• Added buffering for twib pipes (speeds up throughput on small writes from ~315 KiB/s to ~2.35 MiB/s)
• Made twib pipes faster (max speed I've been able to measure is ~20 MiB/s throughput)
  
• Changed process launch to store executable on SD card instead of in memory (allows larger applications and fixes romfs for libnx)
• Added configuration file
  
• Fixed NextLoadPath for non-hbmenu (like RetroArch)
  
• Fixed for Stratosphere compiled with newer libnx
• Fixed thread handles for libnx
• Fixed IAppletController
• Fixed applet launch when first try fails
• Fixed broken device ID calculation for TCP backend
• Fixed 2000-0000 fatals
• Fixed race condition in initial device probing (twib list-devices could report before scan was complete)
  
• Refactored request handling
• Refactored twib event loops and platform code
  
• Reduced default buffer size (drastically improves memory usage: ~29 MiB -> 9 MiB)
• Made twibd exit cleanly on SIGINT
• Made twibd stop complaining about thread termination
• Various improvements for Windows

 

[hippy dave]

This is sounding good. I haven't used gdb in years, maybe time to get back into it.

 

[Blackspoon]

I cant get it work on 8.1.0 Atmos 0.9.3.

When I try to load hbl.nsp the console will do nothing.

twilibd will not recognize the console.

When I switch the usb_bridge and the usb log off it will boot into hbmenu and let me connect over tcp. But twilib still says that no device detected altough connect-tcp returned "OK".

Does twili support atmos 0.9.3 or have I go to 0.8.3 instead?

 

[wiitendo84]

I noticed it doesn't support breakpoints yet. Is that a feature you're looking to add if possible?

 

[misson20000]

When I switch the usb_bridge and the usb log off it will boot into hbmenu and let me connect over tcp. But twilib still says that no device detected altough connect-tcp returned "OK".

Does twili support atmos 0.9.3 or have I go to 0.8.3 instead?

It's supposed to work on latest Atmosphere version.

I noticed it doesn't support breakpoints yet. Is that a feature you're looking to add if possible?

Software breakpoints are supported- overwriting the target instruction with a break instruction and putting it back when we need to continue over it. Watchpoints are not supported. Comex has done some work on hardware breakpoints and watchpoints.

 

[wiitendo84]

It's supposed to work on latest Atmosphere version.



Software breakpoints are supported- overwriting the target instruction with a break instruction and putting it back when we need to continue over it. Watchpoints are not supported. Comex has done some work on hardware breakpoints and watchpoints.

Essentially the type I was wondering about would be setting a breakpoint on a address in a game to see what's writing to it so you could essentially nop it or stop it from subtracting or adding to it.

 

[skiddd]

I am trying to set things up on a 9.1.0 switch with Atmosphere 0.10.1.

I am on Windows 7 64-bit and have successfully installed the USB driver.
However, in my device manager I could only see one "TransistorUSB" device.
Nothing else is in Other Devices or devices without drivers installed.

I followed the Switch side installation and replaced hbl.nsp. Now when I try to
enter the Album section of the Switch it just shows me a blank gray page with
"Continue A" and nothing else.

Ran twibd_win64.exe and allowed it in the firewall and then tried twib_win64 list-devices.
The Switch was not detected

Any hints on how to get this working on a Windows machine?

B.R.

 

[misson20000]

Twili 1.3.0 is out. See GitHub for detailed changelog and release notes. This release fixes a lot of bugs and improves quality of life.

Any hints on how to get this working on a Windows machine?

Sorry, the Windows situation is quite a struggle right now. Ping me in the ReSwitched discord sometime and we can see about getting it working?