Homebrew 3DSControllerPlus - PC-->3DS video and input streaming

[Sono]

What libraries do these two Python programs need?

3dsp.py requires Xlib, sockimg.py requires PIL (Python Image Library), everything else is part of the Python standard library. You also need Python 2.7, because none of these programs have been updated to Python 3.x

 

[placebo_yue]

You have an old3DS? It's a known problem. The networking is overloaded, and it just dies... Sadly it also happens on new3DS, but it takes much longer to disconnect..

Yeah i have an O3DS. Is there anything i can do to avoid that? or maybe it's something you're working to fix/improve? or there's not much to do about it?

Oh, about the people wanting a streaming way for noobs, using CMD is not really that hard, give it a try!
If anyone wants i can write a little guide for people that barely know what they're doing like me, and if it works maybe it can be added to the main post i guess? it sure would've helped me when i first tried to use this, too.

 

[Deleted member 446461]

3dsp.py requires Xlib, sockimg.py requires PIL (Python Image Library), everything else is part of the Python standard library. You also need Python 2.7, because none of these programs have been updated to Python 3.x

Is Python 2.7.16 valid? Python 3 has been apparently assigned to the command python3.

 

[Sono]

Yeah i have an O3DS. Is there anything i can do to avoid that? or maybe it's something you're working to fix/improve? or there's not much to do about it?

Oh, about the people wanting a streaming way for noobs, using CMD is not really that hard, give it a try!
If anyone wants i can write a little guide for people that barely know what they're doing like me, and if it works maybe it can be added to the main post i guess? it sure would've helped me when i first tried to use this, too.

Oh, my, there is DEFINITELY things to improve. The video decoding portion is an unoptimized unreadable state management mess locked to 60FPS, so if a video frame takes two networking "cycles" to download, the framerate will just die, even though the 3DS could definitely download more data and cap out at 60FPS. It's really old bad code.

The problem with UIs is that I have written these programs in C or C++, and back when I wrote these, I didn't even know how to create a window in WINAPI. And since it's really difficult to do screen capture right, I just gave up and said "it's for advanced users only" to try deter people from using it and asking how to use commandline.

Too bad I don't have time to fix this project, even though it would have a lot of potential if I didn't let it rot.

Is Python 2.7.16 valid? Python 3 has been apparently assigned to the command python3.

As long as it's 2.7, it doesn't matter. That should be valid.

 

[Deleted member 446461]

Also, what's the name of the Python Image Library package? It seems like it doesn't find it as pil nor PIL.

 

[placebo_yue]

Oh, my, there is DEFINITELY things to improve. The video decoding portion is an unoptimized unreadable state management mess locked to 60FPS, so if a video frame takes two networking "cycles" to download, the framerate will just die, even though the 3DS could definitely download more data and cap out at 60FPS. It's really old bad code.

The problem with UIs is that I have written these programs in C or C++, and back when I wrote these, I didn't even know how to create a window in WINAPI. And since it's really difficult to do screen capture right, I just gave up and said "it's for advanced users only" to try deter people from using it and asking how to use commandline.

Too bad I don't have time to fix this project, even though it would have a lot of potential if I didn't let it rot..

oh, well i hope you eventually find the time to work on this again, it's got a lot of potential.
It'd be nuts to play PS2 or some Pc classic like "Sonic R" from my livingroom while my PC streams it from my room. An UI is not really necessary, that's work to be done when the software is super polished an in a final release state of sorts, i'd say.

why is people talking about python here? i didn't need to use it

 

[Sono]

Also, what's the name of the Python Image Library package? It seems like it doesn't find it as pil nor PIL.

It might be built-in, not sure.

oh, well i hope you eventually find the time to work on this again, it's got a lot of potential.
It'd be nuts to play PS2 or some Pc classic like "Sonic R" from my livingroom while my PC streams it from my room. An UI is not really necessary, that's work to be done when the software is super polished an in a final release state of sorts, i'd say.

why is people talking about python here? i didn't need to use it

Yeah, I'm no good at UIs. I'm good at making UIs usable when I'm using a sane UI library I know (like WinForms), but otherwise I just fall back to the dummy UI (like some versions of HorizonScreen have a really basic UI with text) or just console application.

There are Python scripts which work on Linux, as a replacement for 3DSCPlusDummy's functionality.

 

[Deleted member 446461]

Hmm. I've managed to run 3dscontrollerplus.py alone, but when I put it to connect to the 3DS's IP address, the New Nintendo 3DS seems to refuse the connection, clearing the screen and then printing...

"Disconnected
Listening on 172.20.10.7:6956 (6957 for video)"

And then every 5 seconds that same text flickers, as if the screen was being printed and the text was being printed over and over again.

I thought it was the VPN, but I disabled it and it still happens.

 

[Sono]

Hmm. I've managed to run 3dscontrollerplus.py alone, but when I put it to connect to the 3DS's IP address, the New Nintendo 3DS seems to refuse the connection, clearing the screen and then printing...

"Disconnected
Listening on 172.20.10.7:6956 (6957 for video)"

And then every 5 seconds that same text flickers, as if the screen was being printed and the text was being printed over and over again.

I thought it was the VPN, but I disabled it and it still happens.

There are debug prints prefixed with a #, remove the # from #print, so it becomes print, and post debug output.

 

[Deleted member 446461]

Almost forgot to say that even though sockimg.py is in the files, the program complains that sockimg can't be found.

Now, the computer returns the following message in the terminal repeatedly every 3 seconds:

"Sending ping packet to 172.20.10.7
received message 1 from ('172.20.10.7', 6956)"

The problem, in that case, must be the Nintendo 3DS, I presume; disabling the VPN doesn't change the outcome.

Installing the python-pil package now prevents the program from complaining sockimg doesn't exist, and now returns:

"Sending ping packet to 172.20.10.7
received message 1 from ('172.20.10.7', 6956)
Can't send image!"

 

[Sono]

Almost forgot to say that even though sockimg.py is in the files, the program complains that sockimg can't be found.

Now, the computer returns the following message in the terminal repeatedly every 3 seconds:

"Sending ping packet to 172.20.10.7
received message 1 from ('172.20.10.7', 6956)"

The problem, in that case, must be the Nintendo 3DS, I presume; disabling the VPN doesn't change the outcome.

Installing the python-pil package now prevents the program from complaining sockimg doesn't exist, and now returns:

"Sending ping packet to 172.20.10.7
received message 1 from ('172.20.10.7', 6956)
Can't send image!"

Ah, found it! There is a line around the top of 3dsp.py, set img = None

--------------------- MERGED ---------------------------

Nope, I was wrong. It seems like I have updated the 3DS side, and this change was made to prevent older versions of the dummy to connect to newer version of PaintController.

Find these lines:
print("Sending ping packet to",host)
rawdata = struct.pack('<BBxxI', 0, 0, altkey)

change it to
rawdata = struct.pack('<BBBBI', 0, 0, 1, 0, altkey)

And see if it works. I have no guarantee it'll work, but it's worth a try.

 

[Deleted member 446461]

Ah, found it! There is a line around the top of 3dsp.py, set img = None

--------------------- MERGED ---------------------------

Nope, I was wrong. It seems like I have updated the 3DS side, and this change was made to prevent older versions of the dummy to connect to newer version of PaintController.

Find these lines:
print("Sending ping packet to",host)
rawdata = struct.pack('<BBxxI', 0, 0, altkey)

change it to
rawdata = struct.pack('<BBBBI', 0, 0, 1, 0, altkey)

And see if it works. I have no guarantee it'll work, but it's worth a try.

It works. And it feels weird, but it's cool! I can't notice any bit of lag.

The thing is that it doesn't stream the screen. But it doesn't matter for now.

I would like to know if there's a way to map all of this to just "gamepad" instead of keyboard.

 

[Sono]

It works. And it feels weird, but it's cool! I can't notice any bit of lag.

The thing is that it doesn't stream the screen. But it doesn't matter for now.

I would like to know if there's a way to map all of this to just "gamepad" instead of keyboard.

Gamepad requires DummyJoy, but people have been having problems with it.

I don't have time, but once I do, I have a version ready which emulates an Xbox360 controller. The problem is that the library I use doesn't work properly, and I need to make modifications to it so it works better.

 

[Deleted member 446461]

Gamepad requires DummyJoy, but people have been having problems with it.

I don't have time, but once I do, I have a version ready which emulates an Xbox360 controller. The problem is that the library I use doesn't work properly, and I need to make modifications to it so it works better.

Oh. Well then.

Also, I just noticed the lag comes up if the Internet's active. Or anything that uses W.L.A.N.

According to 3dsp.py, Y enables alternative input mode, but pressing Y instead turns on backlight. And the program still doesn't send image.

EDIT: Figured out how to change the Alt-Input hotkey(s). I also figured out how to make the C-Stick (must call it C-Nub) move the mouse instead of the Circle Pad, in order to play games without having pain in my finger because of the D-Pad clickiness. It's weird to think people like clicky buttons. Ugh.

I'll try to give away the client I have with my configuration.

 

[Deleted member 446461]

Sorry for the delay. I can't access the Internet without mobile data for now.

Copy and paste the text in the spoilers into a python code file.

Spoiler: 3dsp.py

#!/usr/bin/env python

from __future__ import print_function
import socket, select, struct, time
import sys, signal
import Xlib, Xlib.display, Xlib.XK
LMouse = []; RMouse = []; MMouse = []; MouseSU = []; MouseSD = []; MouseSL = []; MouseSR = []; MouseF = [];
Button = []; MouseAbs = []; MouseRel = []; MouseAbsClick = []; MouseRelClick = []

##########################################################
# CONFIGURABLE REGION START - Don't touch anything above #
##########################################################

#3DS IP address and port (port is hardcoded in the 3DS client)
host = '172.20.10.7'
port = 6956

#interval to check if the 3DS has been disconnected
polltimeout = 3

#alternative mode activator keycombo
altkey = ["L","R","X","Y"]

#exit program if the 3DS disconnects
dcexit = False

#overlay image to send (None or (filename, timeout))
img = ('3dso.png', 0.02)

#This tells what the touch screen does if touched.
#Valid values: Button, MouseAbs, MouseRel, MouseRelClick, MouseRelClick
#Button sends the Tap button.
#MouseAbs moves your mouse to the same part of the screen as the touch screen was touched.
#MouseRel moves your mouse by the same distance as you drag across the touch screen.
#MouseAbsClick and MouseRelClick send the primary mouse button event if the screen is tapped, not held.
touch = MouseRel

mouse_speed = 2
mouse_speedup = 1
# The number of pixels on each side of the 3DS screen which are ignored, since you can't reach the outermost corners.
abs_deadzone = 10

#Valid values can be found in any of these locations on your Linux system (some may not exist):
# /usr/include/X11/keysymdef.h

btn_map = \
{
"A": ["X"],
"B": ["Z"],
"Select": [Xlib.XK.XK_Escape],
"Start": [Xlib.XK.XK_Return],
"Right": [Xlib.XK.XK_Right],
"Left": [Xlib.XK.XK_Left],
"Up": [Xlib.XK.XK_Up],
"Down": [Xlib.XK.XK_Down],
"L": [Xlib.XK.XK_Control_L],
"R": [Xlib.XK.XK_Shift_L],
"X": ["V"],
"Y": ["C"],
"ZL": [LMouse],
"ZR": [RMouse],
"Tap": [],
"CSRight": [],
"CSLeft": [],
"CSUp": [],
"CSDown": [],
"CRight": ["D"],
"CLeft": ["A"],
"CUp": ["W"],
"CDown": ["S"],
}

alt_map = \
[
(90, 90, 140, 60, [" "]),
(0, 0, 16, 16, [Xlib.XK.XK_Control_L, "S"])
]
########################################################
# CONFIGURABLE REGION END - Don't touch anything below #
########################################################

def pprint(obj):
import pprint
pprint.PrettyPrinter().pprint(obj)

if img:
try:
import sockimg
except:
print("sockimg can't be found, image won't be sent!")
img = None
pass

class x: pass

command = x()
command.CONNECT = 0
command.DISCONNECT = 1
command.KEYS = 2
command.SCREENSHOT = 3

keynames = \
[
"A",
"B",
"Select",
"Start",
"Right",
"Left",
"Up",
"Down",
"R",
"L",
"X",
"Y",
None,
None,
"ZL",
"ZR",
None,
None,
None,
None,
"Tap",
None,
None,
None,
"CSRight",
"CSLeft",
"CSUp",
"CSDown",
"CRight",
"CLeft",
"CUp",
"CDown"
]

keys = x()
keys.A = 1<<0
keys.B = 1<<1
keys.Select = 1<<2
keys.Start = 1<<3
keys.Right = 1<<4
keys.Left = 1<<5
keys.Up = 1<<6
keys.Down = 1<<7
keys.R = 1<<8
keys.L = 1<<9
keys.X = 1<<10
keys.Y = 1<<11
keys.ZL = 1<<14 # (new 3DS only)
keys.ZR = 1<<15 # (new 3DS only)
keys.Tap = 1<<20 # Not actually provided by HID
keys.CSRight = 1<<24 # c-stick (new 3DS only)
keys.CSLeft = 1<<25 # c-stick (new 3DS only)
keys.CSUp = 1<<26 # c-stick (new 3DS only)
keys.CSDown = 1<<27 # c-stick (new 3DS only)
keys.CRight = 1<<28 # circle pad
keys.CLeft = 1<<29 # circle pad
keys.CUp = 1<<30 # circle pad
keys.CDown = 1<<31 # circle pad

def currentKeyboardKey(x, y):
for i in alt_map:
if i[0] <= x and (i[0] + i[2]) > x and i[1] <= y and (i[1] + i[3]) > y:
return i[4]
return None

def key_to_keysym(key):
if not key: return 0

if isinstance(key,str):
if key=="\x08": return Xlib.XK.XK_BackSpace
if key=="\13": return Xlib.XK.XK_Return
if key==" ": return Xlib.XK.XK_space
return Xlib.XK.string_to_keysym(key)

return key

def action_key(key, action):
x_action = Xlib.X.ButtonRelease
x_action2 = Xlib.X.KeyRelease
if action:
x_action = Xlib.X.ButtonPress
x_action2 = Xlib.X.KeyPress

if key is LMouse or key is RMouse or key is MMouse or key is MouseSU or key is MouseSD or key is MouseSL or key is MouseSR:
if key is LMouse: button = 1
if key is MMouse: button = 2
if key is RMouse: button = 3
if key is MouseSU: button = 4
if key is MouseSD: button = 5
if key is MouseSL: button = 6
if key is MouseSR: button = 7
button = disp.get_pointer_mapping()[button-1] # account for left-handed mice
disp.xtest_fake_input(x_action, button)
disp.sync()
return

if key is MouseF:
global mouse_speed
if action: mouse_speed += mouse_speedup
if not action: mouse_speed -= mouse_speedup
return

keysym = key_to_keysym(key)
if not keysym: return

keycode = disp.keysym_to_keycode(keysym)
disp.xtest_fake_input(x_action2, keycode)
disp.sync()

def press_key(key):
action_key(key,True)

def release_key(key):
action_key(key,False)

def move_mouse(x,y):
x=int(x)
y=int(y)
if not x and not y: return

disp.warp_pointer(x,y)
disp.sync()

def move_mouse_abs_frac(x,y):
root = disp.screen().root
geom = root.get_geometry()

root.warp_pointer(int(x*geom.width), int(y*geom.height))
disp.sync()

disp = Xlib.display.Display()

touch_click = (touch is MouseAbsClick or touch is MouseRelClick)
if touch is MouseAbsClick: touch = MouseAbs
if touch is MouseRelClick: touch = MouseRel

if touch is MouseAbs and disp.screen_count()!=1:
print("Sorry, but MouseAbs only supports a single monitor. I'll use MouseRel instead.")
touch = MouseRel

sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)

prevkeys = 0

touch_start = 0
touch_last_x = 0
touch_last_y = 0

keyboard_prevkey = None

altkeyi = altkey
altkey = 0

for ke in altkeyi:
for k,v in enumerate(keynames):
if v == ke:
altkey += 1 << k

def sigterm_handler(_signo, _stack_frame):
rawdata = struct.pack('<BBxxI', 1, 0, 0)
sock.sendto(rawdata,(host,port))
sys.exit(0)

signal.signal(signal.SIGTERM, sigterm_handler)

def loopme():
global rawdata
global sock
global addr
global prevkeys
global keyboard_prevkey
global touch_start
global touch_last_x
global touch_last_y
global connectd

connectd = False
_timeout = 0

while True:
sel = select.select([sock], [], [], _timeout)
if not sel[0]:
if connectd:
#print("Socket timeout")
connectd = False
_timeout = polltimeout
#print("Sending ping packet to",host)
rawdata = struct.pack('<BBBBI', 0, 0, 1, 0, altkey)
sock.sendto(rawdata,(host,port))
continue

rawdata, addr = sock.recvfrom(4096)
if addr[0] != host:
#print("ignoring message", rawdata, "from", addr[0])
continue

rawdata = bytearray(rawdata)
#print("received message %i" % rawdata[0], "from", addr)

if not connectd or rawdata[0]==command.CONNECT:
connectd = True
if img:
try:
sockimg.sockimg(sock, img[0], (host,port), img[1])
except:
print("Can't send image!")
pass

if not connectd: continue

if rawdata[0]==command.DISCONNECT:
if dcexit: raise KeyboardInterrupt
else:
connectd = False
continue

if rawdata[0]==command.KEYS:
fields = struct.unpack("<BBxxIHHhhhh", rawdata)

data = \
{
"command": fields[0],
"altcmd": fields[1],
"keys": fields[2],
"touchX": fields[3],
"touchY": fields[4],
"circleX": fields[5],
"circleY": fields[6],
"cstickX": fields[7],
"cstickY": fields[8],
}
#print(data)

newkeys = data["keys"] & ~prevkeys
oldkeys = ~data["keys"] & prevkeys
prevkeys = data["keys"]

for btnid in range(32):
if newkeys & (1<<btnid):
for ke in btn_map[keynames[btnid]]:
press_key(ke)
if oldkeys & (1<<btnid):
for ke in reversed(btn_map[keynames[btnid]]):
release_key(ke)
if newkeys & keys.Tap:
if data["altcmd"]:
keyboard_prevkey = currentKeyboardKey(data["touchX"], data["touchY"])
if keyboard_prevkey:
for ke in keyboard_prevkey:
press_key(ke)
elif touch is Button:
for ke in btn_map["Tap"]:
press_key(ke)
touch_start = time.time()
if oldkeys & keys.Tap:
if keyboard_prevkey:
for ke in reversed(keyboard_prevkey):
release_key(ke)
keyboard_prevkey = None
elif touch is Button:
for ke in btn_map["Tap"]:
release_key(ke)
if data["keys"] & keys.Tap:
if touch is MouseAbs:
x = (data["touchX"]-abs_deadzone) / (320.0-abs_deadzone*2)
y = (data["touchY"]-abs_deadzone) / (240.0-abs_deadzone*2)
move_mouse_abs_frac(x, y)
if touch is MouseRel and not newkeys & keys.Tap:
x = (data["touchX"]-touch_last_x) * mouse_speed
y = (data["touchY"]-touch_last_y) * mouse_speed
move_mouse(x, y)
touch_last_x = data["touchX"]
touch_last_y = data["touchY"]

if oldkeys & keys.Tap and touch_click and time.time()-touch_start < 0.1 and not keyboard_prevkey:
press_key(LMouse)
release_key(LMouse)

if abs(data["cstickX"])>=16 or abs(data["cstickY"])>=16:
move_mouse(data["cstickX"]*mouse_speed/32.0, -data["cstickY"]*mouse_speed/32.0)

if rawdata[0]==command.SCREENSHOT:
pass # unused by both 3DS and PC applications

try:
loopme()
except KeyboardInterrupt:
sigterm_handler(None, None)
pass

Also, I still can't get image to work, even after I installed python-pil.

Nor I know how to bring up the """touchscreen keyboard""" on the Nintendo 3DS.

 

[Sono]

Sorry for the delay. I can't access the Internet without mobile data for now.

Copy and paste the text in the spoilers into a python code file.

Spoiler: 3dsp.py

-snip-

Also, I still can't get image to work, even after I installed python-pil.

Nor I know how to bring up the """touchscreen keyboard""" on the Nintendo 3DS.

The reason you can't get image working is because real-time image streaming is only available via FileStreamer. The built-in "image" "streaming" only sends a cover image for use with "alternative mode", which replaces the keyboard found in the original version. You can change the key combo which triggers alternative mode, but it's usually set as altk=00000800, which is Y on the 3DS.

In the Python script, there is a variable named "alt_map", and it contains rectangles with actions associated to them. That's the only way you can get a "touchscreen keyboard".

 

[Deleted member 446461]

The reason you can't get image working is because real-time image streaming is only available via FileStreamer. The built-in "image" "streaming" only sends a cover image for use with "alternative mode", which replaces the keyboard found in the original version. You can change the key combo which triggers alternative mode, but it's usually set as altk=00000800, which is Y on the 3DS.

In the Python script, there is a variable named "alt_map", and it contains rectangles with actions associated to them. That's the only way you can get a "touchscreen keyboard".

Oh. I already managed to change the alternative mode hotkey to L+R+X+Y.

Also, I found an alt_map text that says

"alt_map = \"
...and some stuff I don't quite understand.
" (90, 90, 140, 60, [" "]),
(0, 0, 16, 16, [Xlib.XK.XK_Control_L, "S"])"

I changed that space after 60, [" to an I. Not sure if that would make the letter visible. Is that close enough to what am I looking for?

 

[Sono]

Oh. I already managed to change the alternative mode hotkey to L+R+X+Y.

Also, I found an alt_map text that says

"alt_map = \"
...and some stuff I don't quite understand.
" (90, 90, 140, 60, [" "]),
(0, 0, 16, 16, [Xlib.XK.XK_Control_L, "S"])"

I changed that space after 60, [" to an I. Not sure if that would make the letter visible. Is that close enough to what am I looking for?

The PC version draws a rectangle on top of the image so you can find your rectangles, but this version doesn't. You'll have to HOLD the altkey combo, and then touch the screen in the rectangle you specified. It should behave as if you pressed a virtual key, so holding it keeps it held on your PC.

 

[Sono]

I have fixed two bugs:
1) fixed the "60FPS lock" bug, so new3DS can reach past 60FPS (although past ~91FPS you get graphical glitches)
2) FINALLY, AFTER ALL THIS TIME! I fixed the dreaded "screen turns off when streaming and playing at the same time" bug

Edit: sadly this doesn't help with old3DS that much. The old3DS hardware is too weak for video streaming AND input sending, so only one of them is recommended, unless you like a painful stuttery fuzzy mess.

This is just a test build, so unless you just can't wait to test out the bugfixes at the cost of possibly increased buggyness, I don't recommend trying these out.

Spoiler: Install instructions

PaintController: if you're using the cia version, uninstall the existing version (Linux Controller, not osu!Controller), reboot your 3DS (this is important!), then install out\PaintController.cia

FileStreamer: you don't really need to use this version, but you get better results with this one when used with the PaintController included in the download. The exe name is FileStreamer_test.exe, so it doesn't overwrite the existing one you may have.

Spoiler: Let me in!

Do not download without reading the text above!

https://puu.sh/FTeOU/2a65769297.zip

 

[MarioKartFan]

Thanks @Sono. You, @Gericom, @MasterFeizz , and @Robz8 seem determined to make the 3DS the greatest and most versatile handheld of all time.