Hacking WiiUTablet - use the Gamepad as PC drawing tablet

[PewnyPL]

A 2-part application, a HTML5 webpage to run on WiiU and a Java server program to run on target PC. Currently it's in very early alpha, but it can already establish conenction between Wii U and a PC and click+drag the cursor around.

To run it, two things must be changed:

• host variable in WiiUTablet.js so it points to the target PC
• resX and resY static variables in Main.java so they correspond to PC's screen resolution

First the server must be started, then the webpage loaded on the Wii U. If for whatever reason the connection is lost, simply refreshing the page will do the trick.

However, for some unknown reason there seems to be some lag on the mouse movement. While not a single packet is lost, the smooth and quick movement of stylus on the gamepad (ie. drawing a circle) creates a jagged line. It seems to be a problem of Java's AWT Robot class, but I may be wrong, could anyone check this?
Currently the jagginess is under control, sadly, the latency between Wii U and PC started to be noticeable (a split second delay between touch on the gamepad and the click showing on PC), this however seems impossible to fix.

Here is the link to Github: https://github.com/PewnyPL/WiiUTablet

Note: This project was only tested on Windows, however it should work on every Java-enabled system. Also, the pre-compiled binary attached here is designed with 1920x1080 resolution in mind, if your computer has a lower one it may cause strange behavior when moving beyond certain point on the touchscreen.

EDIT: Reuploaded the file, apparently I forgot the library.

 

[laudern]

Would anyone actually want this? Sure I can understand the challenge of accomplishing such a feat, but really? A relatively small, touch resistive, single touch screen, is that the ideal setup? Just go buy a pen input device or a MS surface.

 

[PewnyPL]

Would anyone actually want this?

Someone who already has a Wii U but doesn't want to spend money on a drawing tablet (yet). In some parts of the world it's not easy to get one, as either the only ones readily available are the more expensive ones, or if you want to order one from abroad, the shipping costs are ridiculous.

 

[yuyuyup]

A 2-part application, a HTML5 webpage to run on WiiU and a Java server program to run on target PC. Currently it's in very early alpha, but it can already establish conenction between Wii U and a PC and click+drag the cursor around.

To run it, two things must be changed:

• host variable in WiiUTablet.js so it points to the target PC
• resX and resY static variables in Main.java so they correspond to PC's screen resolution

First the server must be started, then the webpage loaded on the Wii U. If for whatever reason the connection is lost, simply refreshing the page will do the trick.

However, for some unknown reason there seems to be some lag on the mouse movement. While not a single packet is lost, the smooth and quick movement of stylus on the gamepad (ie. drawing a circle) creates a jagged line. It seems to be a problem of Java's AWT Robot class, but I may be wrong, could anyone check this?

Here is the link to Github: https://github.com/PewnyPL/WiiUTablet

Note: This project was only tested on Windows, however it should work on every Java-enabled system.

Believe it or not, this sounds great to me (I guess it's just not good enough for some people.) Are you aware of http://libdrc.org/

 

[PewnyPL]

Are you aware of http://libdrc.org/

I am aware of this and I SO would rather use it, but this is sadly a Linux only solution, and while I have no problems using it myself, I'd preffer to do it on Windows.

 

[endoverend]

I get the javascript to run on the Wii U browser, but unfortunately nothing happens to the pointer on the PC even after changing the host variable to the PC's local IP and running the .jar file. Maybe I need to change the port?

 

[PewnyPL]

I get the javascript to run on the Wii U browser, but unfortunately nothing happens to the pointer on the PC even after changing the host variable to the PC's local IP. Maybe I need to change the port?

If the Java app compiles and starts without throwing an exception, then it should work. Make sure it's not blocked by a firewall, and that on the Wii U gamepad you see the touchscreen variables.

 

[Deleted User]

Whoa, cool for sharing something like this.
Though, er, how do I exactly run the java files?
I have no experience with Java.

 

[endoverend]

If the Java app compiles and starts without throwing an exception, then it should work. Make sure it's not blocked by a firewall, and that on the Wii U gamepad you see the touchscreen variables.

I double click the java_websocket.jar file, and absolutely nothing happens. I get the variables on the gamepad screen. Is there something I'm doing wrong?

 

[PewnyPL]

I double click the java_websocket.jar file, and absolutely nothing happens. I get the variables on the gamepad screen. Is there something I'm doing wrong?

java_websocket.jar is just a library. I did not add a pre-compiled binary there, adding it now to the first post.

Edit: Uploaded

 

[endoverend]

Was able to compile & run it through eclipse, thanks for the advice. However this is impossible to use for drawing in its current iteration (lines are VERY jagged and you have to take a guess as to where you're actually putting the mouse on the screen lol), but it's still cool and a good idea.

 

[PewnyPL]

Was able to compile & run it through eclipse, thanks for the advice. However this is impossible to use for drawing in its current iteration (lines are VERY jagged and you have to take a guess as to where you're actually putting the mouse on the screen lol), but it's still cool and a good idea.

Yeah, I know, I have no idea what causes those jagged lines as, again, no packets are dropped, everything sent from Wii U gets to the PC, it's like the mouse movement methods can't keep up. As for the 'guessing', as far as I know all drawing tablets work exactly like that, if you press the pen on a point in 1/4th of the X and Y location on the tablet, it's like if you pressed it in 1/4th X and Y pixel on your screen, so it's replicating that.

 

[endoverend]

As for the 'guessing', as far as I know all drawing tablets work exactly like that, if you press the pen on a point in 1/4th of the X and Y location on the tablet, it's like if you pressed it in 1/4th X and Y pixel on your screen, so it's replicating that.

Lol, I've never tried a real drawing tablet, so I wouldn't know, but that seems really imprecise to me. What I meant to say is that there's no way to move the cursor without clicking. I heard most draw tablets allow you to hover over them with the pen to move without clicking, but that's obviously impossible with the gamepad.

 

[Bu2d85]

Lol, I've never tried a real drawing tablet, so I wouldn't know, but that seems really imprecise to me. What I meant to say is that there's no way to move the cursor without clicking. I heard most draw tablets allow you to hover over them with the pen to move without clicking, but that's obviously impossible with the gamepad.

If the program recognizes button presses it could be setup to not click when holding the left trigger or something. That way you could move the mouse without clicking.

 

[endoverend]

If the program recognizes button presses it could be setup to not click when holding the left trigger or something. That way you could move the mouse without clicking.

It can't recognize button presses, it's just a web browser and the buttons are the gamepad are assigned for other things. A JS application wouldn't be able to detect that.

 

[PewnyPL]

It can't recognize button presses, it's just a web browser and the buttons are the gamepad are assigned for other things. A JS application wouldn't be able to detect that.

Actually, JS can recognize every button on the Wii U gamepad, even the home button (even tho it immediately exits the app), I just didn't implement that yet in this.

 

[PewnyPL]

Managed to get the jagginess under control, sadly, the latency between Wii U and PC started to be noticeable (a split second delay between touch on the gamepad and the click showing on PC). It may be impossible to fix however, so this project may get scrapped...
Github and first post have been updated.

 

[Pelayo]

Well, I don't know exactly which file do I have to run to start the server... I tried running WiiUTablet.jar(nothing appears), but I just can't get in the page on the Wii U browser. It gives me an error. What am I doing wrong? I already changed the text in the .js and .JAVA files... Is it maybe the location of the .jar file(I did put it in the main folder)?
Anyways, this looks great, have you thought of adding this to USendMii as a feature?

 

[PewnyPL]

Well, I don't know exactly which file do I have to run to start the server... I tried running WiiUTablet.jar(nothing appears)

That's weird, it should start without any problems, unless the port used by it is already taken. Maybe you have something wrong with Java?

but I just can't get in the page on the Wii U browser. It gives me an error.

What error does it give you? Did you host the .html and .js file on a webhosting? The .jar file doesn't start any web servers.

Anyways, this looks great, have you thought of adding this to USendMii as a feature?

If it was open source, I'd try.

 

[NhgrtPlayer]

Hi, how are you supposed to use this please ?