Hardware Emulator Project Creating a low-cost, dual screen emulation companion

[SecureBoot]

We all know that dual screen emulation is imperfect. Namely, the Wii U, 3DS, and DS all have some aspect of their control scheme that makes it difficult to have that authentic experience. Today, I have finally begun working on an idea that has been kicking around in my head for years. Here is what this project aims to achieve.

A handheld dedicated hardware device that has the following features:
- The wireless capability to decode a low latency video feed
- A full set of (detachable?) controls with built in gyroscope.
- Controllers have the ability to toggle between two profiles, one is a direct connection to the device. The other is a bluetooth (2.4gHz dongle?) to a host PC.
- 60 Hz capacitive touchscreen. ~ 7 inches
- Speakers and/or headphone jack
- Integrated capacitive stylus (but like, an actually good one. Not one of those squishy gross things)

What features I would like to include:
- Built in microphone
- Built in camera (can be disconnected if it is present at all)
- Display port over usb-c that causes device to output directly from host pc rather than stream wireless virtual display (decrease latency)

What features will NOT be included
-3D Display
-NFC reader

In theory, a device with these features would be able to play every game available on the Wii U, 3DS, and DS as intended by the developers with no exceptions.


The PoC was created a few weeks ago using an iPad, apple pencil, and set of joycon controllers. Here is a video of this in action



This setup is actually most of the way to what I'm looking for. With a 3d printed case to hold everything in place, it could even be a viable replacement for what I am making. I still have some major problems with this setup.

The main issue is the cost of admission. Between the iPad, Apple Pencil, and Joycon, you are looking at ~800 for a DS/Wii U screen. Now, if you already had all that stuff laying around that's one thing, but I intend to make this something that a gainfully employed person could reasonably purchase (i.e. <$200 USD all in). There are lofty long term ambitions of there being a large enough demand that I could manufacture custom components in bulk to drive the cost even lower, but for now, I will just settle for the BoM being as cheap as possible.

That brings me to the actual prototyping phase. After some research, I settled for the Radxa Zero3W as the heart of the device. After shipping, it came out to $45.22 USD and was the most cost effective way to get Wi-Fi 6, BT 5.4, and hardware decoding support. The largest downside to this device is its lack of good documentation. I spent most of the day just trying to get Android loaded onto its emmc, but eventually had to give up and stick to loading it on and SD card. Even then, I had to try two different Android images before finding one that actually supported the wireless chipset. This android image shows promise, but its wireless signal is extremely weak. This is not a hardware problem because I loaded up a debian image that had a stronger wireless connection than my phone, but on Android, even at 2 meters, the connection never allowed me to surpass 10mbps and usually stayed around 5.

Ultimately, I think I will have to create my own image of android that improves the wireless quality, but that will take some time and learning since I have never touched Android from a programming side. I'm sure the Android documentation will be better than the Radxa documentation though. My guess is that the Antenna gain isn't tuned correctly, so I hope it will be a simpler fix than I think (I doubt it though).

So obviously, fixing the wireless connection is the next step. Shortly after that, I hope to find a quality screen. I have begun some quick preliminary searching, and it seems like this might be the most expensive part of the device. I am hoping to keep it under $50, but I will need to find a good source to provide that.

I think the controller will be the hardest part because I have lofty goals for it. Getting a controller absolutely right is crucial for a handheld device. Especially if this does evolve into a commercial product down the line. Not only that, but I want to make sure that the user has the option to connect the controllers to either the device (for remote play) or the host PC (to unlock the gyroscope because I haven't found any software that consistently allows you to send gyroscopic data over a remote desktop app). We'll cross that bridge when we get there though because we are way far away from having to tackle that problem.

So why make a post for the first baby steps of a project that will take months, or more likely years to complete? Well, one reason is to document progress, but the other reason is I want to gather some preliminary feedback. When presenting the idea to my soon-to-be wife, she seemed hesitant that this would be of any interest to people. I don't know how right she is, but since this is a crew who are generally pretty big on emulation, would you be interested in a device like this for yourself? Either to build on your own or purchase from a manufacturer. What would be the ideal cost for something like this (again, either in material cost if you were building your own or if you were purchasing from a retailer)?

Also, this is a big project and I am already way in over my head. That doesn't discourage me from it, but it does encourage me to seek help. If anyone here takes up any interest and would like to provide advice or feedback, feel free to leave that here or PM me!

 

[IC]

I feel like the most effective way to do this on a budget would be to simply find a suitable mass-produced model of Android tablet, which includes all the needed features. So, adequate wireless connectivity hardware (which is rarely a problem on a mobile device like a tablet or smartphone), adequate video processing power, quality display at an adequate size and resolution, and possibly even a built-in stylus pen digitizer (this requirement might make the price much higher, though).
With a mass-produced android tablet, you wouldn't have to worry designing your own case for the device, and a mass-produced tablet will probably be a lot cheaper than buying the "computer" part and standalone display panel separately, in low amounts. It will also already include the other things: audio system with speakers and headphone jack, microphone, camera, perhaps even NFC.

There are lots of detachable gamepad devices made for connecting directly to smartphones and tablets, such as Gamevice which existed before the Nintendo Switch and even attempted to sue Nintendo for allegedly stealing their invention! Some are made for specific models of devices, and some are universal and adjustable. With some research and buying some devices for testing, you might be able to find one which is just right for this project, in terms of button arrangement, build quality, etc.

You should then create your own companion software for this exact purpose. That being an Android app for the device, and desktop PC Windows/Mac/Linux software to communicate with the Android device. If you (or someone you hire) write some software from scratch for this exact purpose, rather than messing with existing remote desktop software, it will surely be a lot more robust and you'll be able to solve a lot of software issues/incompatibilities.

The software doesn't need to be advanced, just simple enough to connect with the Android app with the software on the PC (probably simply over the Wi-Fi network), stream the audio and video of the DS/3DS/WiiU bottom screen from the emulator on the PC with adequate quality(bitrate) and latency, and send the controller inputs from the device to the emulator on the PC. With your own software, you'll be easily able to add support for all the other controller features, like the touchscreen, microphone audio, motion controls, and possibly even the cameras on the Android device. With general-purpose software, it would be pretty hard to get all of these things working just right. However, you might also end up having to modify the code of the DS/3DS/WiiU emulators in order to have them support receiving all of those various inputs from the Android device.

If you do all of this, you could then share your research and your software with others, so they can buy the best Android tablet and detachable gamepad for this exact purpose, based on your research, and then use your software made with this exact purpose in mind, which at that point should require little setup and work fairly effortlessly.

I don't know if there would be enough demand for this to go down the route of making a mass-produced device, and doing that would require you to spend tons of money on it, so for now it's better to just start prototyping, like you're already doing. Good luck with your project.

 

[SecureBoot]

I feel like the most effective way to do this on a budget would be to simply find a suitable mass-produced model of Android tablet, which includes all the needed features. So, adequate wireless connectivity hardware (which is rarely a problem on a mobile device like a tablet or smartphone), adequate video processing power, quality display at an adequate size and resolution, and possibly even a built-in stylus pen digitizer (this requirement might make the price much higher, though).
With a mass-produced android tablet, you wouldn't have to worry designing your own case for the device, and a mass-produced tablet will probably be a lot cheaper than buying the "computer" part and standalone display panel separately, in low amounts. It will also already include the other things: audio system with speakers and headphone jack, microphone, camera, perhaps even NFC.

There are lots of detachable gamepad devices made for connecting directly to smartphones and tablets, such as Gamevice which existed before the Nintendo Switch and even attempted to sue Nintendo for allegedly stealing their invention! Some are made for specific models of devices, and some are universal and adjustable. With some research and buying some devices for testing, you might be able to find one which is just right for this project, in terms of button arrangement, build quality, etc.

You should then create your own companion software for this exact purpose. That being an Android app for the device, and desktop PC Windows/Mac/Linux software to communicate with the Android device. If you (or someone you hire) write some software from scratch for this exact purpose, rather than messing with existing remote desktop software, it will surely be a lot more robust and you'll be able to solve a lot of software issues/incompatibilities.

The software doesn't need to be advanced, just simple enough to connect with the Android app with the software on the PC (probably simply over the Wi-Fi network), stream the audio and video of the DS/3DS/WiiU bottom screen from the emulator on the PC with adequate quality(bitrate) and latency, and send the controller inputs from the device to the emulator on the PC. With your own software, you'll be easily able to add support for all the other controller features, like the touchscreen, microphone audio, motion controls, and possibly even the cameras on the Android device. With general-purpose software, it would be pretty hard to get all of these things working just right. However, you might also end up having to modify the code of the DS/3DS/WiiU emulators in order to have them support receiving all of those various inputs from the Android device.

If you do all of this, you could then share your research and your software with others, so they can buy the best Android tablet and detachable gamepad for this exact purpose, based on your research, and then use your software made with this exact purpose in mind, which at that point should require little setup and work fairly effortlessly.

I don't know if there would be enough demand for this to go down the route of making a mass-produced device, and doing that would require you to spend tons of money on it, so for now it's better to just start prototyping, like you're already doing. Good luck with your project.

Thanks for your perspective!

There are definitely two distinct routes to go down here. Build out hardware to interface with existing software, or build out software to interface with existing software.

While I am not even close to ruling out your suggestion, there are several concerns I have with using an existing tablet. For starters, I have not been able to find one that can be obtained new cheaply enough that isn't a piece of crap. Bad screens, bad wireless cards, OSs that are stuffed with bloatware and lead to overall bad performance and instability, poor battery heavy, etc. Some of this stuff can be circumvented with a custom android rom, but a lot of it can't.

Another issue is latency. Now typically, the latency between the local network is pretty good for both controls and display, but there is definitely still noticable latency if you're spending a lot of time on the touchscreen. I would like to multiplex the connection to the display and the controls so that a direct connection can be made to the display and remove any wireless latency.

The last thing is software maintainability. To develop my own moonlight client, or even entirely new streaming service would mean I would have to maintain my own moonlight client indefinitely. I'm just not at the stage in my life where I can do that. I'm getting married in two weeks, and between my job and marital responsibilities, there just isn't enough time to maintain a FOSS project. The hardware, on theory, would only need to be designed once.

Please understand that I am not disregarding your suggestion. In fact, I think it is a great idea! There are just a lot of remaining concerns I would have to address. With the hardware approach, these concerns can be built away from the ground up. I will definitely do more research into the software approach though.

 

[IC]

I feel that even with the route of building your own hardware, you wouldn't get away from having to write some custom software for your hardware, software that would be specifically catered to this purpose. I don't necessarily think that the software would have to be maintained in the future, I think that could also be a thing that's only designed once. Well, unless you'd be forced to modify the code of the emulators, then you'd somehow have to apply your patches to future updated of the emulators, but if I was doing this myself, I would also be avoiding modifying the emulator code in any way, since that would add a lot of complexity to the project.

I don't know what moonlight is, but an entire streaming "service" is definitely something you wouldn't have to develop. Sending a video feed over LAN to a different device (peer-to-peer) should be a relatively simple thing, compared to an entire streaming service which depends on remote servers on the Internet. What I had in mind is a fairly simple set of programs for the PC and your device to send the needed data between the device and the emulator on the PC. It wouldn't need any advanced streaming service features, but just exactly all the features you'd need for a device like this, for local network video/gamepad streaming.
I just doubt that there's already any software out there that would fit all your needs in a single reliable package - streaming video/audio from the PC, and sending back to the PC the gamepad input data, motion/gyroscope data, touchscreen input data, microphone audio, and possibly even video from a built-in camera.

If you're going to continue building your own hardware, I wish you all the best of luck with that. I just feel like that won't replace having to create your own software - rather, designing your own hardware would be an addition to designing your own software.

Sidenote: I myself also recently finished a pretty large electronics hardware project - in that project I also had to write my own firmware for the microprocessor. If you're curious, you can read about it in my blog here on GBAtemp. I didn't write much about the firmware in the blog entry, but writing the firmware was in fact a pretty complex piece of the project as well.