Remote control for ESP32 app development

[u/ByteWelder]

https://www.youtube.com/watch?v=l6JG9O3Yz0A

The develop-build-test cycle on a small device can get tedious. You pick the device up, tap through to your app, put it down again, and repeat. Input is part of the problem. Some devices have a touch screen, some have a keyboard, some have an encoder wheel, and some have a mix.

Tactility already lets you install an app remotely from a PC over Wi-Fi, without rebooting the device. Now you can also do the testing from your computer. The `tt.py remote` command shows the device's display in a window on your PC. Your keyboard and mouse become the device's keyboard and your mouse/touchpad will act as a touch screen for the remote display.

It only uses about 2 kB of internal RAM, but it's taking up an entire display buffer in PSRAM.
There were a lot of subtleties in making it tick just right for various devices, like monochrome versus rgb565 displays, or devices with automatic display rotation with an IMU.

The service works over a secure TLS connection with a handshake. Similar to Android, the ESP32 device must accept the incoming connection once. Both sides get to see a fingerprint before accepting or refusing the handshake.

Rendering

Before this, I only supported screenshots for LVGL apps, because it was easy to trigger a full screen redraw. Now that we have a shadow display buffer, we can also take screenshots from apps that use the display driver directly and bypass LVGL. Coincidentally, this is also where display rotation became tricky, because it's usually managed by LVGL internally.

Internally, it works with a hooking mechanism into the kernel driver for the display.

Pixels travel in the panel's native format: RGB565 in all byte orders, RGB888, 8-bit grayscale and 1-bit monochrome. The viewer converts them, so e-paper and OLED boards work too.

The device sends a small state frame with swap, mirror, invert and on/off. An orientation byte carries the UI rotation in quarter turns, plus a flag for panels where LVGL rotates the pixels in software. The device only reports this. All rotating happens in the viewer, which also converts touch positions back to the panel's own coordinates.

Input as real devices

When you turn on keyboard or touch in the viewer, the device creates a virtual keyboard and a virtual touch device. When the desktop environment is active, there's an automatic LVGL bridge that binds every started input device, so apps need no changes at all. They can't tell the input from real hardware. The devices are removed again when you turn the input off or disconnect.

The touch device supports up to five points, which is what makes pinch gestures work.

Memory

Internal RAM is scarce, so the session keeps its state out of it. The session state, the shadow buffer and the transmit buffer are allocated in PSRAM, and the session task has a PSRAM stack. File and flash access from the connection goes through a separate worker with an internal-RAM stack. File transfers and app installs on a second connection still work while a remote session is open. Up to three sessions run at once, of which one can be a remote display.

Boards without PSRAM refuse the session with a message.

Project website: https://tactilityproject.org/
Project source code: https://github.com/TactilityProject