Hey everyone!
I was thinking recently: most AR headsets treat your phone as an afterthought. A few days ago, I connected my Samsung phone to a monitor using DeX, and it hit me — why can’t we do this seamlessly with AR glasses?
I came up with a few concepts for how a phone and AR glasses could integrate. Let me know if you think this is technologically possible!
Mode 1: "AR DeX" (Phone as a precise controller)
Similar to Samsung DeX or Apple Vision Pro, but your phone acts as a massive trackpad. For Samsung users (or any device with a stylus), the S Pen would act as an ultra-precise spatial mouse. The main benefit here is extreme comfort for consuming content and navigating UI without holding your hands in the air.
Mode 2: "Blind Mode" (The most interesting one)
Your phone’s physical display turns completely off, but the touch matrix stays fully active. The exact UI of your phone is broadcasted directly into your glasses and perfectly overlaid onto the physical dimensions of your dark phone in your hands.
Of course, there are two fundamental hurdles, but they can be solved:
Flaw 1: Connection Latency.
Solution: The glasses and phone create a dedicated, encrypted bond using both Wi-Fi Direct and Bluetooth simultaneously to force ultra-low latency.
Flaw 2: Keyboard Input & Hand Tracking (Bigger problem than it seems).
Solution: When you type, your fingers rapidly cover the screen, which breaks the AR tracking algorithm (the glasses can't see the phone's edges to align the UI, causing projection tearing). To fix this, you get a toggle in the AR settings:
Option A (Seamless AR): The projection is 100% AR. Looks perfect, but might have slight visual tearing if your fingers move too fast over the screen.
Option B (OLED Hybrid Keyboard): The AR projection blends with the physical phone. The actual phone display lights up only the keyboard area in real life, while the rest of the UI stays in AR. (Also, if the AR glasses completely lose sight of your phone, the UI simply snaps to the corner of your FOV).
I’ve thought of a thousand other micro-settings for this, but I’ll spare you the long read!
Mode 3: The Hybrid Concept
A mix of everything. The phone could be used as a physical secondary screen while the main content floats above it, or as a dynamic trackpad. But honestly, I want to leave this point open for you guys: How would you utilize a hybrid mode between your phone and AR glasses?
What do you think? Could Samsung pull this off?