User Experience Challenges in Augmented Reality Glasses

AR faces a challenge that most developers never encounter… reality refuses to stay still. A storefront changes its display weekly. A stadium transforms from a baseball field to a concert venue overnight. Lighting shifts from harsh noon sun to soft evening glow. Crowds ebb and flow, creating constantly changing occlusion patterns. Traditional computer vision often expects consistency. It matches predefined patterns against stable environments. But real spaces are fluid, dynamic, and unpredictable. We learned this early in our journey while testing at major venues. Out-of-the-box visual positioning systems (VPS) often perform well in static, controlled conditions but struggle when confronted with even minor real-world changes…. like a sudden holiday decoration or unexpected weather shifts. This forced us to rethink AR from the ground up. Instead of trying to catalog every possible variation, we built systems that understand the fundamental structure of spaces. They recognize when change is meaningful versus superficial. Today, our AR experiences maintain accuracy even as environments evolve. They can adapt to seasonal transitions, temporary changes, and dynamic lighting conditions without requiring constant updates. Perfect isn't real. Real is what matters.

Excited to share research from our lab at Georgia Tech, in collaboration with researchers at Google and the University of Toronto! As we get closer to monocular AR or HUD glasses that actually look like everyday eyewear, a key HCI question is where to position a monocular HUD so it’s comfortable to consume content—without being distracting or interruptive, especially with proactive AI. This review paper brings together decades of research, including recent work from our lab post-Google Glass—especially by my team over the past four years. We synthesize findings across performance, comfort, interruptions, and social perception, including design recommendations in Section 10. Read the preprint: https://lnkd.in/emjNqYqU Please share with any teams building in this space! Big thanks to all my co-authors: Ethan Kimmel, Katherine Huang, Tyler Kwok, Yukun Song, Sofia Vempala, Blue (Georgianna) Lin, Ozan Cakmakci, and Thad Starner #AR #HUD #AugmentedReality #Wearables #HCI #UX #ARglasses #HumanFactors #Hypernova Meta Google Snap Inc. Samsung Electronics Qualcomm Vuzix Corporation Even Realities XREAL Lumus Ltd. Applied Materials Dispelix DigiLens Inc. Avegant JBD

I've been noticing a lot of unconstructive reviews of the Apple Vision Pro. I wanted to take a different approach by offering a few suggestions that I think would improve the overall user experience. 1. Rotating windows on the Vision Pro feels awkward - you have to pinch and drag the Window Bar while tilting your head. What if there was a better interaction? Introducing the Rotate Bar - pinch and drag the Rotate Bar while moving your hand to rotate a window. 2. I spend too much time organizing my workspace on the Vision Pro and when I change locations, I have to do it over again. What if you could save location-based workspaces? In this mock, a workspace automatically populates at its saved location. 3. Sometimes it’s awkward to capture experiences from the POV cameras on the Vision Pro. What if you could capture AR scenes from other angles? Here’s a mock that uses an iPhone to capture the same scene that’s on the Vision Pro. 4. I love Vision Pro's Desktop Mirroring, but it's overwhelming to navigate inside a window with mouse/keyboard and everything else with pinch/gaze. What if your mouse could also control system level UI? Here's a mock up of using a mouse to adjust a window's position and scale. 5. The Vision Pro is difficult to use when on the go - windows stay anchored to the ground rather than following you when walking around. What if you could pin any window to the screen? Gaze at the Window Bar and double tap to pin a window in screen space.