The Space in Front of the Screen as the UI

Ghosts of the Mechanical Age

Most interactions with computers are based on a keyboard invented in Milwaukee in the early 1870s and a mouse created in Menlo Park in 1964. Both deserve their place in history, but they've also shaped us: for 150 years, we've had to put our hands on mechanical devices to interact with our machines. Form follows hardware. Today, AI and ML are opening doors for designers exploring interfaces organized around more natural intfaces.

Designing For the 3-D Space Humans Inhabit

This project explores moving the center of gravity of HCI design out of the world of screens and into the 3-D space that humans inhabit, which is a natural place for interations: People have been expressing themselves with speech and gestures for millennia.

How it Works

The language model is local

The app uses a locally-running version of Google’s language model, Gemma2-b. The LLM is accessed through Chrome’s API, enabling incredibly fast responses.

Gestures are JavaScript Events

Using MediaPipe’s open-source gesture capture library¹, I created a JS event handler that allowed me to rapidly experiment and fine-tune interactions in code.

Designing Multi-Modal Interaction Loops

The end-to-end gesture-to-chat cycle involves several interlocking loops of interaction². The outer loop has three sections:

1) Activation --> 2) Dialog --> 3) Resolution

Each of these, in turn, have their own micro interaction loops that guide the user to the next action. All loops are an interplay of speech, gesture, and system feedback through graphics and motion.

The activation loop

The activation loop starts with an invitation / affordance to the user to raise a hand, which triggers another signal that conveys "live mic".

The dialog loop

The user has now gone from intent (“I want to chat”) to raised hand (action) to live mic (system responded to action). Now they can speak. Once a word is detected, the next loop starts.

The Close / Resolution Loop

Once the user is done with their dialog, they can close the session with a thumbs-up.

Key Challenges and Learnings

human Design judgement is more relevant than ever

Boiled to its essence, design is about externalizing ideas, evaluating them, and iterating based on learnings. That hasn’t changed; it’s just in a new context.

I relied a lot on my muscle memory and general interaction design knowledge accumulated over the years. I also augmented this with HCI research, such as Don Norman’s work around interaction loops. I’ve become comfortable making a call about a design trade-off, writing code, then testing.

Tinkering as discovery

For a self-directed project like this, I find myself starting off by tinkering with new technology like gesture capture. After that, my design process is a combination of sketching, coding in Python, visual refinement, coding in React, etc. But there is less structure than, say, trying to follow a double diamond process or anything like that.

Conversational uncanny valley

The speed at which the local LLM responded was too fast. It felt like the LLM was starting to answer your question before you finished asking. There was no hard rule, it just felt weird.

Adding a 500ms pause before the LLM’s response fixed it. I believe this is because we learn how a “normal” conversation is paced from talking to other people who need a heartbeat to process before speaking. That’s pure design judgement.

System Feedback for Voice and Gesture

I’ll confess I’ve looked at animations in UIs with skepticism. Most seem excessive and often distracting. But for this prototype, I learned Framer Motion out of necessity: When designing naturalistic UIs, you can’t assume the user is holding the device or within range of the trackpad or mouse. Also, a gesture event does not have a graphical control (e.g., a button) as its trigger that would anchor the user's attention to a space on the screen. Animations became an important part of conveying system status by drawing attention and conveying changes to state.

Further Explorations

beyond chat: User intent and Agent Delegation

What if instead of just answering questions directly, the local LLM’s job was to detect intent, then reacting appropriately? If the user asks about the weather, it first detects user need (“Will it rain today”), then makes a function call to a weather API before formulating an answer.

I wrote a quick test that takes text (e.g., a user question) input and uses evals to try to detect fit into four categories: location, person, weather or other.

Modality-Agnostic Interaction Design

I dream of fine-tuning local LLMs to understand user intent through gestures, speech, clicks, typing, or whatever. The local fine-tuned LLM would understand user intent in context of system state.

I'm drawn to this because LLMs are good at semantic stuff: System state and gestures can be codified in words. Think of JS components with meta data that describe what they are and what they do, with evolving context provided by a state machine.