2 THE HEADS-UP COMPUTING PARADIGM

Three main characteristics define Heads-Up Computing: 1) body-compatible hardware components, 2) multimodal voice and gesture interaction, and 3) resource-aware interaction model. Its overarching goal is to offer a more seamless, just-in-time, and intelligent computing support for humans’ daily activities.

2.1 Body-compatible hardware components
To address the shortcomings of device-centered design, Heads-Up computing will distribute the input and output modules of the device to match the human input and output channels. Leveraging the fact that our head and hands are the two most important sensing & actuating hubs, Heads-Up computing introduces a quintessential design that comprises two main components: the head-piece and the hand-piece. In particular, smart glasses and earphones will directly provide visual and audio output to the human eyes and ears. Likewise, a microphone will receive audio input from humans, while a hand-worn device (i.e. ring, wristband) will be used to receive manual input. Note that while we advocate the importance of a hand-piece. Current smartwatches and smartphones are not designed within the principle of Heads-Up computing, as they require the user to adjust their head and hand position to interact with the device, thus are not synergistic enough with our daily activities. Our current implementation of a hand-piece consists of a tiny ring mouse can be worn on the index finger to serve as a small trackpad for controlling a cursor, as demonstrated by EYEditor [11] and Focal [33]. It provides a relatively rich set of gestures to work with, which can provide manual input for smart glasses. While this is a base setup, many additional capabilities can be integrated into the smart glasses (e.g. eye-tracking [21], emotion-sensing [ 12]) and the ring mouse (e.g. multi-finger gesture sensing, vibration output) for more advanced interactions. For individuals with limited body functionalities, Heads-Up computing can be custom designed to redistribute the input and output channels to the person’s available input/output capabilities. For example, in the case of visually impaired individuals, the Heads-Up platform can focus on the capability of audio output with the earphone, and haptic input from the ring mouse to make it easier for them to access digital information in everyday living. Heads-Up computing exemplifies a potential design paradigm for the next generation, which necessitates a style of interaction highly compatible with natural body capabilities in diverse contexts.

2.2 Multimodal voice + gesture interaction
Similar to hardware components, every interaction paradigm also introduces new interactive approaches and interfaces. With a head- and hand-piece in place, users would have the option to input commands through various modalities: gaze, voice, and gestures of the head, mouth, and fingers. But given the technical limitations [17 ], like being error-prone, requiring frequent calibration, and involving obtrusive hardware, it seems like only voice and finger gestures are currently the more promising modalities to be exploited for Heads-Up computing. As mentioned previously, voice modality is an underutilized input method that 1) is convenient, 2) allows us to free up both hands and vision, potentially, for doing another activity simultaneously, 3) is faster, and 4) is more intuitive. However, one of its drawbacks is that voice input can be inappropriate in noisy environments and sometimes socially awkward to perform [16]. Hence, it has become more important than ever to consider how users could exploit subtle gestures to employ less effort for input and generally do less. In fact, a recent preliminary study [ 31] revealed that thumb-index-finger gestures could offer an overall balance and were preferred as a cross-scenario subtle interaction technique. More studies need to be conducted to maximize the synergy of finger gestures or other subtle interaction designs during everyday scenarios. For now, the complementary voice and gestural input method is a good starting point to support Heads-Up computing. It has been demonstrated by EYEditor [11], which facilitates on-the-go text-editing by using 1) voice to insert and modify text and 2) finger gestures to select and navigate the text. When compared to standard smartphone-based solutions, participants could correct text significantly faster while maintaining a higher average walking speed with EYEditor. Overall, we are optimistic about the applicability of multimodal voice + gesture interaction across many hand-busy scenarios and the general active lifestyle of humans.

2.3 Resource-aware interaction model
The final piece of Heads-Up Computing is its software framework. This framework allows the system to understand when to use which human resource.

Firstly, the ability to sense and recognize ADL is made possible by applying deep learning approaches to audio [ 18 ] and visual [ 22 ] recordings. The head- and hand-piece configuration can be embedded with wearable sensors to infer the status of both the user and the environment. For instance, is the user stationary, walking, or running? What are the noise levels and lighting levels of the space occupied by the user? These are essential factors that could influence users’ ability to take in information. In the context of on-the-go video learning, Ram and Zhao [26] recommended visual information be presented serially, persistently, and against a transparent background, to better distribute users’ attention between learning and walking tasks. But more can be done to investigate the effects of various mobility speeds [10] on performance and preference of visual presentation style. It is also unclear how audio channels can be used to offload visual processing. Subtler forms of output like haptic feedback can also be used for low priority message notifications [29], or remain in the background of primary tasks [34].

Secondly, the resource-aware system integrates feedforward concepts [5 ] when communicating to users. It presents what commands are available and how they can be invoked. While designers may want to minimize visual clutter on the smart glasses, it is also important that relevant head- and hand-piece functions are made known to users. To manage this, the system needs to assess resource availability for each human input channel in any particular situation. For instance, to update a marathon runner about his/her physiological status, the system should sense if finger gestures or audio commands are optimal, and the front-end interface dynamically configures its interaction accordingly. Previous works primarily explored feedforward for finger/hand gestural input [ 9, 14 ], but to the best of our knowledge, none have yet to address this growing need for voice input.

An important area of expansion for the Heads-Up paradigm is its quantitative model, one that could optimize interactions with the system by predicting the relationship between human perceptual space constraints and primary tasks. Such a model will be responsible for delivering just-in-time information to and from the head- and hand-piece. We hope future developers can leverage essential back-end capabilities through this model as they write their applications. The resource-aware interaction model holds great potential for research expansion, and presents exciting opportunities for Heads-Up technology of the future.