People can find themselves needing an expert instructor for a variety of reasons. It could be for training, perhaps to prepare for doing surgery or improving physical fitness. They might be seeking a deeper understanding of a complex subject, or wish to explore a cultural interest, such as language, music, or art. They might need to relearn motor skills after disease or injury, a process that often demands intensive training or rehabilitative therapy, with real-time feedback from a trained therapist.

Sometimes, however, the expert we need might not be readily accessible to us, perhaps because demand for a certain kind of expert exceeds the supply, or because financial constraints, geographical barriers, or other factors restrict our access. And while a world of instructional books, videos, apps, and other online services exists to teach almost any imaginable skill, none can match the impact of realtime instructor feedback. 

Misha Sra, an assistant professor in the Computer Science Department at UC Santa Barbara, wants to ensure that such realtime guidance and feedback is available to anyone, anywhere, at any time, regardless of their circumstances. To conduct research in pursuit of this goal, she has been awarded the prestigious National Science Foundation (NSF) Early CAREER Award 2023. 

In the abstract for her five-year, $606,000 CAREER award, she wrote, “This project will provide foundational knowledge about interactions between humans and AI agents that can replicate the verbal and non-verbal behaviors of human expert trainers in the context of motor-skill learning.” The potential benefits include fostering inclusivity in motor-skill learning by reducing barriers and enabling affordable access at scale.

As for the award itself, she says. “This funding will be instrumental in launching a big new project, and  will help me as a junior faculty member establish my unique area of research, which, I believe, offers significant potential for societal benefit. On a personal level, the award validates the ongoing hard work in my lab and offers a sense of accomplishment and recognition that serves as motivation for future work. Approval from expert reviewers carries immense significance, providing reassurance that our work has intellectual merit and potential for broad impact.”

Sra’s research integrates AI with extended reality (XR), which refers to a spectrum of technologies that combine the physical and virtual worlds to create immersive and interactive experiences. XR includes virtual reality (VR), which creates a fully simulated digital environment; augmented reality (AR), which overlays digital content onto the real world; and mixed reality (MR), which blends virtual and real elements, allowing users to interact with both. AI, on the other hand, enables machines or computer programs to learn, reason, and make decisions based on data and algorithms. “This project focuses on harnessing AI and XR to enhance human physical abilities,” Sra, the John and Eileen Gerngross Chair in Computer Science, says. “It is aimed at bringing AI technology out from two-dimensional screens into the real world, offering much-needed support and guidance for learning and recovering motor skills.”

Motor tasks are challenging to design for because the person is often mobile and not sitting in front of a computer. That’s where XR technologies play a role, enabling motor-skill learners to maintain mobility, such as during fitness training, physical rehab therapy, or dancing. Head-worn XR devices facilitate interactions with AI, resulting in the new AI + XR interactive design paradigm, in which embodied AI agents (in Sra’s lab, they are called extended-reality agents, or ERAs) aim to replicate the effectiveness of learning with a human expert. 

A key component of such a system is multimodal guidance and feedback through visual, auditory, and haptic (touch) channels to provide the necessary explanations, demonstrations, and feedback. The ERAs in Sra’s project will be designed to recognize and respond to a user’s physical actions in or near realtime, appear and act human-like, and use verbal and non-verbal behaviors, such as eye gaze, head nods, facial expressions, body movement, and gestures, to communicate.

In one possible scenario that Sra has considered, someone who lives far from a city and is recovering after an injury or a surgery might be able to see a physical therapist only once every three months. In that case, she says, “The person might not be sure what to do between appointments to help speed up their own recovery, or, if they follow (often printed) instructions, they may not be sure if what they are doing is correct or will, perhaps, cause injury. Lacking the ability to track their own progress may also lead to a loss of motivation. 

“Now, imagine an AI system that can provide guidance and feedback in real time. Wouldn’t that be great?” she says. “You’d be able to speed up your recovery considerably by making the most of your time between appointments with your clinician/therapist or knowing how to continue once your therapy is officially over but you are far from fully recovered. “

Sra acknowledges that many knowledge gaps and challenges need to be addressed to make the realtime feedback system a reality, and she hopes that her lab can “address some of them in our work and address others through collaborations with researchers in areas such as computer vision, natural language processing, and cognitive science. Continuous advancements in these areas will contribute to the development of AI agents that can be used in the real world with far-reaching societal benefits.”