Empathics Project
Emotion Recognition Smart Glasses
Problem
Autism Spectrum Disorder: Developmental disability characterized by difficulties with social interaction and communication
Social-Emotional Agnosia: Disability characterized by difficulties in perceiving emotions of others.
Emotion awareness difficulties in autistic adults makes developing relationships challenging, and often leads to alienation and social withdrawal. One result of this is an unemployment rate estimated as high as 90%.
Problem Statement: How may we help autistic adults save time and energy in perceiving the emotions of others in casual conversations, allowing them to express themselves and communicate with neurotypical people easier?
Solution
We developed an app for the Vuzix Blade Smart Glasses that uses an emotion recognition AI algorithm to determine the emotion of the person a user is speaking with, and communicates the emotion to the user in real time with the glasses’ AR capabilities.
1) The glasses use cameras and microphones to capture visual and audio data during conversations.
3) The glasses then displays the detected emotion visually on the lens or vibrates to notify the user in real time.
2) The data is sent to the cloud, and our machine learning algorithm uses it to determine the emotions of the conversation partner.
4) The user can acknowledge and respond to their conversation partner’s emotional cues
My Process and Role as User Research Lead
Engineering and Design Team: Tian Feng, Leann Huang, Shine Lin, Christopher Tran
Mentors: Richard Zaragoza, Jenna James, and Emily Admunson (UX Design, UX Research), Chris Emura and Jay Thiagarajan (Engineering), Linda Wagner, Blake Hannaford, and John Raiti (Faculty Mentors)
Generative Research to understand communication, the health industry, and autism
Methods: Interviews with users and SMEs, literature reviews, participation on online communities
Usability Tests to answer design questions
Methods: Usability tests with users in testing rooms
Prototype Evaluations to deepen our understanding of user needs and usability
Methods: Observations and interviews
Challenge 1: Complexity in Medical Communication
We started with this design question: “How can we improve patient health through improving communication between doctors and patients?”
We interviewed 5 different doctors and 5 patients to research pain points in communication between them.
We interviewed a medical student, a medical communications expert, a nurse practitioner, and a clinical staff member to research the context around doctor’s work routines and communication tendencies.
Initially, we determined an opportunity in improving a doctors’ awareness for their patients’ emotions and ability to emotionally support them. Through our research, we found that communication was dependent upon various factors outside of a doctor’s abilities; most notably the clinical regulations the doctors work under. We had many questions surrounding the doctor’s desire to use a technology like this in their already-busy lives.
Outcome:
Upon circulating our research concept with our advisers and networks, we determined that our technology to improve emotion awareness could have a greater impact for many autistic adults who are medically unable to perceive others’ emotions. We decided to pivot our project to support this under-voiced community.
Challenge 2: Output Design
I conducted usability tests to determine how to design our product to communicate to the user clearly and non-disruptively. We tested with 3 sensory output formats: haptic vibrations, visual projections on the glasses lens, and audio feedback. We started this project with only 2 outputs, “positive” and “negative“ emotions, for simplicity.
Haptic Test:
We had 5 sets of 2 people have a conversation, one would be wearing a bracelet that buzzes whenever emotion is detected in the other
After 5-10 minutes of conversation, we briefly interview each person over the experience, and had them rate how clear and disruptive the signals were
Icon Test:
With 10 users, we showed them different sets of icons and asked them to state the words that intuitively came to their heads
We analyzed which icons were more uniformly described as “positive” and “negative“
Audio Test:
With 10 users, we played sets of jingles, and asked them to label whether they sounded positive, neutral, or negative
We calculated which jingles and tones were consistently labeled
Outcomes:
As output frequency increases, the disruptiveness increases. We decided to impose a frequency output limit.
While haptic outputs were obvious to sense, visual outputs were more clearly descriptive. Visual outputs could be misinterpreted unintentionally, such as “good” and “bad“ instead of “positive” and “negative”. Audio signals were the least clearly descriptive. A combination of outputs seemed most effective.
Autistic adults show varying environmental hypersensitivities, and the adjustability of the Vuzix Blade to ouput in all three of these different formats solidified our decision to use it as the hardware platform
Challenge 3: Societal Expectations
After prototyping, I met with 3 of our autistic adult users to have them evaluate the prototypes. Two of them alerted us that the concept seemed non-inclusive, and supported behavioral masking.
Behavioral masking is the act of changing one’s behavior to meet a group’s expectations. In this case, the society expectations are for autistic adults to change their behavior to be like neurotypical behavior. The most recognized assistance for autistic people is behavioral therapy (teaching them neurotypical norms), and many autistic adults are exhausted by this. Our users led us to understand that as our device was intended solely for autistic adults to wear and use, and presented no responsibility to neurotypical people to change their behavior, this concept was subtly supporting this non-inclusive perspective.
They introduced us to the thought models of disability. Currently, the medical model is the most popular, seeing autism as a disease, and autistic adults as victims that need constant help. Alternately, autistic adults prefer the social model perspective of autism, advocating that they are a part of diversity. Under the social model, an approach to change society’s perspective of their behavior would be more effective and desirable.
Outcomes:
To support inclusivity, I recommend a pivot in the product concept towards an environmental augmentation, rather than a wearable device for autistic adults. Environmental augmentations, such as ramps for disabled people are changes that effect everyone, and therefore, are not socially separating. I recommend designing a product that moves society’s bias towards the social model perspective of autism. We may even consider designing this device to be used by everyone to have more empathy in conversations. Ultimately, to ensure inclusivity in the solution, we must continue collaborative design with our users.
Take-Aways
1. Awareness of the potential impacts of the product on the user environment and society, in addition to an understanding the user’s needs and desires
In order to design this product for autistic adults, we first had to learn about behavioral masking, and how different products and therapies force people to mask.
2. High prioritization of finding access to user populations and having deep user participation within the design process
We had lots of difficulty reaching autistic adults; they are a community that is skeptical to neurotypical-led efforts to help. Once we finally were able to evaluate our prototype with one, we learned that our product was not desirable.
3. Awareness of pivoting opportunities, and ability to assess the resources and consequences of the project pivot.
We pivoted our project to help autistic adults because the complexity of medical communication and medical innovation.
4. Deepened understanding of Smart Glasses technology, AR, machine learning, and connected devices
We worked with the Vuzix Blade Smart Glasses, and evaluated that the real-time AR capabilities allows for many new opportunities in improving communication and work performance. Some limitations include the social stigma of recording, physical discomfort, and battery limitations. I think that currently, it can only be effectively used in controlled locations where transparency of use and ethical use can be ensured.