Despite the inherent challenges that voice-interaction may create, researchers at the Penn State College of Information Sciences and Technology recently found that deaf and hard-of-hearing users regularly use smart assistants like Amazon’s Alexa and Apple’s Siri in homes, workplaces and mobile devices.
The work highlights a clear need for more inclusive design, and presents an opportunity for deaf and hard-of-hearing users to have a more active role in the research and development of new systems, according to Johnna Blair, an IST doctoral student and member of the research team.
“As smart assistants become more common, are preloaded on every smartphone, and continue to provide benefits to the user beyond just the ease of voice activation, it’s important to understand how deaf and hard-of-hearing users have made smart assistants work for them and the realistic challenges they continue to face,” said Blair.
In their study, Blair and Saeed Abdullah, assistant professor of information sciences and technology, conducted in-depth interviews with deaf users of smart assistants and collected survey data from individuals with mild to profound hearing loss.
They found that users from these populations — even those with profound deafness — used smart assistants to accomplish daily tasks, such as checking the weather, setting reminders and initiating GPS directions. They also identified common challenges these users face in their interactions with smart assistants, such as the default higher-pitched female voices used on many smart assistants being incompatible with hearing aids, and difficulty of use in public places with background noise that competes for their attention, such as nearby conversations or the average commotion in a grocery store.
“When we started working on this specific research problem, I initially had no idea how individuals with hearing loss would potentially use these smart assistants given they were relying mostly on audio and voice interactions,” said Abdullah. “So, it is really interesting to see how different people have quite different ways of engaging with smart assistants.”
Added Blair, “By empathizing with these users, we want to highlight new design features that can make smart assistants more accessible to hearing needs.”
Using smart assistants with all these constraints — in a quiet home, facing the device and dropping all other tasks — hinders the naturalistic, “imbedded in daily life” experience that people specifically seek from smart assistants. Therefore, the researchers suggest future design features that could better accommodate hearing needs, such as customizable voice characteristics; tangible information about common commands and error codes; and improved visual feedback, including more intuitive light patterns on devices or screen-based feedback.
“Hearing loss is not a straight line. There is a lot of fluctuation in both directions — hearing loss can worsen as people age, but hearing aids can also improve over time — so devices need to be able to evolve with the user right in step with these changes,” said Blair.
The researchers suggest recalibrating smart assistant voices through a means similar to how hearing devices are recalibrated by audiologists — to map out the ideal pitch range for individual users.
“With improvements, many saw promise in this type of technology to address other accessibility concerns in their daily lives, such as providing assistance in large work meetings or helping them fill in communication gaps when talking to others in loud public spaces,” said Blair.
She concluded, “Better accommodating the needs of hard-of-hearing users could also provide new opportunities for use, such as speech therapy, and speak to the needs of the wider population as they change over time. Making smart assistants more flexible and customizable for one group of users can help improve accessibility for all.”
The work was published in the December 2020 proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies (IMWUT). The Tronzo Medical Informatics Endowment partially funded the research.
