Toward User-Driven Sound Recognizer Personalization with People Who Are d/Deaf or Hard of Hearing

By Steven M. Goodman, Ping Liu, Emma J. McDonnell, Jon E. Froehlich, Dhruv Jain. 2021

37 citations recorded by the source corpus.

Automated sound recognition tools can be a useful complement to d/Deaf and hard of hearing (DHH) people's typical communication and environmental awareness strategies. Pre-trained sound recognition models, however, may not meet the diverse needs of individual DHH users. While approaches from human-centered machine learning can enable non-expert users to build their own automated systems, end-user ML solutions that augment human sensory abilities present a unique challenge for users who have sensory disabilities: how can a DHH user, who has difficulty hearing a sound themselves, effectively record samples to train an ML system to recognize that sound? To better understand how DHH users can drive personalization of their own assistive sound recognition tools, we conducted a three-part study with 14 DHH participants: (1) an initial interview and demo of a personalizable sound recognizer, (2) a week-long field study of in situ recording, and (3) a follow-up interview and ideation session. Our results highlight a positive subjective experience when recording and interpreting training data in situ, but we uncover several key pitfalls unique to DHH users---such as inhibited judgement of representative samples due to limited audiological experience. We share implications of these results for the design of recording interfaces and human-the-the-loop systems that can support DHH users to build sound recognizers for their personal needs.

Subfields: Cognitive Neuroscience, Speech & Audio, Human-Computer Interaction, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

1. Ali Farhadi
2. Yejin Choi
3. Rowan Zellers
4. Jack Hessel

In Human-Computer Interaction

1. Ali Farhadi
2. Yejin Choi
3. Louis-philippe Morency

In Signal Processing

1. Ali Farhadi
2. Yejin Choi
3. Rowan Zellers
4. Jack Hessel
5. Yanpeng Zhao
6. Ximing Lu
7. Aditya Kusupati
8. Youngjae Yu

Citations in the corpus

• MERLOT RESERVE: Neural Script Knowledge through Vision and Language and Sound (2022, background): “, to improve accessibility for low vision and blind users [75] or for people who are d/Deaf [47].”