Looking Enhances Listening: Recovering Missing Speech Using Images
By Tejas Srinivasan, Ramon Sanabria, Florian Metze. 2020
15 citations recorded by the source corpus.
Speech is understood better by using visual context; for this reason, there have been many attempts to use images to adapt automatic speech recognition (ASR) systems. Current work, however, has shown that visually adapted ASR models only use images as a regularization signal, while completely ignoring their semantic content. In this paper, we present a set of experiments where we show the utility of the visual modality under noisy conditions. Our results show that multimodal ASR models can recover words which are masked in the input acoustic signal, by grounding its transcriptions using the visual representations. We observe that integrating visual context can result in up to 35% relative improvement in masked word recovery. These results demonstrate that end-to-end multimodal ASR systems can become more robust to noise by leveraging the visual context.
Subfields: Multimodal Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Multimodal Learning
- Yonatan Bisk
- Angeliki Lazaridou
- Ari Holtzman
- Joyce Yue Chai
- Mirella Lapata
- Yoshua Bengio
- Jacob Andreas
- Jesse Thomason
Citations in the corpus
- Experience Grounds Language (2020, background): “Vision can also help ground speech signals (Srinivasan et al., 2020; Harwath et al., 2019) to facilitate discovery of linguistic concepts (Harwath et al.”