Large Scale Audiovisual Learning of Sounds with Weakly Labeled Data
By Haytham M. Fayek, Anurag Kumar. 2020
39 citations recorded by the source corpus.
Recognizing sounds is a key aspect of computational audio scene analysis and machine perception. In this paper, we advocate that sound recognition is inherently a multi-modal audiovisual task in that it is easier to differentiate sounds using both the audio and visual modalities as opposed to one or the other. We present an audiovisual fusion model that learns to recognize sounds from weakly labeled video recordings. The proposed fusion model utilizes an attention mechanism to dynamically combine the outputs of the individual audio and visual models. Experiments on the large scale sound events dataset, AudioSet, demonstrate the efficacy of the proposed model, which outperforms the single-modal models, and state-of-the-art fusion and multi-modal models. We achieve a mean Average Precision (mAP) of 46.16 on Audioset, outperforming prior state of the art by approximately +4.35 mAP (relative: 10.4%).
Subfields: Multimodal Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Multimodal Learning
- Cordelia Schmid
- Arsha Nagrani
- Chen Sun
- Anurag Arnab
- Andrew Zisserman
- Aren Jansen
- Shan Yang
- Hang Zhao
Citations in the corpus
- Perceiver: General Perception with Iterative Attention (2021, background): “Audio+video fusion leads to solid improvements over single modalities (and outperforms specialized fusion optimization approaches (Wang et al., 2020c)) but is still lower than the state-of-the-art approach that uses separate models with late fusion (Fayek & Kumar, 2020).”
- What Makes Multimodal Learning Better than Single (Provably) (2021, background): “, [3, 12, 14, 44, 43, 24], appear to have utilized this framework in one way or another, even though the contributors did not clearly summarize the relationship between their methods and this common underlying structure.”
- Attention Bottlenecks for Multimodal Fusion (2021, methodology): “We outperform all previous works on fusion (even though we only train on a quarter of the training set – 500K samples), including the recently introduced Perceiver [32] which uses early fusion followed by multiple self attention layers, and Attn Audio-Visual [21] which uses self-attention fusion on ”