Self-Supervised Learning of Audio-Visual Objects from Video
By Triantafyllos Afouras, Andrew Owens, Joon Son Chung, Andrew Zisserman. 2020
287 citations recorded by the source corpus.
Our objective is to transform a video into a set of discrete audio-visual objects using self-supervised learning. To this end, we introduce a model that uses attention to localize and group sound sources, and optical flow to aggregate information over time. We demonstrate the effectiveness of the audio-visual object embeddings that our model learns by using them for four downstream speech-oriented tasks: (a) multi-speaker sound source separation, (b) localizing and tracking speakers, (c) correcting misaligned audio-visual data, and (d) active speaker detection. Using our representation, these tasks can be solved entirely by training on unlabeled video, without the aid of object detectors. We also demonstrate the generality of our method by applying it to non-human speakers, including cartoons and puppets.Our model significantly outperforms other self-supervised approaches, and obtains performance competitive with methods that use supervised face detection.
Subfields: Artificial Intelligence, Multimodal Learning, Representation & Self-Supervised Learning, Signal Processing
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Kristen Grauman
- Luc Van Gool
- Trevor Darrell
- Giovanni Maria Farinella
- Antonino Furnari
- Abhinav Kumar Gupta
- Jitendra Malik
- Cordelia Schmid
In Multimodal Learning
- Kristen Grauman
- Lorenzo Torresani
- Dhruv Batra
- Emanuele Alberti
- Chiara Plizzari
- Barbara Caputo
- Mirco Planamente
- Antonio Torralba
In Representation & Self-Supervised Learning
- Kristen Grauman
- Antonio Torralba
- Jitendra Malik
- Andrea Vedaldi
- Christoph Feichtenhofer
- Aude Oliva
- Lorenzo Torresani
- Barbara Caputo
In Signal Processing
- Eduardo Fonseca
- Xavier Serra
- Kristen Grauman
- Jitendra Malik
- Lorenzo Torresani
- Antonio Torralba
- David J. Crandall
- Christoph Feichtenhofer
Citations in the corpus
- Ego4D: Around the World in 3,000 Hours of Egocentric Video (2021, background): “The TTM task is related to audio-visual speaker detection [7,193] and meeting understanding [21,132,154].”
- Cross-Domain First Person Audio-Visual Action Recognition through Relative Norm Alignment (2021, methodology): “By solving this pretext task, the network is induced to find a correspondence between audio and visual cues, making the resulting representations perfect for tasks like sound-source localization [3, 1, 62], active speaker detection [12, 1], and multispeaker source separation [42, 1].”
- Concealed Object Detection (2021, background): “supervised learning [97], [98] have achieved worldrenowned achievements, attracting much attention.”
- Contrastive Representation Learning: A Framework and Review (2020, methodology): “[1] used de-synchronisation to select negative samples by mis-aligning (shifting) the video and audio features.”
- Unsupervised Contrastive Learning of Sound Event Representations (2020, background): “Then, their corresponding embedding representations are compared using a contrastive loss [9, 17, 18] that pulls together representations of positive examples, while pushing apart those of negative ones (i.e., unrelated examples).”
- Keeping Your Eye on the Ball: Trajectory Attention in Video Transformers (2021, unknown): “Optical flow-based pooling can be used instead of temporal convolutions to improve the representation’s robustness to camera and object motions [1].”