AVLnet: Learning Audio-Visual Language Representations from Instructional Videos

By Andrew Rouditchenko, Angie Boggust, David F. Harwath, Dhiraj Joshi, Samuel Thomas, Kartik Audhkhasi, Rogério Schmidt Feris, Brian Kingsbury, Michael A. Picheny, Antonio Torralba and 1 more. 2020

148 citations recorded by the source corpus.

Current methods for learning visually grounded language from videos often rely on time-consuming and expensive data collection, such as human annotated textual summaries or machine generated automatic speech recognition transcripts. In this work, we introduce Audio-Video Language Network (AVLnet), a self-supervised network that learns a shared audio-visual embedding space directly from raw video inputs. We circumvent the need for annotation and instead learn audio-visual language representations directly from randomly segmented video clips and their raw audio waveforms. We train AVLnet on publicly available instructional videos and evaluate our model on video clip and language retrieval tasks on three video datasets. Our proposed model outperforms several state-of-the-art text-video baselines by up to 11.8% in a video clip retrieval task, despite operating on the raw audio instead of manually annotated text captions. Further, we show AVLnet is capable of integrating textual information, increasing its modularity and improving performance by up to 20.3% on the video clip retrieval task. Finally, we perform analysis of AVLnet's learned representations, showing our model has learned to relate visual objects with salient words and natural sounds.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Signal Processing

Citations in the corpus

Self-Supervised MultiModal Versatile Networks (2020, methodology): “Taking this idea into the self-supervised domain [4], multiple works use a pretext task of predicting whether visual and audio signals come from the same video [4, 5, 35, 51, 62].”
MERLOT RESERVE: Neural Script Knowledge through Vision and Language and Sound (2022, background): “Recent work has explored this paradigm for matching video frames with their transcripts [121], with their audio signal [96, 114], or both [3, 2]; these works likewise perform well on single-modality tasks like audio classification and activity recognition.”
VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modeling (2021, methodology): “To enhance the performance, there have been parallel interests in bringing in more modalities from raw video inputs [31,47,48] and end-to-end training till the raw pixels [18,19,25,49], both aiming to elevate video representations for VidL modeling.”
VLM: Task-agnostic Video-Language Model Pre-training for Video Understanding (2021, neutral): “For example, (Miech et al., 2019, 2020; Rouditchenko et al., 2020; Ging et al., 2020; Gabeur et al., 2020; Alayrac et al., 2020; Patrick et al., 2021; Huang et al., 2021) adopts a retrieval task for pre-training.”
Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval (2021, methodology): “using loss functions such as MIL-NCE [35] or using the raw audio [1, 48] directly to increase robustness.”
Support-set bottlenecks for video-text representation learning (2020, background): “Other works have leveraged the contrastive loss (Miech et al., 2020) and show that using the raw audio (Alayrac et al., 2020; Rouditchenko et al., 2020) and other modalities (Gabeur et al., 2020) can be used to better align and improve video-text representations.”
Semi-Supervised and Unsupervised Deep Visual Learning: A Survey (2022, background): “, from web data [307] or narrated materials [308], [309], [310], [311], [312], [313]) to guide unsupervised representation learning by aligning images and languages in the shared latent space, as exemplified by CLIP [312] and ALIGN [313].”
Video-Text Pre-training with Learned Regions (2021, background): “Though some of these methods adopt expert features including object [79], sound [19,58] and speech [19] information.”
BridgeFormer: Bridging Video-text Retrieval with Multiple Choice Questions (2022, background): “Methods in the first category [6, 11, 13, 22, 25, 28, 33, 41, 43] adopt two individual encoders to embed video features and text features, and project them into the same latent space.”
MILES: Visual BERT Pre-training with Injected Language Semantics for Video-text Retrieval (2022, methodology): “Year Video Input Pre-train Dataset Pairs R@1 R@5 R@10 MedR ActBERT [55] 2020 Method R@1 R@5 R@10 MedR AVLnet [42] 1.4 cant improvement of the performance under the zero-shot evaluation protocol indicates that our pre-trained model is more generalizable that can be used for out-of-domain…”
Self-Supervised Speech Representation Learning: A Review (2022, background): “Typical approaches involve learning a neural representation model for each modality, with a multi-modal contrastive loss that encourages paired examples in the two modalities to have similar representations while unpaired examples remain different [171], [172], [173], [174], [175], [176].”
A Straightforward Framework For Video Retrieval Using CLIP (2021, methodology): “HT100M is a dataset of 100 million video-text pairs, used only as a pre-training set for other Video Retrieval works [5,11,13,16].”