Look Before you Speak: Visually Contextualized Utterances

By Paul Hongsuck Seo, Arsha Nagrani, Cordelia Schmid. 2020

71 citations recorded by the source corpus.

While most conversational AI systems focus on textual dialogue only, conditioning utterances on visual context (when it’s available) can lead to more realistic conversations. Unfortunately, a major challenge for incorporating visual context into conversational dialogue is the lack of large-scale labeled datasets. We provide a solution in the form of a new visually conditioned Future Utterance Prediction task. Our task involves predicting the next utterance in a video, using both visual frames and transcribed speech as context. By exploiting the large number of instructional videos online, we train a model to solve this task at scale, without the need for manual annotations. Leveraging recent advances in multimodal learning, our model consists of a novel co-attentional multimodal video transformer, and when trained on both textual and visual context, outperforms baselines that use textual inputs alone. Further, we demonstrate that our model trained for this task on unlabelled videos achieves state-of-the-art performance on a number of downstream VideoQA benchmarks such as MSRVTT-QA, MSVD-QA, ActivityNet-QA and How2QA.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Dialogue & Conversational AI

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Dialogue & Conversational AI

Citations in the corpus

Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval (2021, background): “However, since the release of the HowTo100M dataset [37], a large-scale instructional video dataset, there has been a flurry of works leveraging large-scale pretraining to improve video-text representations for tasks such as video question-answering [50], text-video retrieval [41] and video captioni”
Just Ask: Learning to Answer Questions from Millions of Narrated Videos (2020, neutral): “[66] propose a pretraining approach based on next utterance prediction and finetune for VideoQA.”
GIT: A Generative Image-to-text Transformer for Vision and Language (2022, background): “[93] Paul Hongsuck Seo, Arsha Nagrani, and Cordelia Schmid.”
Video Graph Transformer for Video Question Answering (2022, methodology): “Pioneer works [32,48,63,64,72] learn generalizable representations from HowTo100M [40] by either applying various proxy tasks [72], or curating more tailored-made supervisions (e.”
Zero-Shot Video Question Answering via Frozen Bidirectional Language Models (2022, unknown): “[88] Paul Hongsuck Seo, Arsha Nagrani, and Cordelia Schmid.”
Attention Bottlenecks for Multimodal Fusion (2021, methodology): “Multimodal transformers have been applied to various tasks including audio enhancement [19, 60], speech recognition [27], image segmentation [66, 60], crossmodal sequence generation [43, 41, 56], image and video retrieval [28, 23, 8], visual navigation [51] and image/video captioning/classification ”
Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioning (2023, methodology): “Second, we design a pretraining strategy that effectively leverages cross-modal supervision in the form of transcribed speech from unlabeled narrated videos by reformulating sentence boundaries as pseudo event boundaries, as presented in Section 3.2 and Figure 3.”
End-to-end Generative Pretraining for Multimodal Video Captioning (2022, methodology): “Recently introduced datasets such as Cooking312K [45] and HowTo100M [34] leverage such instructional videos with associated captions from ASR (automatic speech recognition) to learn joint video-and-text embeddings [33, 44] or to train multimodal video encoders [27,42].”