Localizing Moments in Video with Natural Language

By Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, Bryan C. Russell. 2017

1,236 citations recorded by the source corpus.

We consider retrieving a specific temporal segment, or moment, from a video given a natural language text description. Methods designed to retrieve whole video clips with natural language determine what occurs in a video but not when. To address this issue, we propose the Moment Context Network (MCN) which effectively localizes natural language queries in videos by integrating local and global video features over time. A key obstacle to training our MCN model is that current video datasets do not include pairs of localized video segments and referring expressions, or text descriptions which uniquely identify a corresponding moment. Therefore, we collect the Distinct Describable Moments (DiDeMo) dataset which consists of over 10,000 unedited, personal videos in diverse visual settings with pairs of localized video segments and referring expressions. We demonstrate that MCN outperforms several baseline methods and believe that our initial results together with the release of DiDeMo will inspire further research on localizing video moments with natural language.

Subfields: Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

End-to-End Learning of Visual Representations From Uncurated Instructional Videos (2019, neutral): “We compare the NCE based approach (ours) to the frequently used max margin ranking loss [19, 21, 33, 40, 51, 55, 79, 80, 83, 84] and a binary classification loss (i.”
HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips (2019, methodology): “However, existing datasets (e.g. MSR-VTT [58], DiDeMo [15], EPIC- KITCHENS [7]), are on the scale of tens to hundreds of thousands of such pairs that have been annotated manually.”
VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modeling (2021, methodology): “Popular VidL tasks were introduced, such as text-to-video retrieval [1–5], video question answering [6–9], text-based video moment retrieval [2, 4, 10, 11], and video captioning [1, 3, 12, 13].”
Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval (2021, methodology): “However, their development mostly proceeds on two independent tracks: one for images, with its own architectures, training datasets and benchmarks [25, 29, 52]; and the other for videos with a similar separation of training datasets and benchmarks [3, 5, 24, 46, 65, 70].”
Hero: Hierarchical Encoder for Video+Language Omni-representation Pre-training (2020, background): “Anne Hendricks et al. (2017) and Gao et al. (2017) introduce the task of Single Video Moment Retrieval (SVMR), which aims at retrieving a moment from a single video via a natural language query.”
From Recognition to Cognition: Visual Commonsense Reasoning (2018, background): “An orthogonal line of work has been on referring expressions: asking to what image region a natural language sentence refers to [60, 52, 65, 87, 88, 59, 36, 33].”
TVQA: Localized, Compositional Video Question Answering (2018, background): “Most recently, moment localization was proposed in (Hendricks et al., 2017; Gao et al., 2017), where the goal is to localize a short moment from a long video sequence given a query description.”
Continual and Multi-Task Architecture Search (2019, methodology): “Similarly, we show that the cell structure learned from jointly training on the MSR-VTT and MSVD video captioning datasets performs better on the DiDeMo dataset than the individually-learned cell structures.”
CLEVRER: CoLlision Events for Video REpresentation and Reasoning (2019, background): “This includes video captioning (Guadarrama et al., 2013; Venugopalan et al., 2015; Gan et al., 2017), localizing video segments from natural language queries (Gao et al., 2017; Hendricks et al., 2017), and video question answering.”
Learning a Text-Video Embedding from Incomplete and Heterogeneous Data (2018, methodology): “Towards this goal recent works study representations of vision and language addressing tasks such as visual question answering [1], [2], action learning and discovery [3], [4], [5], text-based event localization [6] as well as video captioning, retrieval and summarization [7], [8], [9], [10].”
TVQA+: Spatio-Temporal Grounding for Video Question Answering (2019, background): “mIoU) following previous works [12, 11] on language-guided video moment retrieval.”
TVR: A Large-Scale Dataset for Video-Subtitle Moment Retrieval (2020, methodology): “With late fusion and ConvSE, we further show XML outperforms previous methods [14,8,11], and does this with better computational efficiency.”
VALUE: A Multi-Task Benchmark for Video-and-Language Understanding Evaluation (2021, background): “5 There are other VidL datasets on single-channel videos that are not included in VALUE, due to the difficulties in collecting hidden test set [11, 70, 8], unnatural annotations [69], or the lack of subtitle channel in GIF videos [25].”
Use What You Have: Video retrieval using representations from collaborative experts (2019, methodology): “4 we provide a comparison with previously reported numbers on the DiDeMo benchmark and see that CE again strongly outperforms prior work.”
CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval (2021, background): “DiDeMo (Hendricks et al., 2017) contains 10,000 videos annotated with 40,000 sentences.”