TALL: Temporal Activity Localization via Language Query

By Jiyang Gao, Chen Sun, Zhenheng Yang, Ramkant Nevatia. 2017

1,142 citations recorded by the source corpus.

This paper focuses on temporal localization of actions in untrimmed videos. Existing methods typically train classifiers for a pre-defined list of actions and apply them in a sliding window fashion. However, activities in the wild consist of a wide combination of actors, actions and objects; it is difficult to design a proper activity list that meets users’ needs. We propose to localize activities by natural language queries. Temporal Activity Localization via Language (TALL) is challenging as it requires: (1) suitable design of text and video representations to allow cross-modal matching of actions and language queries; (2) ability to locate actions accurately given features from sliding windows of limited granularity. We propose a novel Cross-modal Temporal Regression Localizer (CTRL) to jointly model text query and video clips, output alignment scores and action boundary regression results for candidate clips. Lor evaluation, we adopt TaCoS dataset, and build a new dataset for this task on top of Charades by adding sentence temporal annotations, called Charades-STA. We also build complex sentence queries in Charades-STA for test. Experimental results show that CTRL outperforms previous methods significantly on both datasets.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modeling (2021, methodology): “Popular VidL tasks were introduced, such as text-to-video retrieval [1–5], video question answering [6–9], text-based video moment retrieval [2, 4, 10, 11], and video captioning [1, 3, 12, 13].”
Hero: Hierarchical Encoder for Video+Language Omni-representation Pre-training (2020, background): “Anne Hendricks et al. (2017) and Gao et al. (2017) introduce the task of Single Video Moment Retrieval (SVMR), which aims at retrieving a moment from a single video via a natural language query.”
TVQA: Localized, Compositional Video Question Answering (2018, background): “Most recently, moment localization was proposed in (Hendricks et al., 2017; Gao et al., 2017), where the goal is to localize a short moment from a long video sequence given a query description.”
Less is More: CLIPBERT for Video-and-Language Learning via Sparse Sampling (2021, background): “A wide range of tasks based on real-life videos have been designed to test such ability, including text-to-video retrieval [75, 28, 54], video captioning [54, 75, 82], video question answering [74, 23, 33, 34], and video moment retrieval [2, 18, 35].”
Localizing Moments in Video with Natural Language (2017, background): “t the temporal relationship between different actions (“talk” and “bend down”). Localizing natural language queries in video is an important challenge, recently studied in Hendricks et al. (2017) and Gao et al. (2017) with applications in areas such as video search and retrieval. We argue that to Wo”
CLEVRER: CoLlision Events for Video REpresentation and Reasoning (2019, background): “This includes video captioning (Guadarrama et al., 2013; Venugopalan et al., 2015; Gan et al., 2017), localizing video segments from natural language queries (Gao et al., 2017; Hendricks et al., 2017), and video question answering.”
TVQA+: Spatio-Temporal Grounding for Video Question Answering (2019, background): “Recently, there has been a growing interest in moment retrieval tasks [12, 11, 9], where the goal is to localize a short clip from a long video via a natural language query.”
TVR: A Large-Scale Dataset for Video-Subtitle Moment Retrieval (2020, methodology): “Recent works [1, 9] introduced the task of Single Video Moment Retrieval (SVMR), whose goal is to retrieve a moment from a single video via a nat-ural language query.”
VALUE: A Multi-Task Benchmark for Video-and-Language Understanding Evaluation (2021, background): “Stand-alone evaluation on temporal moment localization tasks [7, 21] is not included in our benchmark, as the two VCMR tasks already evaluate the ability of the model to localize relevant moment as a sub-task.”
Motion-Appearance Co-memory Networks for Video Question Answering (2018, neutral): “[8] and Hendricks et al.”
RED: Reinforced Encoder-Decoder Networks for Action Anticipation (2017, methodology): “TALL [3] used natural language as query to localize actions in long videos and designed a cross-modal regression model to solve it.”
Temporal Localization of Moments in Video Collections with Natural Language (2019, methodology): “In CTRL [10], aggregated video moment features (Ψ) and language features pass through a neural network (Φ).”
What Is More Likely to Happen Next? Video-and-Language Future Event Prediction (2020, background): “…have been introduced in this area, such as video captioning (Xu et al., 2016; Rohrbach et al., 2017; Wang et al., 2019; Lei et al., 2020c), video QA (Tapaswi et al., 2016; Jang et al., 2017; Lei et al., 2018), and moment retrieval (Hendricks et al., 2017; Gao et al., 2017a; Lei et al., 2020c).”
COOT: Cooperative Hierarchical Transformer for Video-Text Representation Learning (2020, background): “There has been growing interest in temporal localization of natural language in videos [7, 71, 72, 73, 2].”
VTimeLLM: Empower LLM to Grasp Video Moments (2023, methodology): “When integrated with natural language, there are two primary tasks: Temporal Video Grounding [1, 8] and Dense Video Caption-ing [12, 25].”
Egocentric Video-Language Pretraining (2022, positive): “The Natural Language Query task is modelled as a natural language grounding problem [38, 39, 40].”
MAN: Moment Alignment Network for Natural Language Moment Retrieval via Iterative Graph Adjustment (2018, methodology): “task: DiDeMo [18] and Charades-STA [14].”
Span-based Localizing Network for Natural Language Video Localization (2020, methodology): “…task, which is solved by applying multimodal The Ground Truth Moment matching architecture to ﬁnd the best matching video segment for a given language query (Gao et al., 2017; Hendricks et al., 2018; Liu et al., 2018a; Ge et al., 2019; Xu et al., 2019; Chen and Jiang, 2019; Zhang et al., 2019).”
Violin: A Large-Scale Dataset for Video-and-Language Inference (2020, neutral): “Video-and-Language Research With the emergence of large-scale video datasets [6, 1, 29, 11, 58], several videoand-language tasks have been proposed, such as video captioning [21, 56, 62, 18, 33, 16, 47, 59], localizing video segments from natural language queries [19, 3, 8, 37], video reasoning [65]”
A Survey on Deep Learning Technique for Video Segmentation (2021, background): “Although there have already existed some efforts [247] in the intersection of language and video understanding, none of them addresses pixel-level video-language reasoning.”