Weakly-Supervised Alignment of Video with Text

By Piotr Bojanowski, Rémi Lajugie, Edouard Grave, Francis R. Bach, Ivan Laptev, Jean Ponce, Cordelia Schmid. 2015

142 citations recorded by the source corpus.

Suppose that we are given a set of videos, along with natural language descriptions in the form of multiple sentences (e.g., manual annotations, movie scripts, sport summaries etc.), and that these sentences appear in the same temporal order as their visual counterparts. We propose in this paper a method for aligning the two modalities, i.e., automatically providing a time (frame) stamp for every sentence. Given vectorial features for both video and text, this can be cast as a temporal assignment problem, with an implicit linear mapping between the two feature modalities. We formulate this problem as an integer quadratic program, and solve its continuous convex relaxation using an efficient conditional gradient algorithm. Several rounding procedures are proposed to construct the final integer solution. After demonstrating significant improvements over the state of the art on the related task of aligning video with symbolic labels [7], we evaluate our method on a challenging dataset of videos with associated textual descriptions [37], and explore bag-of-words and continuous representations for text.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

End-to-End Learning of Visual Representations From Uncurated Instructional Videos (2019, methodology): “Examples include: unsupervised learning from narrated instruction videos [1], text-to-video alignment [6], co-segmentation [16], co-localization in videos and images [18] or instance-level segmentation [31], which can be now scaled-up to an order of magnitude larger datasets.”
Joint Discovery of Object States and Manipulation Actions (2017, methodology): “the approach of [3, 6, 7], each video is divided into chunks of 10 frames that are represented by a motion and appearance descriptor averaged over 30 frames.”
Unsupervised Learning from Narrated Instruction Videos (2015, background): “The values of precision and recall are given at the bottom of Table 1.”
Localizing Moments in Video with Natural Language (2017, background): “In a similar vein, [1, 4, 36] consider aligning textual instructions to videos.”
Grounding of Textual Phrases in Images by Reconstruction (2015, background): “There have been a number of works aiming to temporally align language descriptions to video [2, 24, 30, 41].”
Learning a Text-Video Embedding from Incomplete and Heterogeneous Data (2018, neutral): “e temporal localization) in videos [4,6].”
Cross-Task Weakly Supervised Learning From Instructional Videos (2019, methodology): “For instance, we make use of ordering constraints to obtain supervision, as was done in [5, 18, 22, 26, 6].”
TALL: Temporal Activity Localization via Language Query (2017, background): “[1] tackled the problem of video-text alignment: given a video and a set of sentences with temporal ordering, assigning a temporal interval for each sentence.”
Connectionist Temporal Modeling for Weakly Supervised Action Labeling (2016, background): “An alternative is to develop methods that require weak supervision, which may be automatically extracted from movie scripts [3,5,9] or instructional videos [2,4,50] at a lower cost.”
W-TALC: Weakly-supervised Temporal Activity Localization and Classification (2018, unknown): “A similar approach was taken in [7].”
Learning from Narrated Instruction Videos (2015, methodology): “pare to the improved model from [6], which does not require a “background class” and yields a stronger baseline equivalent to our model (7) without the weak textual constraints.”
Textually Customized Video Summaries (2017, unknown): “ms of time and human effort for customized video summarization. Our contextually customized video summarization approach is motivated by recent developments on joint embeddings of images and captions [3, 5, 22], while the goal is different in that our multimodal embedding module aims to learn contex”
Action Recognition From Single Timestamp Supervision in Untrimmed Videos (2019, background): “Transcript supervision offers an ordered list of action labels in untrimmed videos, without any temporal annotations [4, 5, 14, 24, 18, 25, 26, 10].”
Projection-Free Optimization on Uniformly Convex Sets (2020, methodology): “…the recent revival and extensions of the Frank-Wolfe algorithm (Clarkson, 2010; Jaggi, 2011) used for instance in large-scale structured prediction (Bojanowski et al., 2014, 2015; Alayrac et al., 2016; Seguin et al., 2016; Miech et al., 2017; Peyre et al., 2017; Miech et al., 2018), quadrature…”
Temporal Localization of Moments in Video Collections with Natural Language (2019, methodology): “Datasets of videos with temporally aligned text [10, 16, 17, 22, 29] have been used for aligning movie scripts, textual instructions, and sentences in a paragraph with a single video [3, 25, 32, 39], video object segmentation [20], and retrieving moments in a single video given a text query [4, 5, 1”
Multimodal Machine Learning: A Survey and Taxonomy (2017, methodology): “[23], [24] proposed a method similar to canonical time warping, but have also extended it to take advantage of existing (weak) supervisory alignment data for model training.”
D3TW: Discriminative Differentiable Dynamic Time Warping for Weakly Supervised Action Alignment and Segmentation (2019, background): “The supervision usually comes from movie scripts [8, 2, 41] or transcription of instructional videos [1, 33, 25, 14].”
Learning from Video and Text via Large-Scale Discriminative Clustering (2017, methodology): “In computer vision, it has been used in combination with discriminative clustering [2] for action localization [5], text-to-video alignment [1, 6], object co-localization in videos and images [18] or instance-level segmentation [32].”