Discriminative Unsupervised Alignment of Natural Language Instructions with Corresponding Video Segments

By Iftekhar Naim, Y. Song, Qiguang Liu, Liang Huang, Henry A. Kautz, Jiebo Luo, Daniel Gildea. 2015

31 citations recorded by the source corpus.

We address the problem of automatically aligning natural language sentences with corresponding video segments without any direct supervision. Most existing algorithms for integrating language with videos rely on handaligned parallel data, where each natural language sentence is manually aligned with its corresponding image or video segment. Recently, fully unsupervised alignment of text with video has been shown to be feasible using hierarchical generative models. In contrast to the previous generative models, we propose three latent-variable discriminative models for the unsupervised alignment task. The proposed discriminative models are capable of incorporating domain knowledge, by adding diverse and overlapping features. The results show that discriminative models outperform the generative models in terms of alignment accuracy.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Multimodal Learning

Citations in the corpus

A Case Study on Combining ASR and Visual Features for Generating Instructional Video Captions (2019, background): “…these potential difficulties, previous work has demonstrated that ASR can be informative in a variety of instructional video understanding tasks (Naim et al., 2014, 2015; Malmaud et al., 2015; Sener et al., 2015; Alayrac et al., 2016; Huang et al., 2017); though less work has focused on…”
Dense Procedure Captioning in Narrated Instructional Videos (2019, methodology): “(Naim et al., 2015) utilize latent-variable based discriminative models (CRF, Structured Per-ceptron) for unsupervised alignment.”
Collaborative Language Grounding Toward Situated Human-Robot Dialogue (2017, background): “Recent years have seen an increasing amount of work on grounding language to visual perception (Tellex et al. 2011; Matuszek et al. 2012; Yu and Siskind 2013; Naim et al. 2015; Yang et al. 2016).”
Unsupervised Visual-Linguistic Reference Resolution in Instructional Videos (2017, background): “Another line of work aim to learn visual classifiers based on only textual supervision [5, 7, 11, 48].”
Multimodal Machine Learning: A Survey and Taxonomy (2017, methodology): “This model was then extended to use latent conditional random fields for alignments [152] and to incorporate verb alignment to actions in addition to nouns and objects [203].”
Learning from Narrated Instruction Videos (2015, methodology): “A similar task to ours is addressed in [28] using latent variable struc-”
Physical Causality of Action Verbs in Grounded Language Understanding (2016, background): “Although recent years have seen an increasing amount of work on grounding language to perception (Yu and Siskind, 2013; Walter et al., 2013; Liu et al., 2014; Naim et al., 2015; Liu and Chai, 2015), no previous work has investigated the link between physical causality denoted by action verbs and the”