Grounded Language Learning from Video Described with Sentences

By Haonan Yu, Jeffrey Mark Siskind. 2013

140 citations recorded by the source corpus.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Grounded Language Learning in a Simulated 3D World (2017, background): “Broadly this can be seen as a precursor to more recent work on mapping language to actions in video and similar modalities (Siskind, 2001; Chen and Mooney, 2008; Yu and Siskind, 2013).”
ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks (2019, background): “Please see the supplemental material for example demonstrations and language for each task. and concepts [1, 45, 42, 55, 58].”
Towards Automatic Learning of Procedures From Web Instructional Videos (2017, methodology): “The output procedure segments of ProcNets can be applied for other tasks, including full agent-based procedure learning (Yu and Siskind 2013) or smaller-scale video description generation (Yu et al. 2016; Krishna et al. 2017).”
End-to-End Dense Video Captioning with Masked Transformer (2018, neutral): “In contrast to earlier video captioning papers, which are based on models like hidden Markov models and ontologies [39, 6], recent work on captioning is dominated by deep neural network-based methods [32, 34, 37, 43, 36, 26].”
Translating Videos to Natural Language Using Deep Recurrent Neural Networks (2014, background): “The memory cell is at the core of the LSTM unit and it is modulated by the input, output and forget gates controlling how much knowledge is transferred at each time step. which incorporate explicitly controllable memory units, are known to be able to learn long-range temporal dependencies.”
Localizing Moments in Video with Natural Language (2017, background): “However prior methods in both these areas ([43, 55]) severely constrain natural language vocabulary (e.g., [55] only considers four objects and four verbs) and consider constrained visual domains in small datasets (e.g., 127 videos from a ﬁxed laboratory kitchen [28] and [20] only includes 520…”
Grounding of Textual Phrases in Images by Reconstruction (2015, background): “In video domain some of the representative works on spatial-temporal language grounding are [22] and [39].”
What’s Cookin’? Interpreting Cooking Videos using Text, Speech and Vision (2015, methodology): “(Yu and Siskind, 2013) describes a technique for estimating the compatibility between a video clip and a sentence, based on relative motion of the objects (which are tracked using HMMs).”
Jointly Modeling Deep Video and Compositional Text to Bridge Vision and Language in a Unified Framework (2015, background): “(Ramanathan, Liang, and Fei-Fei 2013) use a topic-based semantic relatedness measure to help action and role classification, (Yu and Siskind 2013) use HMMs to track sentences and learn the representation for word meanings with the help of video clips.”
Automatic differentiation in machine learning: a survey (2015, methodology): “Related with the work on video event detection in the previous section, Yu and Siskind (2013) report their work on sentence tracking, representing an instance of grounded language learning paired with computer vision, where the system learns word meanings from short video clips paired with…”
Unsupervised Semantic Parsing of Video Collections (2015, methodology): “Relations extracted from image-caption pairs, are further used to help semantic parsing [61] and activity recognition [41].”
Survey of the State of the Art in Natural Language Generation: Core tasks, applications and evaluation (2017, background): “…alignments between data and text, also in the broader context of grounded language acquisition, i.e., modelling how we learn language by looking at correspondences between objects and events in the world and the way we refer to them in language (Roy, 2002; Yu & Ballard, 2004; Yu & Siskind, 2013).”
See No Evil, Say No Evil: Description Generation from Densely Labeled Images (2014, background): “These problems also extend to video caption systems (Yu and Siskind, 2013; Krishnamoorthy et al., 2013).”
Toward Self-Supervised Object Detection in Unlabeled Videos (2019, background): “These approaches target learning from a large number of unlabeled videos [2, 5, 12, 18, 19, 26, 28] capitalizing on the natural synchronization of the visual and audio modalities, to learn models that discover audio-visual correspondence.”
Learning to Detect and Retrieve Objects From Unlabeled Videos (2019, background): “Another line of studies target Audio-Visual alignment in unlabeled videos [2, 3, 8, 11, 12, 18, 20] capitalizing on the natural synchronization of the visual and audio modalities to learn models that discover audio-visual correspondence.”
What Are You Talking About? Text-to-Image Coreference (2014, background): “In [35], representations for word meanings are learned from short video clips paired with sentences by learning the correspondence between words and video concepts in an unsupervised fashion.”
Temporally Grounding Natural Sentence in Video (2018, background): “Earlier works (Yu and Siskind, 2013; Lin et al., 2014) learn the semantics of sentences, which are then matched to visual concepts via exploiting object appearance,”