Seeing What You're Told: Sentence-Guided Activity Recognition in Video

By Siddharth Narayanaswamy, Andrei Barbu, Jeffrey Mark Siskind. 2013

41 citations recorded by the source corpus.

We present a system that demonstrates how the compositional structure of events, in concert with the compositional structure of language, can interplay with the underlying focusing mechanisms in video action recognition, providing a medium for top-down and bottom-up integration as well as multi-modal integration between vision and language. We show how the roles played by participants (nouns), their characteristics (adjectives), the actions performed (verbs), the manner of such actions (adverbs), and changing spatial relations between participants (prepositions), in the form of whole-sentence descriptions mediated by a grammar, guides the activity-recognition process. Further, the utility and expressiveness of our framework is demonstrated by performing three separate tasks in the domain of multi-activity video: sentence-guided focus of attention, generation of sentential description, and query-based search, simply by leveraging the framework in different manners.

Subfields: Computer Vision and Pattern Recognition, Human-Computer Interaction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Human-Computer Interaction

Citations in the corpus

Space-Time Representation of People Based on 3D Skeletal Data: A Review (2016, background): “We also provide a brief overview of skeleton acquisition devices and construction methods, enlist a number of benchmark datasets with skeleton data, and discuss potential future research directions.”
Learning Disentangled Representations with Semi-Supervised Deep Generative Models (2017, background): “[31]) may be necessary to define a suitable likelihood function.”
Learning to Describe Video with Weak Supervision by Exploiting Negative Sentential Information (2015, methodology): “Both scores are within [0, 1] and are adopted from Siddharth, Barbu, and Siskind (2014).”
Efficient grounding of abstract spatial concepts for natural language interaction with robot platforms (2018, unknown): “[44] present a model that combines event tracking in videos with language groundings and further demonstrate resolution of linguistic ambiguities with visual evidence.”
Visually Grounded Neural Syntax Acquisition (2019, unknown): “Grounded language acquisition has been studied for image-caption data (Christie et al., 2016a), video-caption data (Siddharth et al., 2014; Yu et al., 2015), and visual reasoning (Mao et al., 2019).”
Grounding language acquisition by training semantic parsers using captioned videos (2018, methodology): “The visual-linguistic compatibility function is similar to the Sentence Tracker developed in Siddharth et al. (2014) and Yu et al. (2015).”