Grounding the Lexical Semantics of Verbs in Visual Perception using Force Dynamics and Event Logic

By Jeffrey Mark Siskind. 1999

This paper presents an implemented system for recognizing the occurrence of events described by simple spatial-motion verbs in short image sequences. The semantics of these verbs is specified with event-logic expressions that describe changes in the state of force-dynamic relations between the participants of the event. An efficient finite representation is introduced for the infinite sets of intervals that occur when describing liquid and semi-liquid events. Additionally, an efficient procedure using this representation is presented for inferring occurrences of compound events, described with event-logic expressions, from occurrences of primitive events. Using force dynamics and event logic to specify the lexical semantics of events allows the system to be more robust than prior systems based on motion profile.

Subfields: Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

Citations in the corpus

Grounded Language Learning in a Simulated 3D World (2017, background): “y seeking to link objects and events in stick-gure animations to language. Broadly this can be seen as a precursor to more recent work on mapping language to actions in video and similar modalities (Siskind, 2001; Chen and Mooney, 2008; Yu and Siskind, 2013). In a similar vein, the work of Roy and P”
Localizing Moments in Video with Natural Language (2017, background): “we speciﬁcally consider how to understand temporal words commonly used when referring to video content. Other work has modeled dynamics for words which represent a change of state (e.g., “pick up”) ( Siskind (2001), Yu et al. (2015)) in limited environments. Though we limit the selection of temporal”
Choosing words in computer-generated weather forecasts (2005, background): “Previous models of lexical selection in Natural Language Generation, and previous models of word meaning in language-and-world systems, have generally used fixed (non-context-dependent) models of meaning [43,33,44,27,34].”
Envisioning the qualitative effects of robot manipulation actions using simulation-based projections (2017, result): “This is similar to work by Siskind (2001), who grounded semantics in visual perception.”
A review of verbal and non-verbal human-robot interactive communication (2014, background): “Furthermore, regarding the grounding of attachment and support relations in videos, there is the classic work by [78].”
Learning High-Level Planning from Text (2012, methodology): “Learning Semantics via Language Grounding Our work fits into the broad area of grounded language acquisition, where the goal is to learn linguistic analysis from a situated context (Oates, 2001; Siskind, 2001; Yu and Ballard, 2004; Fleischman and Roy, 2005; Mooney, 2008a; Mooney, 2008b; Branavan et ”
The challenges of joint attention (2006, background): “Most works implement a set of perceptual primitives capable of extracting relevant features in action sequences (e.g.(Roy and Pentland, 2002; Siskind, 2001; Dominey, 2003; Steels, 2003)).”
Revisiting active perception (2016, unknown): “in the integration of the system. This avenue contributes to the grounding of the meaning of language and is an important research direction. (Teo et al. 2012; Kollar et al. 2010; Tellex et al. 2009; Siskind, 2001; Yu et al. 2011; Yang et al. 2014; Zampogiannis et al. 2014; Yang et al. 2015). John T”
Human activity analysis (2011, methodology): “Allen’s temporal predicates [Allen 1983; Allen and Ferguson 1994] have been widely adopted for these approaches to specify relationships between time intervals [Pinhanez and Bobick 1998; Siskind 2001; Nevatia et al. 2003; Vu et al. 2003; Ryoo and Aggarwal 2006a; Gupta et al. 2009].”
Estimating contact dynamics (2009, unknown): “At a high-level, physics-based models and contact have been used for image interpretation of simple scene domains [3, 12, 17 ].”
A survey on activity recognition and behavior understanding in video surveillance (2013, background): “Siskind [175] also proposed a hierarchical description-based approach for human activity recognition.”