Physical Causality of Action Verbs in Grounded Language Understanding

By Qiaozi Gao, Malcolm Doering, Shaohua Yang, Joyce Yue Chai. 2016

49 citations recorded by the source corpus.

Linguistics studies have shown that action verbs often denote some Change of State (CoS) as the result of an action. However, the causality of action verbs and its potential connection with the physical world has not been systematically explored. To address this limitation, this paper presents a study on physical causality of action verbs and their implied changes in the physical world. We ﬁrst conducted a crowd-sourcing experiment and identiﬁed eighteen categories of physical causality for action verbs. For a subset of these categories, we then deﬁned a set of detectors that detect the corresponding change from visual perception of the physical environment. We further incorporated physical causality modeling and state detection in grounded language understanding. Our empirical studies have demonstrated the effectiveness of causality modeling in grounding language to perception.

Subfields: Artificial Intelligence, Experimental and Cognitive Psychology, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Experimental and Cognitive Psychology

In Natural Language Processing

Citations in the corpus

PIQA: Reasoning about Physical Commonsense in Natural Language (2019, background): “Language tools should provide strong initial priors for learning (Tellex et al. 2011; Matuszek 2018) that are then reﬁned through interaction and dialogue (Gao et al. 2016).”
Published as a conference paper at ICLR 2018 S IMULATING A CTION D YNAMICS WITH N EURAL P ROCESS N ETWORKS (2018, background): “Additional work in tracking states with visual or multimodal context has focused on 1) building graph representations for how entities change in goal-oriented domains (Gao et al., 2016; Liu et al., 2016; Si et al., 2011) or 2) tracking visual state changes based on decisions taken by agents in envir”
Verb Physics: Relative Physical Knowledge of Actions and Objects (2017, background): “Another work in this line does a deep investigation of eleven verbs, modeling their physical effect via annotated images along eighteen attributes (Gao et al., 2016).”
Collaborative Language Grounding Toward Situated Human-Robot Dialogue (2017, methodology): “To further support interactive robot learning through natural language dialogue, our current work is to develop approaches to ground language to participants of actions in more complex visual scenes (for example, a kitchen environment) (Yang et al. 2016, Gao et al. 2016).”
Language to Action: Towards Interactive Task Learning with Physical Agents (2018, methodology): “We ﬁrst collected a dataset of commonly used verb-noun pairs (in a kitchen setting) and their effect descriptions using crowd-sourcing [ Gao et al. , 2016 ] .”
Interactive Grounded Language Acquisition and Generalization in a 2D World (2018, methodology): “An adaptation of a model devised by Yang et al. (2016) which was originally proposed for VQA.”