Training a Multilingual Sportscaster: Using Perceptual Context to Learn Language

By David L. Chen, Joohyun Kim, Raymond J. Mooney. 2014

116 citations recorded by the source corpus.

We present a novel framework for learning to interpret and generate language using only perceptual context as supervision. We demonstrate its capabilities by developing a system that learns to sportscast simulated robot soccer games in both English and Korean without any language-specific prior knowledge. Training employs only ambiguous supervision consisting of a stream of descriptive textual comments and a sequence of events extracted from the simulation trace. The system simultaneously establishes correspondences between individual comments and the events that they describe while building a translation model that supports both parsing and generation. We also present a novel algorithm for learning which events are worth describing. Human evaluations of the generated commentaries indicate they are of reasonable quality and in some cases even on par with those produced by humans for our limited domain.

Subfields: Artificial Intelligence, Natural Language Processing, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Multimodal Learning

Citations in the corpus

Neural Generation for Czech: Data and Baselines (2019, background): “This goes hand in hand with the scarcity of non-English training datasets for NLG – the only data-to-text NLG set known to us is a small sportscasting Korean dataset (Chen et al., 2010),1 which only contains a limited number of named entities, reducing the need for their inflection.”
Parsing Algebraic Word Problems into Equations (2015, background): “More closely related is work on language grounding, whose goal is the interpretation of a sentence in the context of a world representation (Branavan et al., 2009; Liang et al., 2009; Chen et al., 2010; Bordes et al., 2010; Feng and Lapata, 2010; Hajishirzi et al., 2011; Matuszek et al., 2012; Hajis”
Learning to Solve Arithmetic Word Problems with Verb Categorization (2014, background): “Our work is also closely related to the grounded language acquisition research (Snyder and Barzilay, 2007; Branavan et al., 2009; Branavan et al., 2012; Vogel and Jurafsky, 2010; Chen et al., 2010; Hajishirzi et al., 2011; Chambers and Jurafsky, 2009; Liang et al., 2009; Bordes et al., 2010) where t”
Weakly Supervised Learning of Semantic Parsers for Mapping Instructions to Actions (2013, background): “Clarke et al. (2010), Goldwasser and Roth (2011) and Liang et al.”
Learning to Interpret Natural Language Navigation Instructions from Observations (2011, methodology): “…training (Snyder and Barzilay 2007), iteratively retraining a semantic parser/language generator to disambiguate the context (Kate and Mooney 2007; Chen, Kim, and Mooney 2010), building a generative model of the content selection process (Liang, Jordan, and Klein 2009; Kim and Mooney 2010), and…”
What to talk about and how? Selective Generation using LSTMs with Coarse-to-Fine Alignment (2015, background): “Chen and Mooney (2008) and Chen et al. (2010) learn alignments between comments and their corresponding event records using a translation model for parsing and generation.”
See No Evil, Say No Evil: Description Generation from Densely Labeled Images (2014, background): “Other research generates descriptions in simulated worlds and, like this work, uses feature rich models (Angeli et al., 2010), or syntactic structures like PCFGs (Chen et al., 2010; Konstas and Lapata, 2012) but does not combine the two.”
A Survey of Current Datasets for Vision and Language Research (2015, background): “Video datasets aligned with descriptions (Chen et al., 2010; Rohrbach et al., 2012; Regneri et al., 2013; Naim et al., 2015; Malmaud et al., 2015) generally represent limited domains and small lexicons, which is due to the fact that video processing and understanding is a very compute-intensive task”
Reasoning about RoboCup Soccer Narratives (2011, methodology): “In contrast to other recently proposed approaches (e.g., [25, 6, 12]), we require neither labeled training data nor an agent that receives indirect supervision by interacting with a physical environment (e.g., [3, 23]).”
Efficient grounding of abstract spatial concepts for natural language interaction with robot platforms (2018, background): “Ge and Mooney [32] integrate semantic roles and derivations within a syntactic parsing framework to determine a statical parse representation that integrates both syntax and compositional semantics.”
Solving Geometry Problems: Combining Text and Diagram Interpretation (2015, background): “Our work is part of grounded language acquisition research (Branavan et al., 2012; Vogel and Jurafsky, 2010; Chen et al., 2010; Hajishirzi et al., 2011; Liang et al., 2009; Koncel-Kedziorski et al., 2014; Bordes et al., 2010; Kim and Mooney, 2013; Angeli and Manning, 2014; Hixon et al., 2015;…”
Multi-Resolution Language Grounding with Weak Supervision (2014, background): “Early work in understanding soccer commentaries focused on RoboCup soccer (Chen and Mooney, 2008; Chen et al., 2010; Bordes et al., 2010; Hajishirzi et al., 2011) where simple language describes each event, and events are in a one-to-one correspondence with utterances.”
Semantic Understanding of Professional Soccer Commentaries (2012, methodology): “Similar to previous work [Liang et al., 2009; Chen et al., 2010], we use loose temporal alignments between sentences in commentaries and events of games.”
Large-scale Semantic Parsing via Schema Matching and Lexicon Extension (2013, background): “Semantic parsing is the task of translating natural language utterances to a formal meaning representation language (Chen et al., 2010; Liang et al., 2009; Clarke et al., 2010; Liang et al., 2011; Artzi and Zettlemoyer, 2011).”
Generative Alignment and Semantic Parsing for Learning from Ambiguous Supervision (2010, methodology): “This follows the experimental scheme of Chen et al. (2010), which demonstrated that an improved NL–MR matching from Liang et al. (2009) results in better overall parsing and generation.”
Bootstrapping Semantic Parsers from Conversations (2011, methodology): “…semantic analysis tasks from context-dependent database queries (Miller et al., 1996; Zettlemoyer and Collins, 2009), grounded event streams (Chen et al., 2010; Liang et al., 2009), environment interactions (Branavan et al., 2009; 2010; Vogel and Jurafsky, 2010), and even unanno-tated text…”
Back to the Blocks World: Learning New Actions through Situated Human-Robot Dialogue (2014, background): “Most previous work on following human instructions are based on supervised learning (Kol-lar et al., 2010; Tellex et al., 2011; Chen et al., 2010) or reinforcement learning (Branavan et al., 2012; Branavan et al., 2010).”
Tell me Dave: Context-sensitive grounding of natural language to manipulation instructions (2014, background): “The inverse problem of generating natural language queries/commentaries has also seen increasing interest in robotics [46, 10].”