Speaking with your Sidekick: Understanding Situated Speech in Computer Role Playing Games

By Peter Gorniak, Deb K. Roy. 2005

52 citations recorded by the source corpus.

Speech and natural language are natural and convenient ways to interact with artificial characters. Current use of language in games, however, is limited to menu systems and interplayer communication. To achieve smooth linguistic communication with synthetic agents, research should focus on how language connects to the situation in which it occurs. Taking account of the physical scene (where is the speaker located, what is around her, when does she speak?) as well as the functional aspects of the situation (why did he choose to speak? What are his likely plans?) can disambiguate the linguistic signal in form and content. We present a game environment to collect time synchronized speech and action streams, to visualize these data and to annotate them at different stages of processing. We further sketch a framework for situated speech understanding on such data, taking into account aspects of the physical situation as well as the plans players follow. Our results show that this combination of influences achieves remarkable improvements over the individual situation models despite the very noisy and spontaneous nature of the speech involved. This work provides a basis for developing characters that use situated natural spoken language to communicate meaningfully with human players.

Subfields: Artificial Intelligence, Dialogue & Conversational AI, Speech & Audio

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Dialogue & Conversational AI

In Speech & Audio

Citations in the corpus

Learning to Win by Reading Manuals in a Monte-Carlo Framework (2011, background): “Gorniak and Roy (2005) develop a machine controlled game character which responds to spoken natural language commands.”
Training a Multilingual Sportscaster: Using Perceptual Context to Learn Language (2014, background): “In addition to observing events passively, there has also been work on grounded language learning in more interactive environments such as in computer video games (Gorniak & Roy, 2005).”
Learning to sportscast: a test of grounded language acquisition (2008, background): “There has also been recent work on grounded language learning in simulated computer-game environments (Gorniak & Roy, 2005).”
Learning to Connect Language and Perception (2008, background): “In particular, interactive video-games can provide rich simulated environments for grounded language learning (Fleischman & Roy 2005; Gorniak & Roy 2005).”