Can Prosody Aid the Automatic Classification of Dialog Acts in Conversational Speech?

By Elizabeth Shriberg, Rebecca A. Bates, Andreas Stolcke, Paul A. Taylor, Dan Jurafsky, Klaus Ries, Noah B. Coccaro, Rachel Martin, Marie W. Meteer, Carol Van Ess-Dykema. 1998

Identifying whether an utterance is a statement, question, greeting, and so forth is integral to effective automatic understanding of natural dialog. Little is known, however, about how such dialog acts (DAs) can be automatically classified in truly natural conversation. This study asks whether current approaches, which use mainly word information, could be improved by adding prosodic information. The study is based on more than 1000 conversations from the Switchboard corpus. DAs were hand-annotated, and prosodic features (duration, pause, F0, energy, and speaking rate) were automatically extracted for each DA. In training, decision trees based on these features were inferred; trees were then applied to unseen test data to evaluate performance. Performance was evaluated for prosody models alone, and after combining the prosody models with word information—either from true words or from the output of an automatic speech recognizer. For an overall classification task, as well as three subtasks, prosody made significant contributions to classification. Feature-specific analyses further revealed that although canonical features (such as F0 for questions) were important, less obvious features could compensate if canonical features were removed. Finally, in each task, integrating the prosodic model with a DA-specific statistical language model improved performance over that of the language model alone, especially for the case of recognized words. Results suggest that DAs are redundantly marked in natural conversation, and that a variety of automatically extractable prosodic features could aid dialog processing in speech applications.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Text-Free Prosody-Aware Generative Spoken Language Modeling (2021, background): “…prosody information can improve the performance from text-only models on speech segmentation (Shriberg et al., 2000), dialogue act classiﬁcation (Shriberg et al., 1998; Ward and Tsukahara, 2000), syntactic parsing (Tran et al., 2017), speech–language pathology (Cohen et al., 2019), ASR…”
Prosody-based automatic segmentation of speech into sentences and topics (2000, background): “In a similar vein, although prosodic features depend on the type of events one is trying to detect, the general approach could be extended to tasks beyond sentence and topic segmentation (see for example (Hakkani-T ur et al., 1999; Shriberg et al., 1998)).”
Dialog Act Modeling for Conversational Speech (1998, background): “On Lhe oLher hand, Lo improve performanc:e we will have to revisit om independence assumptions, as ,vell as examine additional knowledge sources.”
Turn-taking cues in task-oriented dialogue (2011, background): “Simultaneous speech poses serious difficulties for the ASR components of IVR systems (Shriberg et al., 2001).”
Dialog act modelling for conversational speech (1998, methodology): “Second, we built a discourse grammar that incorporated constraints on DA sequences in a nonhierarchical way, using maximum entropy (ME) estimation ( Rosenfeld 1996 ).”