Investigating Speech Features for Continuous Turn-Taking Prediction Using LSTMs
By Matthew Roddy, Gabriel Skantze, Naomi Harte. 2018
44 citations recorded by the source corpus.
For spoken dialog systems to conduct fluid conversational interactions with users, the systems must be sensitive to turn-taking cues produced by a user. Models should be designed so that effective decisions can be made as to when it is appropriate, or not, for the system to speak. Traditional end-of-turn models, where decisions are made at utterance end-points, are limited in their ability to model fast turn-switches and overlap. A more flexible approach is to model turn-taking in a continuous manner using RNNs, where the system predicts speech probability scores for discrete frames within a future window. The continuous predictions represent generalized turn-taking behaviors observed in the training data and can be applied to make decisions that are not just limited to end-of-turn detection. In this paper, we investigate optimal speech-related feature sets for making predictions at pauses and overlaps in conversation. We find that while traditional acoustic features perform well, part-of-speech features generally perform worse than word features. We show that our current models outperform previously reported baselines.
Subfields: Artificial Intelligence, Experimental and Cognitive Psychology
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Experimental and Cognitive Psychology
Citations in the corpus
- Generative Spoken Dialogue Language Modeling (2022, neutral): “More recently, more sophisticated machine learning-based models of turn-taking have been introduced (Meena et al., 2014; Skantze, 2017; Roddy et al., 2018; Masumura et al., 2018).”
- TurnGPT: a Transformer-based Language Model for Predicting Turn-taking in Spoken Dialog (2020, methodology): “However, we argue that previous models that have combined linguistic and prosodic cues (cf. Meena et al. 2014; Skantze 2017; Roddy et al. 2018; Masumura et al. 2019) have used too simplistic models of linguistic turn-constructional units.”