Speech Intention Classification with Multimodal Deep Learning

By Yue Gu, Xinyu Li, Shuhong Chen, Jianyu Zhang, Ivan Marsic. 2017

51 citations recorded by the source corpus.

We present a novel multimodal deep learning structure that automatically extracts features from textual-acoustic data for sentence-level speech classification. Textual and acoustic features were first extracted using two independent convolutional neural network structures, then combined into a joint representation, and finally fed into a decision softmax layer. We tested the proposed model in an actual medical setting, using speech recording and its transcribed log. Our model achieved 83.10% average accuracy in detecting 6 different intentions. We also found that our model using automatically extracted features for intention classification outperformed existing models that use manufactured features.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Ivan Marsic
2. Shuhong Chen
3. Shiyu Fu
4. Xinyu Li
5. Kangning Yang
6. Yue Gu

Citations in the corpus

• Privacy-Preserving Adversarial Representation Learning in ASR: Reality or Illusion? (2019, neutral): “Indeed, state-of-the-art speech processing algorithms can infer not only the spoken contents from the speech signal, but also the speaker’s identity [5], intention [6–9], gender [10, 11], emotional state [12–14], pathological condition [15–17], personality [18, 19] and cultural [20, 21] attributes t”
• Multimodal Affective Analysis Using Hierarchical Attention Strategy with Word-Level Alignment (2018, background): “Secondly, it has more dimensions in the frequency domain that aid learning in deep models (Gu et al., 2017).”