Benchmarking Natural Language Understanding Services for building Conversational Agents
By Xingkun Liu, Arash Eshghi, Pawel Swietojanski, Verena Rieser. 2019
302 citations recorded by the source corpus.
We have recently seen the emergence of several publicly available Natural Language Understanding (NLU) toolkits, which map user utterances to structured, but more abstract, Dialogue Act (DA) or Intent specifications, while making this process accessible to the lay developer. In this paper, we present the first wide coverage evaluation and comparison of some of the most popular NLU services, on a large, multi-domain (21 domains) dataset of 25K user utterances that we have collected and annotated with Intent and Entity Type specifications and which will be released as part of this submission. The results show that on Intent classification Watson significantly outperforms the other platforms, namely, Dialogflow, LUIS and Rasa; though these also perform well. Interestingly, on Entity Type recognition, Watson performs significantly worse due to its low Precision. Again, Dialogflow, LUIS and Rasa perform well on this task.
Subfields: Artificial Intelligence, Evaluation & Benchmarks, Dialogue & Conversational AI
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Jianfeng Gao
- Philip S. Yu
- Percy Liang
- Mohit Bansal
- Richard Socher
- Dilek Z. Hakkani-Tür
- Bing Liu
- Ivan Vulic
In Evaluation & Benchmarks
- Dilek Z. Hakkani-Tür
- Yi-Ting Yeh
- Maxine Eskénazi
- Shikib Mehri
- Philip S. Yu
- Congying Xia
- Hoang Nguyen
- Chenwei Zhang
In Dialogue & Conversational AI
- Philip S. Yu
- Congying Xia
- Hoang Nguyen
- Chenwei Zhang
- Dilek Z. Hakkani-Tür
- Matthew Henderson
- Verena Rieser
- Maxine Eskénazi
Citations in the corpus
- Lifelong Intent Detection via Multi-Strategy Rebalancing (2021, methodology): “Based on four widely used datasets, ATIS [10], SNIPS [5], HWU64 [16], CLINC150 [14], we constructed four benchmarks.”
- Prompt Augmented Generative Replay via Supervised Contrastive Learning for Lifelong Intent Detection (2022, background): “HWU64 (Liu et al., 2019) is a crowdsourced multi-domain ( 21 domains such as alarm, cooking etc.) intent detection dataset.”
- An Evaluation Dataset for Intent Classification and Out-of-Scope Prediction (2019, background): “The dataset presented in Liu et al. (2019) has a large number of intent classes, yet also contains a wide range of samples per intent class (ranging from 24 to 5,981 queries per intent, and so is not constrained in all cases).”
- CG-BERT: Conditional Text Generation with BERT for Generalized Few-shot Intent Detection (2020, methodology): “We use the Fold 7 of the NLUED dataset with the 5 shot setting as an example.”
- Semantic Matching and Aggregation Network for Few-shot Intent Detection (2020, methodology): “NLUE: Following (Liu et al., 2019b), we utilize a subset of utterances covering 64 intents.”
- InstructDial: Improving Zero and Few-shot Generalization in Dialogue through Instruction Tuning (2022, background): “Instruction Tuning Constructing natural language prompts to perform NLP tasks is an active area of research (Schick and Schütze, 2021; Liu et al., 2021a).”
- DialoGLUE: A Natural Language Understanding Benchmark for Task-Oriented Dialogue (2020, background): “For intent prediction (BANKING77, CLINC150, HWU64) we use accuracy.”
- NeMo Guardrails: A Toolkit for Controllable and Safe LLM Applications with Programmable Rails (2023, methodology): “There exist a wide range of tools and solutions for both NLU and DM, ranging from open-source solutions like Rasa (Bocklisch et al., 2017) to proprietary platforms, such as Microsoft LUIS or Google DialogFlow (Liu et al., 2021).”
- Efficient Intent Detection with Dual Sentence Encoders (2020, background): “It follows the very recent endeavor of procuring high-quality intent detection data (Liu et al., 2019a; Larson et al., 2019), but is very different in nature than the other datasets.”
- SLURP: A Spoken Language Understanding Resource Package (2020, methodology): “This data has been manually annotated at scenario, action and entity level, and released as a text-only NLU benchmark (Liu et al., 2019).”
- MASSIVE: A 1M-Example Multilingual Natural Language Understanding Dataset with 51 Typologically-Diverse Languages (2022, methodology): “…2020; Goyal et al., 2021), while the growing number and 1https://github.com/alexa/massive size of language models (mBERT (Devlin, 2018), RoBERTa (Liu et al., 2019b), XLM (Lample and Conneau, 2019), XLM-R (Conneau et al., 2020), mBART (Liu et al., 2020), MARGE (Lewis et al., 2020), and mT5 (Xue…”
- Jamba: A Hybrid Transformer-Mamba Language Model (2024, background): “The average input length in these datasets ranges from 6K to 62K tokens.”
- Prompt Conditioned VAE: Enhancing Generative Replay for Lifelong Learning in Task-Oriented Dialogue (2022, methodology): “…intent detection and slot filling based on public NLU benchmarks: For intent detection, we collect six datasets that carry intent annotations: HWU (Liu et al., 2019), BANKING (Casanueva et al., 2020), CLINC (Lar-son et al., 2019), SNIPS (Coucke et al., 2018), AITS (Hemphill et al., 1990), and TOP…”