DialoGLUE: A Natural Language Understanding Benchmark for Task-Oriented Dialogue
By Shikib Mehri, Mihail Eric, Dilek Z. Hakkani-Tür. 2020
148 citations recorded by the source corpus.
A long-standing goal of task-oriented dialogue research is the ability to flexibly adapt dialogue models to new domains. To progress research in this direction, we introduce DialoGLUE (Dialogue Language Understanding Evaluation), a public benchmark consisting of 7 task-oriented dialogue datasets covering 4 distinct natural language understanding tasks, designed to encourage dialogue research in representation-based transfer, domain adaptation, and sample-efficient task learning. We release several strong baseline models, demonstrating performance improvements over a vanilla BERT architecture and state-of-the-art results on 5 out of 7 tasks, by pre-training on a large open-domain dialogue corpus and task-adaptive self-supervised training. Through the DialoGLUE benchmark, the baseline methods, and our evaluation scripts, we hope to facilitate progress towards the goal of developing more general task-oriented dialogue models.
Subfields: Artificial Intelligence, Evaluation & Benchmarks, Dialogue & Conversational AI
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Jianfeng Gao
- Ivan Vulic
- Mohit Bansal
- Nan Duan
- Ming Zhou
- Weizhu Chen
- Jason Aaron Edward Weston
- Richard Socher
In Evaluation & Benchmarks
In Dialogue & Conversational AI
- Matthew Henderson
- Ajit Rajasekharan
- Pei-hao Su
- Sivanesan Sangeetha
- Katikapalli Subramanyam Kalyan
- Daxin Jiang
Citations in the corpus
- Preview, Attend and Review: Schema-Aware Curriculum Learning for Multi-Domain Dialogue State Tracking (2021, background): “The two closest baselines 2 , Con-vBERT (Mehri et al., 2020) and CoCoAug (Li et al., 2021), are also built upon TripPy, where Con-vBERT enhances its performance by using exter-nal large-scale conversational corpora to pre-train a BERT base and CoCoAug leverages a delicate counter-factual…”
- ConveRT: Efficient and Accurate Conversational Representations from Transformers (2019, background): “work will motivate and guide further developments in the areas of retrieval-based task-oriented dialog and large-scale pretraining for conversational applications (Mehri et al., 2020).”
- AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing (2021, methodology): “Besides, we have benchmarks like GLUECoS [269] and LinCE [268] for CodeSwitching, BLUE [48], BLURB [47] and ChineseBLUE [266] in the Biomedical domain, CodeXGLUE [35] in the Code intelligence domain and DialogGLUE [265] to evaluate Dialog models.”
- GLGE: A New General Language Generation Evaluation Benchmark (2020, background): “More recently, A new NLU benchmark called DialoGLUE (Mehri et al., 2020) for task-oriented dialogue is proposed.”
- Continual Few-shot Intent Detection (2022, background): “Popular approaches such as Goo et al. (2018); Qin et al. (2019); Mehri et al. (2020) have achieved promising performance.”