DSTC7 Task 1: Noetic End-to-End Response Selection

By Chulaka Gunasekara, Jonathan K. Kummerfeld, Lazaros C. Polymenakos, Walter S. Lasecki. 2019

52 citations recorded by the source corpus.

Goal-oriented dialogue in complex domains is an extremely challenging problem and there are relatively few datasets. This task provided two new resources that presented different challenges: one was focused but small, while the other was large but diverse. We also considered several new variations on the next utterance selection problem: (1) increasing the number of candidates, (2) including paraphrases, and (3) not including a correct option in the candidate set. Twenty teams participated, developing a range of neural network models, including some that successfully incorporated external data to boost performance. Both datasets have been publicly released, enabling future work to build on these results, working towards robust goal-oriented dialogue systems.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Poly-encoders: Architectures and Pre-training Strategies for Fast and Accurate Multi-sentence Scoring (2019, background): “The former is a task extensively studied and recently featured in two competitions: the Neurips ConvAI2 competition (Dinan et al., 2020), and the DSTC7 challenge, Track 1 (Yoshino et al., 2019; Jonathan K. Kummer-feld & Lasecki, 2018; Chulaka Gunasekara & Lasecki, 2019).”
Domain-matched Pre-training Tasks for Dense Retrieval (2021, methodology): “ConvAI2 DSTC7 Ubuntu v2 R@1 R@1 MRR R@1 MRR (Wolf et al., 2019) BERT base (though this could be contingent on the quality of additional generated questions).”
Survey on evaluation methods for dialogue systems (2019, unknown): “DSTC7 (Flex Data) Student guiding 500 (Gunasekara et al. 2019)”
ConveRT: Efficient and Accurate Conversational Representations from Transformers (2019, methodology): “Retrieval-based dialog is popular because posing dialog as response selection (Gunasekara et al., 2019) simplifies system design (Boussaha et al.”
Distilling Knowledge for Fast Retrieval-based Chat-bots (2020, methodology): “The Ubuntu Dialogue Corpus (UDC) [10] and DSTC7 sentence selection track [2] are collected form a chatroom dedicated to the support of the Ubuntu operating system.”
A Repository of Conversational Datasets (2019, methodology): “This 1-of-100 accuracy metric has been used in previous studies: (Al-Rfou et al., 2016; Henderson et al., 2017; Cer et al., 2018; Kumar et al., 2018; Yang et al., 2018; Gunasekara et al., 2019).”