Evaluating Natural Language Understanding Services for Conversational Question Answering Systems

By Daniel Braun, Adrian Hernandez-Mendez, Florian Matthes, Manfred Langen. 2017

192 citations recorded by the source corpus.

Conversational interfaces recently gained a lot of attention. One of the reasons for the current hype is the fact that chatbots (one particularly popular form of conversational interfaces) nowadays can be created without any programming knowledge, thanks to different toolkits and so-called Natural Language Understanding (NLU) services. While these NLU services are already widely used in both, industry and science, so far, they have not been analysed systematically. In this paper, we present a method to evaluate the classification performance of NLU services. Moreover, we present two new corpora, one consisting of annotated questions and one consisting of annotated questions with the corresponding answers. Based on these corpora, we conduct an evaluation of some of the most popular NLU services. Thereby we want to enable both, researchers and companies to make more educated decisions about which service they should use.

Subfields: Artificial Intelligence, Computer Science Applications

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

An Evaluation Dataset for Intent Classification and Out-of-Scope Prediction (2019, background): “(Li and Roth, 2002) 6 5,952 TREC-50, (Li and Roth, 2002) 50 5,952 Web Apps, (Braun et al., 2017) 8 89 Ask Ubuntu, (Braun et al., 2017) 5 162 Chatbot Corpus, (Braun et al., 2017) 2 206 Snips, (Coucke et al., 2018) 7 14,484 Liu et al. (2019) 54 25,716 Table 4: Classiﬁcation dataset properties.”
Snips Voice Platform: an embedded Spoken Language Understanding system for private-by-design voice interfaces (2018, methodology): “*Benchmark run in August 2017 by the authors of [8].”
Rasa: Open Source Language Understanding and Dialogue Management (2017, background): “Braun et al [3] show that Rasa NLU’s performance compares favourably to various closed-source solutions.”
Efficient Intent Detection with Dual Sentence Encoders (2020, background): “The previous standard datasets such as Web Apps, Ask Ubuntu, the Chatbot Corpus (Braun et al., 2017) or SNIPS (Coucke et al., 2018) are limited to only a small number of classes ( 10 ), which oversimpliﬁes the intent detection task and does not emulate the true environment of commercial systems.”
Deep Learning Inference in Facebook Data Centers: Characterization, Performance Optimizations and Hardware Implications (2018, methodology): “This approach has been successfully applied to many natural language processing tasks, such as summarization [17, 46], speech recognition [23], syntactic and semantic parsing [11, 69] as well as question answering and dialog systems [9, 52].”