SimpleQuestions Nearly Solved: A New Upperbound and Baseline Approach
By Michael Petrochuk, Luke S. Zettlemoyer. 2018
98 citations recorded by the source corpus.
The SimpleQuestions dataset is one of the most commonly used benchmarks for studying single-relation factoid questions. In this paper, we present new evidence that this benchmark can be nearly solved by standard methods. First, we show that ambiguity in the data bounds performance at 83.4%; many questions have more than one equally plausible interpretation. Second, we introduce a baseline that sets a new state-of-the-art performance level at 78.1% accuracy, despite using standard methods. Finally, we report an empirical analysis showing that the upperbound is loose; roughly a quarter of the remaining errors are also not resolvable from the linguistic signal. Together, these results suggest that the SimpleQuestions dataset is nearly solved.
Subfields: Artificial Intelligence, Information Systems, Question Answering & Reading Comprehension, Knowledge Graphs & Representation
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Information Systems
In Knowledge Graphs & Representation
Citations in the corpus
- FreebaseQA: A New Factoid QA Data Set Matching Trivia-Style Question-Answer Pairs with Freebase (2019, background): “According to (Petrochuk and Zettlemoyer, 2018), SimpleQuestions is nearly solved with only standard neural network meth-ods if its linguistic ambiguity is taken into account.”