SelQA: A New Benchmark for Selection-Based Question Answering

By Tomasz Jurczyk, Michael Zhai, Jinho D. Choi. 2016

42 citations recorded by the source corpus.

This paper presents a new selection-based question answering dataset, SelQA. The dataset consists of questions generated through crowdsourcing and sentence length answers that are drawn from the ten most prevalent topics in the English Wikipedia. We introduce a corpus annotation scheme that enhances the generation of large, diverse, and challenging datasets by explicitly aiming to reduce word co-occurrences between the question and answers. Our annotation scheme is composed of a series of crowdsourcing tasks with a view to more effectively utilize crowdsourcing in the creation of question answering datasets in various domains. Several systems are compared on the tasks of answer sentence selection and answer triggering, providing strong baseline results for future work to improve upon.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Coarse-to-Fine Question Answering for Long Documents (2016, methodology): “Answer sentence selection is studied with the TREC QA (Voorhees and Tice, 2000), Wik-iQA (Yang et al., 2016b) and SelQA (Jurczyk et al., 2016) datasets.”
Sanity Check: A Strong Alignment and Information Retrieval Baseline for Question Answering (2018, unknown): “# Supervised Model MAP 1 No Wgt Word Cnt [29] 50.99 2 Yes LCLR [29] 59.93 3 No Our model (One-to-one) 62.77 4 No Our model (One-to-all) 60.91 5 Yes Yang et al. [29] CNN+Cnt 65.20 6 Yes Jurczyk et al. [11]RNN-1way 66.64 7 Yes Jurczyk et al. [11] RNN-Attention_pool 67.47 8 Yes dos Santos et al. [7] 68”
Deep Learning--based Text Classification (2020, methodology): “Other datasets for QA includes SituationsWithAdversarial Generations (SWAG) [201],WikiQA [199], SelQA [202].”
Semantic Sentence Matching with Densely-connected Recurrent and Co-attentive Information (2018, unknown): “0.802 0.875 HyperQA [35] 0.770 0.825 Comp.-Aggr. [1] 0.821 0.899 PR+CNN [29] 0.780 0.834 IWAN [33] 0.822 0.889 DRCN 0.804 0.862 DRCN 0.830 0.908 (a) TrecQA: raw and clean Models MAP MRR RNN:attn-pool [18] 0.864 0.876 CNN-DAN [31] 0.866 0.873 CNN-hinge [31] 0.876 0.881 ACNN [32] 0.874 0.880 AdaQA [32”
Alignment over Heterogeneous Embeddings for Question Answering (2019, unknown): “70 10 Yes RNN-Attention pool (Jurczyk et al., 2016) 67.”
Adaptive Convolutional Filter Generation for Natural Language Understanding (2017, methodology): “Baselines We also compare our ACNN models with twelve strong baseline methods, categorized as follows: (i) Word Cnt, Wgt Word Cnt, LCLR (Yang, Yih, and Meek, 2015): these methods leverage lexical semantic features for prediction; (ii) Paragraph vector (PV) (Le and Mikolov, 2014): an unsupervised alg”
Analysis of Wikipedia-based Corpora for Question Answering (2018, background): “, 2016) and answer triggering (Yang et al., 2015; Jurczyk et al., 2016) find answer sentences instead, although the presence of the answer context is not assumed within the provided document for answer triggering but it is for the other two tasks.”
Learning Context-Sensitive Convolutional Filters for Text Processing (2017, methodology): “To evaluate the effectiveness of proposed AdaQA model, we compare it with several CNN-based sequence matching baselines, including Vanilla CNN (Jurczyk et al., 2016; Santos et al., 2017), attentive pooling networks (dos Santos et al.”