SemEval-2018 Task 11: Machine Comprehension Using Commonsense Knowledge

By Simon Ostermann, Michael Roth, Ashutosh Modi, Stefan Thater, Manfred Pinkal. 2018

113 citations recorded by the source corpus.

This report summarizes the results of the SemEval 2018 task on machine comprehension using commonsense knowledge. For this machine comprehension task, we created a new corpus, MCScript. It contains a high number of questions that require commonsense knowledge for finding the correct answer. 11 teams from 4 different countries participated in this shared task, most of them used neural approaches. The best performing system achieves an accuracy of 83.95%, outperforming the baselines by a large margin, but still far from the human upper bound, which was found to be at 98%.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Improving Question Answering with External Knowledge (2019, background): “A growing number of studies concentrate on the construction of multiple-choice machine reading comprehension (Mostafazadeh et al., 2016; Lai et al., 2017; Khashabi et al., 2018; Ostermann et al., 2018; Sun et al., 2019) or question answering tasks (Clark et al.”
Dual Co-Matching Network for Multi-choice Reading Comprehension (2019, methodology): “…(Mostafazadeh et al. 2016), SemEval-2018 Task 11 (Ostermann et al. 2018), MCTest (Richardson, Burges, and Renshaw 2013), COIN Shared Task 1 (Ostermann et al. 2018)) and achieve the absolute improvement of 4.9% and 2.8% in average accuracy from directly ﬁne-tuned BERT and XLNet,…”
Unsupervised Commonsense Question Answering with Self-Talk (2020, background): “, 2016) and machine comprehension tasks (Kočiskỳ et al., 2018; Ostermann et al., 2018; Clark et al., 2018; Talmor et al., 2019).”
“Going on a vacation” takes longer than “Going for a walk”: A Study of Temporal Commonsense Understanding (2019, background): “This relates our work to a large body of work on question-answering, an area that has seen significant progress in the past few years (Clark et al., 2018; Ostermann et al., 2018; Merkhofer et al., 2018).”
Getting Closer to AI Complete Question Answering: A Set of Prerequisite Real Tasks (2020, background): “It also lends itself naturally to commonsense reasoning tasks, where most datasets are based on crowdsourced narratives (MCScript (Ostermann et al. 2018), RocStories (Mostafazadeh et al.”
Enhancing Pre-Trained Language Representations with Rich Knowledge for Machine Reading Comprehension (2019, methodology): “Incorporating KBs Several MRC datasets that require external knowledge have been proposed, such as ReCoRD (Zhang et al., 2018), ARC (Clark et al., 2018), MCScript (Ostermann et al., 2018), OpenBookQA (Mihaylov et al., 2018) and Com-monsenseQA (Talmor et al., 2018).”
Convolutional Spatial Attention Model for Reading Comprehension with Multiple-Choice Questions (2018, methodology): “• SemEval-2018 Task11 : The dataset provided by the SemEval-2018 Task11 organizer, which mainly focus on solving commonsense reading comprehension.”
Probing Prior Knowledge Needed in Challenging Chinese Machine Reading Comprehension (2019, background): “Compared to relevant datasets (Ostermann et al., 2018; Sun et al., 2019a), besides the oral text, we also need”
TransOMCS: From Linguistic Graphs to Commonsense Knowledge (2020, methodology): “We conduct experiments on two downstream tasks: common-sense reading-comprehension [ Ostermann et al. , 2018 ] and dialogue generation [ Li et al. , 2017 ] .”
Evidence Sentence Extraction for Machine Reading Comprehension (2019, background): “…number of questions (e.g., 87% questions in RACE (Lai et al., 2017; Sun et al., 2019)) are not extractive, which require advanced reading skills such as inference over multiple sentences and utilization of prior knowledge (Lai et al., 2017; Khashabi et al., 2018; Ostermann et al., 2018).”
Yuanfudao at SemEval-2018 Task 11: Three-way Attention and Relational Knowledge for Commonsense Machine Comprehension (2018, methodology): “Statistics for official dataset MCScript (Ostermann et al., 2018a) are shown in Table 1.”
Evidence Extraction for Machine Reading Comprehension with Deep Probabilistic Logic (2018, methodology): “7 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 67”
SemEval-2020 Task 4: Commonsense Validation and Explanation (2020, background): “…with adversarial generations (Zellers et al., 2018), event validation (Wang et al., 2018), reading comprehension (Mostafazadeh et al., 2016; Ostermann et al., 2018b; Ostermann et al., 2018a), dialogue (Cui et al., 2020) and QA problems (Davis, 2016; Talmor et al., 2018; Mihaylov et al.,…”
Analytical Reasoning of Text (2022, background): “There exist several MRC datasets that require commonsense knowledge (Ostermann et al., 2018; Zhang et al., 2018; Huang et al., 2019b).”
WinoWhy: A Deep Diagnosis of Essential Commonsense Knowledge for Answering Winograd Schema Challenge (2020, background): “Commonsense reasoning, as an important problem of natural language understanding, has at-tracted much more attention in the NLP community recently (Levesque et al., 2012; Zhou et al., 2018; Ostermann et al., 2018; Talmor et”