A Knowledge Hunting Framework for Common Sense Reasoning

By Ali Emami, Noelia De La Cruz, Adam Trischler, Kaheer Suleman, Jackie Chi Kit Cheung. 2018

41 citations recorded by the source corpus.

We introduce an automatic system that achieves state-of-the-art results on the Winograd Schema Challenge (WSC), a common sense reasoning task that requires diverse, complex forms of inference and knowledge. Our method uses a knowledge hunting module to gather text from the web, which serves as evidence for candidate problem resolutions. Given an input problem, our system generates relevant queries to send to a search engine, then extracts and classifies knowledge from the returned results and weighs them to make a resolution. Our approach improves F1 performance on the full WSC by 0.21 over the previous best and represents the first system to exceed 0.5 F1. We further demonstrate that the approach is competitive on the Choice of Plausible Alternatives (COPA) task, which suggests that it is generally applicable.

Subfields: Artificial Intelligence, Natural Language Processing, Prompting & Reasoning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Prompting & Reasoning

Citations in the corpus

A Surprisingly Robust Trick for the Winograd Schema Challenge (2019, methodology): “Previous work is based on Google queries for knowledge (Emami et al., 2018) ( 58% ), sequence ranking (Opitz and Frank, 2018) ( 63% ), and using an ensemble of LMs (Trinh and Le, 2018) ( 63% ).”
WinoWhy: A Deep Diagnosis of Essential Commonsense Knowledge for Answering Winograd Schema Challenge (2020, methodology): “an unsupervised way by leveraging either search engines (Emami et al., 2018), linguistic knowledge (Zhang et al.”
On the Evaluation of Common-Sense Reasoning in Natural Language Understanding (2018, methodology): “As of today, state-of-the-art accuracy on the WSC for single model performance is around 55% [8, 1].”
WikiCREM: A Large Unsupervised Corpus for Coreference Resolution (2019, background): “Another approach involves using external knowledge bases (Emami et al., 2018; Fähndrich et al., 2018), however, the accuracy of these models still lags behind that of the aforementioned pre-trained models.”
How Reasonable are Common-Sense Reasoning Tasks: A Case-Study on the Winograd Schema Challenge and SWAG (2018, unknown): “This allows for simplifying assumptions (Liu et al., 2016) or schematizations (Emami et al., 2018).”
ASER: A Large-scale Eventuality Knowledge Graph (2019, background): “Knowledge Hunting [14] 119 79 75 60.”
Contrastive Self-Supervised Learning for Commonsense Reasoning (2020, unknown): “Hunter (Emami et al., 2018) 57.1 % GPT-ft (Kocijan et al., 2019) 67.4 % BERT-ft (Kocijan et al., 2019) 71.4 % WSC-273 (unsupervised) Single LMs (Trinh and Le, 2018) 54.5 % MAS (Klein and Nabi, 2019) 60.3 % DSSM (Wang et al., 2019) 63.0 % Ensemble LMs (Trinh and Le, 2018) 63.8 % CSS (Proposed…”