Reporting bias and knowledge acquisition

By Jonathan Gordon, Benjamin Van Durme. 2013

235 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Information Systems, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Natural Language Processing

Citations in the corpus

HellaSwag: Can a Machine Really Finish Your Sentence? (2019, background): “To answer some Hella-Swag questions correctly without reasoning deeply – like knowing that it is a bad idea to stop at a red light for ‘at most two seconds’ – might require an exponential number of samples, due to prob-8 lems of reporting bias (Gordon and Van Durme, 2013).”
ReCoRD: Bridging the Gap between Human and Machine Commonsense Reading Comprehension (2018, background): “…most of the existing MRC datasets, all queries and passages in Re Co RD are automatically mined from news articles, which maximally reduces the human elicitation bias (Gordon and Van Durme, 2013; Misra et al., 2016; Zhang et al., 2017), and the data collection method we propose is cost-efﬁcient.”
oLMpics-On What Language Model Pre-training Captures (2019, background): “Reporting bias (Gordon and Van Durme, 2013) may play a roll in LMs inability to correctly determine cases such as “A rhinoceros NEVER has fur.” Interestingly, behavioral research conducted on blind humans shows they exhibit a similar bias (Kim et al., 2019).”
UNICORN on RAINBOW: A Universal Commonsense Reasoning Model on a New Multitask Benchmark (2021, background): “Due to reporting bias (Gordon and Van Durme 2013), common sense rarely appears explicitly in text, though it does appear implicitly.”
Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMs (2022, background): “Also known as reporting bias (Gordon and Van Durme, 2013; Lucy and Gauthier, 2017), this phenomenon likely limits LLMs’ ability to learn social commonsense knowledge from static text.”
MERLOT RESERVE: Neural Script Knowledge through Vision and Language and Sound (2022, background): “We suspect (but cannot definitively prove) that the reporting bias of the YouTube data that it was trained onmight make it poor for such a surveillance-focused task [49].”
MERLOT: Multimodal Neural Script Knowledge Models (2021, background): “Temporal self-supervised learning from transcripts may not lead to as powerful zero-shot object detectors because speakers in videos may be less likely to state the obvious [41, 39], e.”
Women also Snowboard: Overcoming Bias in Captioning Models (2018, background): “Natural language data is also subject to reporting bias [11,7,12,13] in which people over-report less common co-occurrences, such as “male nurse” [7] or “green banana” [12].”
Commonsense Knowledge Mining from Pretrained Models (2019, positive): “, 2016; Jastrzebski et al., 2018). Jastrzebski et al. (2018) demonstrated that much of the data in the ConceptNet test set were simply rephrased relations from the training set, and that this train-test set leakage led to artificially inflated test performance metrics.”
Unsupervised Commonsense Question Answering with Self-Talk (2020, background): “Despite the performance boost, LMs as knowledge providers suffer from various shortcomings: (i) insufﬁcient coverage: due to reporting bias, many trivial facts might not be captured by LMs (purple set in Figure 1), because they are rarely written about (Gordon and Van Durme, 2013).”
Surface Form Competition: Why the Highest Probability Answer Isn’t Always Right (2021, background): “Probability, possibility and plausibility are distinct (Van der Helm, 2006), but reporting bias (Gordon and Van Durme, 2013) means that language models only model what people are likely to write (on websites that are easily crawled).”
Inducing Relational Knowledge from BERT (2019, background): “However, for many commonsense relations, we may not have such sentences as commonsense knowledge is rarely asserted explicitly (Gordon and Durme 2013), in which case we have to instead rely on sentences providing indirect evidence, such as (1)–(3).”
Commonsense Knowledge Base Completion (2016, background): “This knowledge is rarely expressed explicitly in textual corpora (Gordon and Van Durme, 2013).”
Don't just listen, use your imagination: Leveraging visual common sense for non-visual tasks (2015, background): “First, it has been shown that people tend not to explicitly talk about common sense knowledge in text [20].”
G-DAug: Generative Data Augmentation for Commonsense Reasoning (2020, background): “In contrast, answering commonsense questions relies on commonsense notions that are seldom stated explicitly (Gordon and Van Durme, 2013; Forbes and Choi, 2017), and authoring such questions can require creativity (see Figure 1).”
CREAK: A Dataset for Commonsense Reasoning over Entity Knowledge (2021, background): “Commonsense Reasoning Commonsense reasoning tasks [Levesque et al., 2011, Zellers et al., 2018, Talmor et al., 2019, Lourie et al., 2021, inter alia] evaluate models’ reasoning skills in the physical world, with reporting bias being a principal challenge [Gordon and Van Durme, 2013].”
Understanding Natural Language Understanding Systems. A Critical Analysis (2023, background): “This is an effect of the so-called reporting bias (Gordon and Van Durme 2013), according to which information that speakers assume to share with the listeners is likely to be omitted from their linguistic productions.”