NLP's Clever Hans Moment has Arrived

By Benjamin Heinzerling. 2020

26 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Evaluation & Benchmarks

Citations in the corpus

AI and the Everything in the Whole Wide World Benchmark (2021, background): “Heinzerling [2019], in a discussion inspired by Niven and Kao [2019], suggests various data ablations that can be applied to test data as well, to investigate what cues a system might be relying on.”
Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data (2020, background): “One approach to providing grounding is to train distributional models on corpora augmented with perceptual data, such as photos (Hossain et al., 2019) or other modalities (Kiela and Clark, 2015; Kiela et al., 2015).”
Do Datasets Have Politics? Disciplinary Values in Computer Vision Dataset Development (2021, neutral): “Dataset development is rarely glamorized like algorithmic developments, which is reflected in peer review processes that make it difficult to publish work focusing exclusively on datasets [54].”
Bringing the People Back In: Contesting Benchmark Machine Learning Datasets (2020, background): “…curriculum (e.g. Goodfellow et al. (2016)), the under-speciﬁcation or data decisions in publications accompanying new datasets (Geiger et al., 2020; Scheuerman et al., 2020), and the routine undervaluing of the work that goes into the construction of datasets (Heinzerling, 2019; Jo & Gebru, 2020).”
On the genealogy of machine learning datasets: A critical history of ImageNet (2021, background): “2021) and publications that focus solely on dataset creation tend to be devalued within traditional peer-review processes (Heinzerling, 2019).”
Reduced, Reused and Recycled: The Life of a Dataset in Machine Learning Research (2021, background): “SOTA-chasing concerns are also compounded by the great capacity ML algorithms have to be “right for the wrong reason” [30], enabling SOTA results that rely on “shortcuts” rather than learning the causal structure dictated by the task [13].”