Understanding Deep Learning Performance through an Examination of Test Set Difficulty: A Psychometric Case Study

By John P. Lalor, Hao Wu, Tsendsuren Munkhdalai, Hong Yu. 2017

41 citations recorded by the source corpus.

Interpreting the performance of deep learning models beyond test set accuracy is challenging. Characteristics of individual data points are often not considered during evaluation, and each data point is treated equally. In this work we examine the impact of a test set question’s difficulty to determine if there is a relationship between difficulty and performance. We model difficulty using well-studied psychometric methods on human response patterns. Experiments on Natural Language Inference (NLI) and Sentiment Analysis (SA) show that the likelihood of answering a question correctly is impacted by the question’s difficulty. In addition, as DNNs are trained on larger datasets easy questions start to have a higher probability of being answered correctly than harder questions.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Understanding Dataset Difficulty with V-Usable Information (2021, neutral): “Item response theory (IRT; Embretson & Reise, 2013) allows the difficulty of instances to be learned via parameters in a probabilistic model meant to explain model performance (Lalor et al., 2018; Rodriguez et al., 2021).”
Information-Theoretic Measures of Dataset Difficulty (2021, background): “Other work has treated instance difﬁculty as learned parameters, allowing estimates to vary depending on what the generative process and parameter distributions are assumed to be (Lalor et al., 2018; Rodriguez et al., 2021).”
Deep Learning Through the Lens of Example Difficulty (2021, background): “There are a number of definitions of example difficulty in the literature (E.g. see Carlini et al. (2019); Hooker et al. (2019); Lalor et al. (2018); Agarwal and Hooker (2020)).”
Evaluation Examples are not Equally Informative: How should that change NLP Leaderboards? (2021, methodology): “We test IRT as a way to guide annotation, but it can also train NLP models; for example, deep models learn “easy” examples faster (Lalor et al., 2018) and maintain test accuracy when training data are down-sampled (Lalor et al.”