Competency Problems: On Finding and Removing Artifacts in Language Data

By Matt Gardner, William Merrill, Jesse Dodge, Matthew E. Peters, Alexis Ross, Sameer Singh, Noah A. Smith. 2021

124 citations recorded by the source corpus.

Much recent work in NLP has documented dataset artifacts, bias, and spurious correlations between input features and output labels. However, how to tell which features have “spurious” instead of legitimate correlations is typically left unspecified. In this work we argue that for complex language understanding tasks, all simple feature correlations are spurious, and we formalize this notion into a class of problems which we call competency problems. For example, the word “amazing” on its own should not give information about a sentiment label independent of the context in which it appears, which could include negation, metaphor, sarcasm, etc. We theoretically analyze the difficulty of creating data for competency problems when human bias is taken into account, showing that realistic datasets will increasingly deviate from competency problems as dataset size increases. This analysis gives us a simple statistical test for dataset artifacts, which we use to show more subtle biases than were described in prior work, including demonstrating that models are inappropriately affected by these less extreme biases. Our theoretical treatment of this problem also allows us to analyze proposed solutions, such as making local edits to dataset instances, and to give recommendations for future data collection and model design efforts that target competency problems.

Subfields: Artificial Intelligence, Evaluation & Benchmarks, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Evaluation & Benchmarks

In Safety, Robustness & Interpretability

Citations in the corpus

Understanding Dataset Difficulty with V-Usable Information (2021, neutral): “Our approach to discovering dataset arte-facts can also complement existing approaches to artefact discovery (Gardner et al., 2021; Pezeshkpour et al., 2021; Le Bras et al., 2020).”
WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creation (2022, background): “All points representing (word, label) pairs above the blue line have detectable correlations (Gardner et al., 2021). premise-only model trained on W A NLI achieves an accuracy of 42 .”
Tailor: Generating and Perturbing Text with Semantic Controls (2021, background): “Gardner et al. (2021) devise a statistical test for dataset artifacts that builds on the argument that no simple feature ( e.g., single token) should show statistically signiﬁcant correlation with labels in a language understanding problem.”
Impact of Pretraining Term Frequencies on Few-Shot Reasoning (2022, background): “A general statistical procedure to test for artifacts is presented in Gardner et al. (2021), who also theoretically show that large datasets are almost certain to contain artifacts under reasonable assumptions.”
Generating Data to Mitigate Spurious Correlations in Natural Language Inference Datasets (2022, methodology): “Our approach is based on the theoretical framework proposed by Gardner et al. (2021) to measure these correlations, known as z-statistics .”
Information-Theoretic Measures of Dataset Difficulty (2021, unknown): “Our approach to discovering dataset artefacts can also complement existing approaches to artefact discovery (Gardner et al., 2021; Pezeshkpour et al., 2021).”
Combining Feature and Instance Attribution to Detect Artifacts (2021, methodology): “In addition to the model-centered diagnostics we have focused on in this work, we also consider a few dataset-centered approaches for artifact discovery: (1) PMI (Gururangan et al., 2018), and (2) competency score (Gardner et al., 2021).”
Informativeness and Invariance: Two Perspectives on Spurious Correlations in Natural Language (2022, background): “Gardner et al. (2021) propose a deﬁnition in terms of conditional probabilities: a feature X i is spuriously correlated with the label Y unless P ( Y | X i ) is uniform.”
The debate over understanding in AI’s large language models (2022, background): “Thus benchmarks or assessments that would be appropriate for measuring human understanding might not be appropriate for assessing such machines [15, 24, 50].”
Changing the World by Changing the Data (2021, background): “The larger the resource, the more difficult it is to avoid them (Gardner et al., 2021).”