Rissanen Data Analysis: Examining Dataset Characteristics via Description Length
By Ethan Perez, Douwe Kiela, Kyunghyun Cho. 2021
26 citations recorded by the source corpus.
We introduce a method to determine if a certain capability helps to achieve an accurate model of given data. We view labels as being generated from the inputs by a program composed of subroutines with different capabilities, and we posit that a subroutine is useful if and only if the minimal program that invokes it is shorter than the one that does not. Since minimum program length is uncomputable, we instead estimate the labels' minimum description length (MDL) as a proxy, giving us a theoretically-grounded method for analyzing dataset characteristics. We call the method Rissanen Data Analysis (RDA) after the father of MDL, and we showcase its applicability on a wide variety of settings in NLP, ranging from evaluating the utility of generating subquestions before answering a question, to analyzing the value of rationales and explanations, to investigating the importance of different parts of speech, and uncovering dataset gender bias.
Subfields: Evaluation & Benchmarks
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Evaluation & Benchmarks
- Douwe Kiela
- Kyunghyun Cho
- Robin Jia
- Ethan Perez
- Koustuv Sinha
- Adina Williams
- Joelle Pineau
- Mohit Bansal
Citations in the corpus
- Understanding Dataset Difficulty with V-Usable Information (2021, neutral): “Rissanen data analysis (Perez et al., 2021) offers a complimentary method for interpretability w.r.t attributes; it involves calculating the minimum description length (MDL): how many bits are needed to transmit the gold labels from a sender to a recipient when both have access to the same model…”
- True Few-Shot Learning with Language Models (2021, methodology): “Recent work has used MDL to determine which learning algorithms are most effective at explaining the given data [Rissanen Data Analysis; 10, 34].”
- What Context Features Can Transformer Language Models Use? (2021, background): “…tasks: Pham et al. (2020) find that shuffling experiments have a limited effect on the accuracy of models for natural language inference, while Perez et al. (2021) describe several experiments aimed at introducing usable information for several question answering and sentence understanding…”
- Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Little (2021, methodology): “Figure 6: Rissanen Data Analysis (Perez et al., 2021) on the GLUE benchmark and PAWS datasets.”