Predicting Inductive Biases of Pre-Trained Models
By Charles Lovering, Rohan Jha, Tal Linzen, Ellie Pavlick. 2021
74 citations recorded by the source corpus.
Subfields: Representation & Self-Supervised Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Representation & Self-Supervised Learning
- Yonatan Belinkov
- Douwe Kiela
- Mohit Bansal
- Ellie Pavlick
- Tal Linzen
- Peter Hase
- Kyunghyun Cho
- Shashank Rajput
Citations in the corpus
- When Can Models Learn From Explanations? A Formal Framework for Understanding the Roles of Explanation Data (2021, methodology): “Following recent work using synthetic data to investigate sequence modeling questions (Liu et al., 2021; Lovering et al., 2021), we design a synthetic dataset so that we can carefully control several important data properties.”
- Probing Classifiers: Promises, Shortcomings, and Advances (2021, positive): “In contrast, Lovering et al. (2021) find that extractability of a property according to MDL ( g, f, D O , D P ) is correlated with f making predictions consistent with that property.”
- Do Prompt-Based Models Really Understand the Meaning of Their Prompts? (2021, background): “…do not ignore their prompts entirely, perhaps it “takes less effort” for models to use the spurious or heuristic features for predictions as opposed to the more complex syntactic or semantic features (Lovering et al., 2021; Warstadt et al., 2020) required to properly comply with the instructions.”
- Rissanen Data Analysis: Examining Dataset Characteristics via Description Length (2021, methodology): “, 2021) and estimating description length (Voita and Titov, 2020; Whitney et al., 2020; Lovering et al., 2021) or other information-”
- Frequency Effects on Syntactic Rule Learning in Transformers (2021, background): “The focus on frequency effects also relates to a more general line of work on understanding the effect of training size and distribution on neural language models’ generalization (Warstadt et al., 2020; Lovering et al., 2021).”
- LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learning Tasks (2022, background): “Similar efforts in this line of work are to analyze the behaviors, representations, and inductive bias of pretrained LMs [130, 131, 132] or investigate different aspects of LMs [133, 134, 135].”