HuggingFace's Transformers: State-of-the-art Natural Language Processing

By Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz and 12 more. 2019

Subfields: Large Language Models, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Large Language Models

In Natural Language Processing

Citations in the corpus

Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (2020, positive): “As shown in Figure 1, our models leverage two components: (i) a retriever pη(z|x) with parameters η that returns (top-K truncated) distributions over text passages given a query x and (ii) a generator pθ(yi|x, z, y1:i−1) parametrized (1)Code to run experiments with RAG has been open-sourced as part ”
High-Resolution Image Synthesis with Latent Diffusion Models (2021, neutral): “[99] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and Jamie Brew.”
Pretrained Transformers Improve Out-of-Distribution Robustness (2020, positive): “We perform our experiments using the HuggingFace Transformers library (Wolf et al., 2019).”
SpanBERT: Improving Pre-training by Representing and Predicting Spans (2019, positive): “Fine-tuning is implemented based on HuggingFace’s codebase (Wolf et al., 2019) and more details are given in Appendix B.”
RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models (2020, positive): “We use the Hugging Face Transformers (Wolf et al., 2019) versions of all pretrained models described in this section, implemented in the PyTorch (Paszke et al., 2019) deep learning framework.”
CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Models (2020, positive): “For these models we use the Transformers library (Wolf et al., 2019).”
Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Survey (2022, positive): “Wolf et al. (2019) apply this method to zero out the probabilities of offensive terms, and King et al. (2022b); Lu et al. (2021a) improve factuality by up-weighting generation probabilities of entities present in the source, using heuristics for transformation tasks.”
BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generation (2021, positive): “For language generation experiments, we use the HuggingFace library [42].”
Cutting Down on Prompts and Parameters: Simple Few-Shot Learning with Language Models (2021, positive): “Following past work (Schick and Schütze, 2021b), we use the RoBERTa (large, 330M params, Liu et al., 2019) and ALBERT (xxl-v2, 223M params, Lan et al., 2019) masked LMs provided by the Hug-gingFace transformers library (Wolf et al., 2020).”
It’s Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners (2020, positive): “Our implementation extends the original implementation of P ET by Schick and Sch¨utze (2020a) which, in turn, is based on the Transformers library (Wolf et al., 2019) and PyTorch (Paszke et al., 2017).”
Quantifying Attention Flow in Transformers (2020, positive): “We train a Transformer encoder, with GPT-2 Transformer blocks as described in (Radford et al., 2019; Wolf et al., 2019) (without masking).”
Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling (2023, positive): “We release all of our models and checkpoints to the public under the Apache 2.0 license via the HuggingFace Hub (Wolf et al., 2019) 2 We additionally release the code used for all evaluations and the raw benchmark scores generated on GitHub.”