HuggingFace's Transformers: State-of-the-art Natural Language Processing

By Thomas Wolf, Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf and 14 more. 2019

4,184 citations recorded by the source corpus.

Recent progress in natural language processing has been driven by advances in both model architecture and model pretraining. Transformer architectures have facilitated building higher-capacity models and pretraining has made it possible to effectively utilize this capacity for a wide variety of tasks. \textit{Transformers} is an open-source library with the goal of opening up these advances to the wider machine learning community. The library consists of carefully engineered state-of-the art Transformer architectures under a unified API. Backing this library is a curated collection of pretrained models made by and available for the community. \textit{Transformers} is designed to be extensible by researchers, simple for practitioners, and fast and robust in industrial deployments. The library is available at \url{this https URL}.

Subfields: Artificial Intelligence, Materials Chemistry, Large Language Models, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Materials Chemistry

In Large Language Models

In Model Architectures

Citations in the corpus

QLoRA: Efficient Finetuning of Quantized LLMs (2023, positive): “We open-source our codebase and CUDA kernels and integrate our methods into the Hugging Face transformers stack [65], making them easily accessible to all.”
Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning (2020, positive): “We use both the SAID and DID subspace optimization methods, which we implemented in the Huggingface Transformers library (Wolf et al., 2019).”
Exploring Versatile Generative Language Model Via Parameter-Efficient Transfer Learning (2020, positive): “We implement VLM based on GPT-2-small (124M) (Wolf et al., 2019a), and experiment with varying adapter bottleneck dimensions in { 10 , 50 , 100 , 300 } and pick the best one in each task to trade-off the performance with the parameter efﬁciency.”
LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale (2022, positive): “We also perform our analysis in two different inference software frameworks: Fairseq and Hugging Face Transformers (Wolf et al., 2019).”
ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers (2022, positive): “2 ), we follow the instruction from Huggingface Transformer Library [72].”
Training Neural Networks with Fixed Sparse Masks (2021, positive): “Additional hyper-parameters, such as choice of optimizer, sequence length, and others, follow from the default conﬁguration for BERT LARGE presented in the Hugging Face library [49].”
Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection (2023, positive): “We leveraged BERT-base [8] from Hugging Face [51] as text backbones.”
CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-X (2023, neutral): “We show in Section 4.2 that multilingual models can benefit from budget allocation strategies and have a higher solve rate than using any single language.”
Learning from the Worst: Dynamically Generated Datasets to Improve Online Hate Detection (2021, positive): “We use the implementation from the Transformers (Wolf et al., 2019) library.”
Mass-Editing Memory in a Transformer (2022, positive): “The language models are loaded using HuggingFace Transformers (Wolf et al., 2019), and PyTorch (Paszke et al., 2019) is used for executing the model editing algorithms on GPUs.”
Fast Model Editing at Scale (2021, neutral): “All models are based on Huggingface Transformers implementations (Wolf et al., 2019) with some modiﬁcations in line with Cao et al. (2021).”
Amnesic Probing: Behavioral Explanation with Amnesic Counterfactuals (2020, positive): “6Specifically, BERT-BASE-UNCASED (Wolf et al., 2019).”
CLUE: A Chinese Language Understanding Evaluation Benchmark (2020, neutral): “Different from existing pre-training model toolkits (Wolf et al., 2019; Zhao et al., 2019”
Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks (2020, positive): “Implementation Our LM implementation uses the HuggingFace transformers library (Wolf et al., 2019)10 and PyTorch XLA for TPU compatibility.11 Each adaptive pretraining experiment was performed on a single v3-8 TPU from 10https://github.com/huggingface/…”