fairseq: A Fast, Extensible Toolkit for Sequence Modeling

By Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, Michael Auli. 2019

3,399 citations recorded by the source corpus.

fairseq is an open-source sequence modeling toolkit that allows researchers and developers to train custom models for translation, summarization, language modeling, and other text generation tasks. The toolkit is based on PyTorch and supports distributed training across multiple GPUs and machines. We also support fast mixed-precision training and inference on modern GPUs. A demo video can be found at https://www.youtube.com/watch?v=OtgDdWtHvto

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

RoBERTa: A Robustly Optimized BERT Pretraining Approach (2019, methodology): “We reimplement BERT in FAIRSEQ (Ott et al., 2019).”
Train Large, Then Compress: Rethinking Model Size for Efficient Training and Inference of Transformers (2020, methodology): “We train using v3-8 TPUs and 8xV100 GPUs with Tensor2Tensor (Vaswani et al., 2018) and fairseq (Ott et al., 2019).”
Multilingual Denoising Pre-training for Neural Machine Translation (2020, neutral): “All experiments are done with Fairseq (Ott et al., 2019).”
SpanBERT: Improving Pre-training by Representing and Predicting Spans (2019, methodology): “We reimplemented BERT’s model and pre-training method in fairseq (Ott et al., 2019).”
Cloze-driven Pretraining of Self-attention Networks (2019, methodology): “We adapt the transformer implementation available in the fairseq toolkit to our two tower architecture (Ott et al., 2019).”
Efficient Large Scale Language Modeling with Mixtures of Experts (2021, methodology): “We use the Adam optimizer (Kingma and Ba, 2015) with β1 = 0.9, β2 = 0.98, = 10−8, weight decay of 0.01 and dropout of 0.1.4 We train our models in PyTorch (Paszke et al., 2017) using FAIRSEQ (Ott et al., 2019).”
Scaling Up Models and Data with t5x and seqio (2022, methodology): “Comparable projects from other research groups include model libraries such as fairseq (Ott et al., 2019), large-scale parallelism libraries such as FairScale (Baines et al., 2021), and libraries that include both kinds of functionality such as DeepSpeed (Rasley et al., 2020) and Megatron (Smith et…”
Reasoning Like Program Executors (2022, methodology): “We implement our models based on transformers (Wolf et al., 2020), fairseq (Ott et al., 2019) and DeepSpeed 4 .”
RoFormer: Enhanced Transformer with Rotary Position Embedding (2021, methodology): “We implement the experiment in PyTorch in the fairseq toolkit (MIT License)Ott et al. [2019].”
WikiLingua: A New Benchmark Dataset for Multilingual Abstractive Summarization (2020, methodology): “We use Fairseq (Ott et al., 2019) for all our experiments, and we follow the hyper-parameter settings that were used by Lewis et al. (2019b) to ﬁne-tune BART for mono-lingual summarization in English.”
InCoder: A Generative Model for Code Infilling and Synthesis (2022, methodology): “We use the same architecture as the dense 6.7B models described in [8]; the Fairseq architecture description can be found in Table 8 in the appendix.”
Recipes for Safety in Open-domain Chatbots (2020, methodology): “The model is referred to in the rest of the paper as BST 2.7B. Training Data The models are trained using maximum likelihood on human-human conversations in English, using the Fairseq (Ott et al., 2019) toolkit.”
Simple and Effective Retrieve-Edit-Rerank Text Generation (2020, methodology): “Our model is built using the Fairseq library (Ott et al., 2019).”
HuggingFace's Transformers: State-of-the-art Natural Language Processing (2019, neutral): “, 2018), fairseq21 (Ott et al., 2019) and Megatron-LM22.”
AmbigQA: Answering Ambiguous Open-domain Questions (2020, methodology): “All models are implemented in PyTorch (Paszke et al., 2017), PyTorch-Transformers (Wolf et al., 2019) (for BERT) and fairseq (Ott et al., 2019) (for BART).”
Autoregressive Entity Retrieval (2020, methodology): “We implemented, trained, and evaluate our model using the fariseq library (Ott et al., 2019).”
End-to-end ASR: from Supervised to Semi-Supervised Learning with Modern Architectures (2019, positive): “n-gram LMs are trained with the KenLM toolkit (Heafield, 2011), while the GCNN and Transformer LMs are trained with fairseq4 toolkit (Ott et al., 2019).”