Scaling Up Models and Data with t5x and seqio

By Adam Roberts, Hyung Won Chung, Anselm Levskaya, Gaurav Mishra, James Bradbury, D. Andor, Sharan Narang, Brian Lester, Colin Gaffney, Afroz Mohiuddin and 33 more. 2022

224 citations recorded by the source corpus.

Recent neural network-based language models have benefited greatly from scaling up the size of training datasets and the number of parameters in the models themselves. Scaling can be complicated due to various factors including the need to distribute computation on supercomputer clusters (e.g., TPUs), prevent bottlenecks when infeeding data, and ensure reproducible results. In this work, we present two software libraries that ease these issues: $\texttt{t5x}$ simplifies the process of building and training large language models at scale while maintaining ease of use, and $\texttt{seqio}$ provides a task-based API for simple creation of fast and reproducible training data and evaluation pipelines. These open-source libraries have been used to train models with hundreds of billions of parameters on datasets with multiple terabytes of training data. Along with the libraries, we release configurations and instructions for T5-like encoder-decoder models as well as GPT-like decoder-only architectures. $\texttt{t5x}$ and $\texttt{seqio}$ are open source and available at https://github.com/google-research/t5x and https://github.com/google/seqio, respectively.

Subfields: Natural Language Processing, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Model Architectures

Citations in the corpus

PaLM: Scaling Language Modeling with Pathways (2022, methodology): “Software: T5X (t5x, 2021).”
TALM: Tool Augmented Language Models (2022, methodology): “We use pretrained T5 models [Raffel et al., 2019, Roberts et al., 2022] for ﬁnetuning, inference and evaluation.”
Fast Inference from Transformers via Speculative Decoding (2022, methodology): “We implement our method and compare actual walltimes for T5-XXL to those of the robust T5X implementation (Roberts et al., 2022), showing an out-of-the-box latency improvement of 2X-3X , without any change to the outputs (Section 4).”
AudioLM: A Language Modeling Approach to Audio Generation (2022, methodology): “We achieve this objective by combining recent advances in adversarial neural audio compression [16], self-supervised representation learning [17] and language modeling [18].”
How to DP-fy ML: A Practical Guide to Machine Learning with Differential Privacy (2023, background): “Another peculiarity of LLMs is that some implementations like Roberts et al. (2022) by default do not normalize the loss to account for the length of the sequence.”
SLiC-HF: Sequence Likelihood Calibration with Human Feedback (2023, methodology): “We conduct all experiments using T5 models [15] in the T5x framework [16].”
Phenaki: Variable Length Video Generation From Open Domain Textual Description (2022, methodology): “To train MaskGIT, we include a text conditioning in the form of T5X embeddings [37] which are used as input through the use of cross attention with the video tokens.”
TokenSplit: Using Discrete Speech Representations for Direct, Refined, and Transcript-Conditioned Speech Separation and Recognition (2023, methodology): “We use T5X [28] to train our proposed models.”
FiDO: Fusion-in-Decoder optimized for stronger performance and faster inference (2022, methodology): “1.1 architecture (Raffel et al., 2020), pre-trained from scratch on C4 (Dodge et al., 2021) using JAX (Brad-bury et al., 2018), FLAX (Heek et al., 2020), and T5X (Roberts et al., 2022).”