Lingvo: a Modular and Scalable Framework for Sequence-to-Sequence Modeling

By Jonathan Shen, Patrick Nguyen, Yonghui Wu, Z. Chen, M. Chen, Ye Jia, Anjuli Kannan, Tara N. Sainath, Yuan Cao, Chung-Cheng Chiu and 82 more. 2019

215 citations recorded by the source corpus.

Lingvo is a Tensorflow framework offering a complete solution for collaborative deep learning research, with a particular focus towards sequence-to-sequence models. Lingvo models are composed of modular building blocks that are flexible and easily extensible, and experiment configurations are centralized and highly customizable. Distributed training and quantized inference are supported directly within the framework, and it contains existing implementations of a large number of utilities, helper functions, and the newest research ideas. Lingvo has been used in collaboration by dozens of researchers in more than 20 papers over the last two years. This document outlines the underlying design of Lingvo and serves as an introduction to the various pieces of the framework, while also offering examples of advanced features that showcase the capabilities of the framework.

Subfields: Artificial Intelligence, Molecular Biology, Natural Language Processing, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Molecular Biology

In Natural Language Processing

In Model Architectures

Citations in the corpus

Massively Multilingual Neural Machine Translation (2019, positive): “All models are implemented in Tensorﬂow-Lingvo (Shen et al., 2019).”
Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges (2019, methodology): “We perform all our experiments with variants of the Trans-former architecture (Vaswani et al., 2017), using the open-source Lingvo framework (Shen et al., 2019).”
LaMDA: Language Models for Dialog Applications (2022, methodology): “We used the Lingvo framework [94] for training and achieved 123 TFLOPS/sec with 56.”
GLaM: Efficient Scaling of Language Models with Mixture-of-Experts (2021, background): “• We skip weight updates for a batch if there are any NaNs or Inf s in the gradients (Shen et al., 2019).”
Scaling Up Models and Data with t5x and seqio (2022, methodology): “Previous Google-released systems based on TensorFlow include Tensor2Tensor (Vaswani et al., 2018), Lingvo (Shen et al., 2019), and the Mesh TensorFlow (Shazeer et al., 2018)-based T5 (Raﬀel et al., 2020).”
GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding (2020, methodology): “We use the following APIs in TensorFlow/Lingvo [31] in our work.”
Evaluating the Cross-Lingual Effectiveness of Massively Multilingual Neural Machine Translation (2019, methodology): “We train our Massively Multilingual NMT system using the Transformer architecture (Vaswani et al., 2017) in the open-source implementation under the Lingvo framework (Shen et al., 2019).”
Tagged Back-Translation (2019, methodology): “We use the transformer-base and transformer-big architectures (Vaswani et al., 2017) implemented in lingvo (Shen et al., 2019).”
Fairseq S2T: Fast Speech-to-Text Modeling with Fairseq (2020, methodology): “, 2020) and Lingvo (Shen et al., 2019), FAIRSEQ S2T pursues the best integration, scalability and reproducibility.”
ContextNet: Improving Convolutional Neural Networks for Automatic Speech Recognition with Global Context (2020, methodology): “All models are implemented with Lingvo toolkit [32].”
Primer: Searching for Efficient Transformers for Language Modeling (2021, methodology): “We conduct our comparisons across three different codebases: Tensor2Tensor (T2T) [25], T5 [5], and Lingvo [50].”
Scaling Autoregressive Models for Content-Rich Text-to-Image Generation (2022, methodology): “We implement our models in Lingvo [39] and scale with GSPMD [28] on CloudTPUv4 hardware for both training and inference.”
Controlling Computation versus Quality for Neural Sequence Models (2020, methodology): “We would like to thank the Google Translate and Tensorflow Lingvo (Shen et al., 2019a) teams for foundational contributions to the project.”
Direct speech-to-speech translation with a sequence-to-sequence model (2019, methodology): “Models were implemented using the Lingvo framework [39].”
Gmail Smart Compose: Real-Time Assisted Writing (2019, methodology): “We followed the implementations of the two models released in Tensorflow-Lingvo[44].”
Model Unit Exploration for Sequence-to-Sequence Speech Recognition (2019, methodology): “We use open-source Tensorflow Lingvo toolkit [29] for all experiments.”