Tensor2Tensor for Neural Machine Translation

By Ashish Vaswani, Samy Bengio, Eugene Brevdo, François Chollet, Aidan N. Gomez, Stephan Gouws, Llion Jones, Lukasz Kaiser, Nal Kalchbrenner, Niki Parmar and 3 more. 2018

553 citations recorded by the source corpus.

Tensor2Tensor is a library for deep learning models that is well-suited for neural machine translation and includes the reference implementation of the state-of-the-art Transformer model.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Train Large, Then Compress: Rethinking Model Size for Efficient Training and Inference of Transformers (2020, methodology): “We train using v3-8 TPUs and 8xV100 GPUs with Tensor2Tensor (Vaswani et al., 2018) and fairseq (Ott et al., 2019).”
Universal Transformers (2018, methodology): “The code used to train and evaluate Universal Transformers is available at https: //github.com/tensorflow/tensor2tensor (Vaswani et al., 2018).”
Music Transformer: Generating Music with Long-Term Structure (2018, methodology): “For the Transformer models (abbreviated as TF), we implemented our attention mechanisms in the Tensor2Tensor framework (Vaswani et al., 2018).”
KERMIT: Generative Insertion-Based Modeling for Sequences (2019, methodology): “We follow the hyperparameter settings of the base Transformer (Vaswani et al., 2018).”
fairseq: A Fast, Extensible Toolkit for Sequence Modeling (2019, background): “Tensor2tensor (Vaswani et al., 2018) and Sockeye (Hieber et al., 2018) focus on production-readiness.”
Scaling Up Models and Data with t5x and seqio (2022, methodology): “Previous Google-released systems based on TensorFlow include Tensor2Tensor (Vaswani et al., 2018), Lingvo (Shen et al.”
Towards a Human-like Open-Domain Chatbot (2020, methodology): “number of steps. We use the Tensor2Tensor code11An Evolved Transformer block is about twice as deep as a Transformer layer 12In the sense that validation loss increases as train loss decreases. base (Vaswani et al., 2018) for training Meena.13 A TPU-v3 core has 16GB of high-bandwidth memory. We maxi”
The NiuTrans Machine Translation Systems for WMT19 (2019, methodology): “While in recent implementations (Vaswani et al., 2018), layer normalization was applied to the input of every sublayer(see Figure 1(b)), which can provide a direct way to pass error gradient from top to bottom.”
HuggingFace's Transformers: State-of-the-art Natural Language Processing (2019, neutral): “Typical examples are the tensor2tensor20 library (Vaswani et al., 2018), fairseq21 (Ott et al.”
Learning Deep Transformer Models for Machine Translation (2019, methodology): “In recent implementations (Klein et al., 2017; Vaswani et al., 2018; Domhan, 2018), layer normalization is applied to the input of every sub-layer (see Figure 1(b)): Eq.”
Image Transformer (2018, methodology): “All code we used to develop, train, and evaluate our models is available in Tensor2Tensor (Vaswani et al., 2018).”
Generating High Fidelity Images with Subscale Pixel Networks and Multidimensional Upscaling (2018, background): “The “attention” parameters listed are configurable hyperparameters of the open source Transformer implementation in Vaswani et al. (2018).”
HuggingFace's Transformers: State-of-the-art Natural Language Processing (2019, positive): “ers and Natural Language Processing. More precisely, organizing the modules around three main components (conﬁguration, tokenizers and models) was inspired by the design of the tensor2tensor library (Vaswani et al., 2018) and the original code repository of Bert (Devlin et al., 2018) from Google Res”
NeMo: a toolkit for building AI applications using Neural Modules (2019, background): “…classiﬁed into two main groups: (1) higher-level neural network APIs such as Keras[11], Sonnet [18], PyTorch Ignite [3], PyTorch Lightning [4] and (2) conﬁguration-driven tookits such as Tensor2Tensor [20], Ludwig [5], OpenSeq2Seq [13], FairSeq [16], OpenNMT [12], Seq2Seq [9] and many others.”
Primer: Searching for Efficient Transformers for Language Modeling (2021, methodology): “In Tensor2Tensor (T2T) these are the TRANSFORMER _ TPU hyperparameters and in T5 and Lingvo these are the open-sourced parameters used in previous T5 studies [5, 49].”
Controlling Computation versus Quality for Neural Sequence Models (2020, background): “1Please refer to the ‘nda’ layout as implemented in the Tensor2Tensor library (Vaswani et al., 2018).”
Blockwise Parallel Decoding for Deep Autoregressive Models (2018, methodology): “Our code is publicly available in the open-source Tensor2Tensor library (Vaswani et al., 2018).”