Mixed-Precision Training for NLP and Speech Recognition with OpenSeq2Seq

By Oleksii Kuchaiev, Boris Ginsburg, Igor Gitman, Vitaly Lavrukhin, Jason Li, Huyen Nguyen, Carl Case, Paulius Micikevicius. 2018

52 citations recorded by the source corpus.

We present OpenSeq2Seq - a TensorFlow-based toolkit for training sequence-to-sequence models that features distributed and mixed-precision training. Benchmarks on machine translation and speech recognition tasks show that models built using OpenSeq2Seq give state-of-the-art performance at 1.5-3x less training time. OpenSeq2Seq currently provides building blocks for models that solve a wide range of tasks including neural machine translation, automatic speech recognition, and speech synthesis.

Subfields: Artificial Intelligence, Natural Language Processing, Efficiency & Compression, Speech & Audio

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Lior Wolf
2. Chris Dyer
3. Graham Neubig
4. Mohit Bansal
5. Yoshua Bengio
6. Yoav Goldberg
7. Noah A. Smith
8. Quoc V. Le

In Natural Language Processing

1. Chris Dyer
2. Lior Wolf
3. Graham Neubig
4. Phil Blunsom
5. Kazuya Kawakami
6. Aäron van den Oord
7. Yoav Goldberg
8. Noah A. Smith

In Efficiency & Compression

1. Lior Wolf
2. Phil Blunsom
3. Dipankar Das
4. Adam Polyak
5. Aäron van den Oord
6. Bharat Kaul
7. Carlo Luschi
8. Naveen Mellempudi

In Speech & Audio

1. Lior Wolf
2. Adam Polyak
3. Yaniv Taigman
4. Phil Blunsom
5. Chris Dyer
6. Kazuya Kawakami
7. Aäron van den Oord
8. Luyu Wang

Citations in the corpus

• Learning Robust and Multilingual Speech Representations (2020, positive): “To do so, we use the format of the Lib-riSpeech dataset, which also ensures that our results are comparable with standard speech recognition systems on that task (Kuchaiev et al., 2018).”
• Mixed Precision Training With 8-bit Floating Point (2019, methodology): “Other networks such as GNMT[26] and Transformer[23] use a more robust dynamic loss scaling method[19].”
• This paper is included in the Proceedings of the 2022 USENIX Annual Technical Conference. (unknown): “[32] presents a TensorFlow-based toolkit for mixed precision training of sequence-to-sequence models, with an implementation of a wrapper around the standard TensorFlow performance optimization facility for mixed precision training.”
• TTS Skins: Speaker Conversion via ASR (2019, methodology): “In our method, the encoder is taken to be a subnetwork of a pre-trained ASR network, specifically the public implementation [1] of Wav2Letter [2].”
• Unit Scaling: Out-of-the-Box Low-Precision Training (2023, background): “Automatic loss scaling The dynamic adjustment of the loss scale during training is termed automatic loss scaling (Kuchaiev et al., 2018).”
• Espnet-TTS: Unified, Reproducible, and Integratable Open Source End-to-End Text-to-Speech Toolkit (2019, unknown): “• r9y9: r9y9/deepvoice3_pytorch [21], • Kyubyong: Kyubyong/tacotron [22], • Rayhane: Rayhane-mamah/Tacotron-2 [23], • NVIDIA: NVIDIA/tacotron2 [24], • Mozilla: mozilla/TTS [25], • OpenSeq2Seq: NVIDIA/OpenSeq2Seq [26].”