OpenSeq2Seq: Extensible Toolkit for Distributed and Mixed Precision Training of Sequence-to-Sequence Models

By Oleksii Kuchaiev, Boris Ginsburg, Igor Gitman, Vitaly Lavrukhin, Carl Case, Paulius Micikevicius. 2018

33 citations recorded by the source corpus.

We present OpenSeq2Seq – an open-source toolkit for training sequence-to-sequence models. The main goal of our toolkit is to allow researchers to most effectively explore different sequence-to-sequence architectures. The efficiency is achieved by fully supporting distributed and mixed-precision training. OpenSeq2Seq provides building blocks for training encoder-decoder models for neural machine translation and automatic speech recognition. We plan to extend it with other modalities in the future.

Subfields: Artificial Intelligence, Natural Language Processing, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Model Architectures

Citations in the corpus

fairseq: A Fast, Extensible Toolkit for Sequence Modeling (2019, methodology): “OpenSeq2Seq (Kuchaiev et al., 2018) provides reference implementations for fast distributed and mixed precision training.”
NeMo: a toolkit for building AI applications using Neural Modules (2019, background): “These toolkits could be loosely classiﬁed into two main groups: (1) higher-level neural network APIs such as Keras[11], Sonnet [18], PyTorch Ignite [3], PyTorch Lightning [4] and (2) conﬁguration-driven tookits such as Tensor2Tensor [20], Ludwig [5], OpenSeq2Seq [13], FairSeq [16], OpenNMT [12], Seq”
Wav2Letter++: A Fast Open-source Speech Recognition System (2018, methodology): “With the growing interest in automatic speech recognition (ASR), the open-source software ecosystem has seen a proliferation of ASR systems and toolkits, including Kaldi [1], ESPNet [2], OpenSeq2Seq [3] and Eesen[4].”
A Spectral Energy Distance for Parallel Speech Synthesis (2020, positive): “…be the sought representation that will be used for computing the (conditional) FDSD; and f kω : R kω (cid:55)→ R k 2 (cid:101)× 1600 be a function that takes (a part of) the waveform a and passes it through the pre-trained Deep Speech 2 (DS2) network [1, 21] to obtain the necessary activations.”
High Fidelity Speech Synthesis with Adversarial Networks (2019, positive): “Speciﬁcally, we use the implementation available in the NVIDIA OpenSeq2Seq library (Kuchaiev et al., 2018) and extract features from the last layer, whose output is used in the CTC loss during training.”
Jasper: An End-to-End Convolutional Neural Acoustic Model (2019, methodology): “All models have been trained on NVIDIA DGX-1 in mixed precision [31] using OpenSeq2Seq [32].”
fairseq Sˆ2: A Scalable and Integrable Speech Synthesis Toolkit (2021, background): “We see from Table 1 that FastSpeech 2 performs comparably well to Transformer with phoneme inputs (g2pE), both achieving 4.2 MOS.”
Neural Machine Translation: A Review (2019, unknown): “[26] TensorFlow Nematus Sennrich et al.”
Stochastic Gradient Methods with Layer-wise Adaptive Moments for Training of Deep Networks (2019, methodology): “We used OpenSeq2Seq [11] transformer-big which differs from the original10 implementation in two ways: (1) we measure batch size in sentence pairs, not tokens and (2) we use mixed precision training [17].”
Training Neural Speech Recognition Systems with Synthetic Speech Augmentation (2018, unknown): “We use the Tacotron-2 like model from the OpenSeq2Seq 3 toolkit (Kuchaiev et al., 2018) and add Global Style Tokens (GST) (Wang et al., 2018) to learn multiple speaker identities.”