RETURNN as a Generic Flexible Neural Toolkit with Application to Translation and Speech Recognition

By Albert Zeyer, Tamer Alkhouli, Hermann Ney. 2018

96 citations recorded by the source corpus.

We compare the fast training and decoding speed of RETURNN of attention models for translation, due to fast CUDA LSTM kernels, and a fast pure TensorFlow beam search decoder. We show that a layer-wise pretraining scheme for recurrent attention models gives over 1% BLEU improvement absolute and it allows to train deeper recurrent encoder networks. Promising preliminary results on max. expected BLEU training are presented. We are able to train state-of-the-art models for translation and end-to-end models for speech recognition and show results on WMT 2017 and Switchboard. The flexibility of RETURNN allows a fast research feedback loop to experiment with alternative architectures, and its generality allows to use it on a wide range of applications.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

RWTH ASR Systems for LibriSpeech: Hybrid vs Attention - w/o Data Augmentation (2019, methodology): “The hybrid model was trained and decoded with RASR [38] and RETURNN [23, 39].”
Language Modeling with Deep Transformers (2019, methodology): “All our implementations are based on the Tensorflow [31] based open-source toolkit RETURNN [32](3).”
End-to-End Speech Recognition: A Survey (2023, unknown): “[116], as well as attention-based ASR models [117], which can be combined with intermediate subsampling schemes [118], and model growing [119].”
A comprehensive analysis on attention models (2019, methodology): “It has been observed that with enough training data, attention models can perform as good as hybrid models, or at least comparable [Chiu et al., 2018, Zeyer et al., 2018b].”
ESPnet-ST: All-in-One Speech Translation Toolkit (2020, unknown): “, 2019) (Zeyer et al., 2018) (Zenkel et al.”
SpeechBrain: A General-Purpose Speech Toolkit (2021, background): “, DeepSpeech [8], RETURNN [9], PyTorch-Kaldi [10], Espresso [11], Lingvo [12], Fairseq [13], ESPnet [14], and NeMo [15].”