Optimal Completion Distillation for Sequence Learning

By Sara Sabour, William Chan, Mohammad Norouzi. 2018

45 citations recorded by the source corpus.

We present Optimal Completion Distillation (OCD), a training procedure for optimizing sequence to sequence models based on edit distance. OCD is efficient, has no hyper-parameters of its own, and does not require pretraining or joint optimization with conditional log-likelihood. Given a partial sequence generated by the model, we first identify the set of optimal suffixes that minimize the total edit distance, using an efficient dynamic programming algorithm. Then, for each position of the generated sequence, we use a target distribution that puts equal probability on the first token of all the optimal suffixes. OCD achieves the state-of-the-art performance on end-to-end speech recognition, on both Wall Street Journal and Librispeech datasets, achieving $9.3\%$ WER and $4.5\%$ WER respectively.

Subfields: Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Efficiency & Compression

Citations in the corpus

Imputer: Sequence Modelling via Imputation and Dynamic Programming (2020, methodology): “As we decrease the block size further, WER increases because of more conditionally independent (parallel) generations.”
RWTH ASR Systems for LibriSpeech: Hybrid vs Attention - w/o Data Augmentation (2019, methodology): “In [7] a training procedure based on edit distance for sequence to sequence model optimization is presented.”
WaveGrad: Estimating Gradients for Waveform Generation (2020, background): “…2019; Chan et al., 2020; Saharia et al., 2020), insertion-based generative models (Stern et al., 2019; Chan et al., 2019b;a;c; Li & Chan, 2019) and edit-based generative models (Sabour et al., 2019; Gu et al., 2019; Ruis et al., 2019) found in the semi-autoregressive sequence generation literature.”
SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition (2019, neutral): “, (2019) [39] 4.”
Transformers with convolutional context for ASR (2019, neutral): “4 LSTM (OCD)[27] None - - 4.”
Model Unit Exploration for Sequence-to-Sequence Speech Recognition (2019, methodology): “’s model trained with optimal completion distillation [33], which is reported to give 4.”
Levenshtein Transformer (2019, methodology): “Additionally, the use of Levenshtein distance with dynamic programming as the oracle policy were also proposed in Sabour et al. (2018); Dong et al. (2019).”
End-to-End Speech Recognition: A Survey (2023, methodology): “Optimal Completion Distillation (OCD) [81] proposes to minimize the total edit distance using an efﬁcient dynamic programming algorithm.”
Self-attention Networks for Connectionist Temporal Classification in Speech Recognition (2019, background): “8%, outdoing all previous end-to-end results except OCD training [41].”
Insertion-Deletion Transformer (2020, unknown): “This approach was also explored by Dong et al. (2019); Sabour et al. (2019).”
Advancing Sequence-to-Sequence Based Speech Recognition (2019, background): “We note that this system already outperforms the lowest stand-alone sequenceto-sequence numbers published on this task [38].”
Adaptive Knowledge Distillation Based on Entropy (2020, result): “To verify the performance of pre-trained models, we compared the results with the recent papers [17] [18] [19].”