Latent Sequence Decompositions

By William Chan, Yu Zhang, Quoc V. Le, Navdeep Jaitly. 2016

62 citations recorded by the source corpus.

We present the Latent Sequence Decompositions (LSD) framework. LSD decomposes sequences with variable lengthed output units as a function of both the input sequence and the output sequence. We present a training algorithm which samples valid extensions and an approximate decoding algorithm. We experiment with the Wall Street Journal speech recognition task. Our LSD model achieves 12.9% WER compared to a character baseline of 14.8% WER. When combined with a convolutional network on the encoder, we achieve 9.6% WER.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Representation & Self-Supervised Learning

In Signal Processing

Citations in the corpus

Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates (2018, background): “Latent Sequence Decompositions (LSDs) (Chan et al., 2016) learns the mapping from the input and the output by marginalizing over all possible segmentations.”
Imputer: Sequence Modelling via Imputation and Dynamic Programming (2020, neutral): “) and Word Error Rate (WER). Model CER WER Iterations seq2seq Bahdanau et al. (2016a) 6.4 18.6 n Bahdanau et al. (2016b) 5.9 18.0 n Chorowski & Jaitly (2017) - 10.6 n Zhang et al. (2017) - 10.5 n Chan et al. (2017) - 9.6 n Kim et al. (2017) 7.4 - n Serdyuk et al. (2018) 6.2 - n Tjandra et al. (2”
State-of-the-Art Speech Recognition with Sequence-to-Sequence Models (2017, methodology): “On the structure side, first, we explore word piece models (WPM) which have been applied to machine translation [7] and more recently to speech in RNN-T [8] and LAS [9].”
Regularizing Neural Networks by Penalizing Confident Output Distributions (2017, background): “We observe that the beneﬁts of label smoothing (WER reduction from 14.2 to 11) improve over the recently proposed Latent Sequence Decompositions (LSD) method (Chan et al., 2016) which reduces the WER from 14.7 to 12.9 by extending the space of output tokens to dynamically chosen character n-grams.”
Learning World Graphs to Accelerate Hierarchical Reinforcement Learning (2019, neutral): “Pivotal state discovery is related to unsupervised sequence segmentation [17, 47, 76, 13, 16] and option or sub-task discovery in the context of RL [48, 6, 72, 30, 56, 53, 52].”
Optimal Completion Distillation for Sequence Learning (2018, methodology): “In terms of WER, our work is also outperforming Chan et al. (2017), which uses subword units while our model emits characters.”
Bytes Are All You Need: End-to-end Multilingual Speech Recognition and Synthesis with Bytes (2018, methodology): “Sub-word units were used in seq2seq [12–14] and recurrent neural network transducer (RNNT) [15] models, and word units were used by [16, 17].”
A comprehensive analysis on attention models (2019, background): “7 attention [Chan et al., 2016] LSD none word piece 9.”
A Fully Differentiable Beam Search Decoder (2019, background): “Because our work is end-to-end, it is also related to seq2seq learning (Sutskever et al., 2014; Bahdanau et al., 2014; Chan et al., 2017; Wiseman and Rush, 2016; Gehring et al., 2017), and in particular training with existing/external language models (Sriram et al., 2018).”
Hybrid CTC/Attention Architecture for End-to-End Speech Recognition (2017, background): “Actually, recent sequence-to-sequence studies have handled this issue by using a subword unit (concatenating several letters to form a new subword unit) [13], [43], which would be a promising direction for our end-to-end ASR.”
Building Competitive Direct Acoustics-to-Word Models for English Conversational Speech Recognition (2017, unknown): “Popular E2E models include recurrent neural networks (RNNs) trained with the connectionist temporal classiﬁcation (CTC) loss function [3–10] and the attention-based encoder-decoder RNNs [11–15].”
Gram-CTC: Automatic Unit Selection and Target Decomposition for Sequence Labelling (2017, methodology): “Comparison experiments on multiple-scale datasets show the proposed Gram-CTC obtains state-of-the-art results on various ASR tasks.”
Advances in Joint CTC-Attention Based End-to-End Speech Recognition with a Deep CNN Encoder and RNN-LM (2017, methodology): “There are two major types of end-to-end architectures for ASR: attention-based methods use an attention mechanism to perform alignment between acoustic frames and recognized symbols [5, 6, 7, 8, 9], and Connectionist Temporal Classification (CTC), uses Markov assumptions to efficiently solve sequent”
Speech-Transformer: A No-Recurrence Sequence-to-Sequence Model for Speech Recognition (2018, methodology): “In recent years, seq2seq models have achieved significant WER reduction benefiting from building deeper encoder [5], the effective label smoothing schemes [4] and the latent sequence decomposition methods [6].”
Differentiable Weighted Finite-State Transducers (2020, background): “Learning n -gram word decompositions with both differentiable (Liu et al., 2017) and non-differentiable (Chan et al., 2017) loss functions has also been explored.”
Speech Recognition with Augmented Synthesized Speech (2019, background): “Targets are drawn from a vocabulary of 16k graphemic word pieces [17].”
Compositional Imitation Learning: Explaining and executing one task at a time (2018, background): “Unsupervised segmentation and encoding of sequence data is a similarly important problem in natural language or speech processing, e.g., in the context of word or phoneme segmentation [32, 33, 34], or in the segmentation of sequential activity data [35, 36].”