Learning online alignments with continuous rewards policy gradient

By Yuping Luo, Chung-Cheng Chiu, Navdeep Jaitly, Ilya Sutskever. 2016

47 citations recorded by the source corpus.

Sequence-to-sequence models with soft attention had significant success in machine translation, speech recognition, and question answering. Though capable and easy to use, they require that the entirety of the input sequence is available at the beginning of inference, an assumption that is not valid for instantaneous translation and speech recognition. To address this problem, we present a new method for solving sequence-to-sequence problems using hard online alignments instead of soft offline alignments. The online alignments model is able to start producing outputs without the need to first process the entire input sequence. A highly accurate online sequence-to-sequence model is useful because it can be used to build an accurate voice-based instantaneous translator. Our model uses hard binary stochastic decisions to select the timesteps at which outputs will be produced. The model is trained to produce these stochastic decisions using a standard policy gradient method. In our experiments, we show that this model achieves encouraging performance on TIMIT and Wall Street Journal (WSJ) speech recognition datasets.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Reinforcement Learning

Citations in the corpus

Imputer: Sequence Modelling via Imputation and Dynamic Programming (2020, background): “Consequently, naively optimizing for Equation 8 directly is difﬁcult in practice (Luo et al., 2016).”
Towards Better Decoding and Language Model Integration in Sequence to Sequence Models (2016, methodology): “Smoothing techniques increase the entropy of a model’s predictions, a technique that was used to promote exploration in reinforcement learning [27, 28, 29].”
Regularizing Neural Networks by Penalizing Confident Output Distributions (2017, methodology): “Penalizing low entropy has also been used when combining reinforcement learning and supervised learning to train a neural speech recognition model to learn when to emit tokens (Luo et al., 2016).”
Online and Linear-Time Attention by Enforcing Monotonic Alignments (2017, methodology): “(Luo et al., 2016) and (Zaremba & Sutskever, 2015) both study a similar framework in which a decoder RNN can decide whether to ingest another entry from the input sequence or emit an entry of the output sequence.”
Monotonic Chunkwise Attention (2017, methodology): “Under review as a conference paper at ICLR 2018 Prior Result WER (Raffel et al., 2017) (CTC baseline) 33.4% (Luo et al., 2016) (Reinforcement Learning) 27.0% (Wang et al., 2016) (CTC) 22.7% (Raffel et al., 2017) Table 2: ROUGE F-scores for document summarization on the CNN/Daily Mail dataset.”
Incremental Learning with Maximum Entropy Regularization: Rethinking Forgetting and Intransigence (2019, methodology): “, 2015), reinforcement learning (Mnih et al., 2016; Luo et al., 2016; Norouzi et al., 2016) and in the supervised settting (Pereyra et al.”
Learning to Translate in Real-time with Neural Machine Translation (2016, methodology): “A similar idea for online ASR is proposed by Luo et al. (2016).”
Training with Confusion for Fine-Grained Visual Classification (2017, background): “Entropy-based regularization has also been shown to signiﬁcantly improve reinforcement learning methods [27, 30].”
Deep Reinforcement Learning: An Overview (2017, background): “There are endeavours for hard attention (Gulcehre et al., 2016; Liang et al., 2017a; Luo et al., 2016; Xu et al., 2015; Zaremba and Sutskever, 2015), which is the way conventional computers access memory.”