Online Segment to Segment Neural Transduction

By Lei Yu, Jan Buys, Phil Blunsom. 2016

82 citations recorded by the source corpus.

We introduce an online neural sequence to sequence model that learns to alternate between encoding and decoding segments of the input as it is read. By independently tracking the encoding and decoding representations our algorithm permits exact polynomial marginalization of the latent segmentation during training, and during decoding beam search is employed to find the best alignment path together with the predicted output sequence. Our model tackles the bottleneck of vanilla encoder-decoders that have to read and memorize the entire input sequence in their fixed-length hidden states before producing any output. It is different from previous attentive models in that, instead of treating the attention weights as output of a deterministic function, our model assigns attention weights to a sequential latent variable which can be marginalized out and permits online generation. Experiments on abstractive sentence summarization and morphological inflection show significant performance gains over the baseline encoder-decoders.

Subfields: Artificial Intelligence, Model Architectures, Machine Translation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Model Architectures

In Machine Translation

Citations in the corpus

Structured Attention Networks (2017, neutral): “latent alignments) for sequence-tosequence transduction (Kong et al., 2016; Lu et al., 2016; Yu et al., 2016).”
Multimodal Transformer for Unaligned Multimodal Language Sequences (2019, neutral): “…0.662 RAVEN (Wang et al., 2019) 33.2 78.0 76.6 0.915 0.691 MCTN (Pham et al., 2019) 35 (Zadeh et al., 2018b) 45.0 76.9 77.0 0.71 0.54 RAVEN (Wang et al., 2019) 50.0 79.1 79.5 0.614 0.662 MCTN (Pham et al., 2019) 49 agonal) crossmodal attention matrix (i.e., mono-tonic attention (Yu et al., 2016)).”
Online and Linear-Time Attention by Enforcing Monotonic Alignments (2017, methodology): “More closely to our model, (Yu et al., 2016b) similarly add “shift” and “emit” operations to an RNN. Finally, the Segmental RNN (Kong et al., 2015) treats a segmentation of the input sequence as a latent random variable.”
The Neural Noisy Channel (2016, background): “Pointing the unknown words (G¨ulc¸ehre et al., 2016) uses pointer networks (Vinyals et al., 2015) to select the output token from the input sequence in order to avoid generating unknown tokens.”
Monotonic Chunkwise Attention (2017, background): “A related class of non-attentive sequence transduction models which can be used in online settings are connectionist temporal classiﬁcation (Graves et al., 2006), the RNN transducer (Graves, 2012), segment-to-segment neural transduction (Yu et al., 2016), and the segmental RNN (Kong et al., 2015).”
Transforming Question Answering Datasets Into Natural Language Inference Datasets (2018, background): “…2017; Vaswani et al., 2017), we believe it should be possible to further increase the data efﬁciency and performance, especially by leveraging models that incorporate syntactic structure (Chen et al., 2017; Eriguchi et al., 2017) or a more transducer-like structure (Graves, 2012; Yu et al., 2016).”
Latent Alignment and Variational Attention (2018, unknown): “monotonicity) on the alignment distribution [83, 82, 58].”
Better Document-Level Machine Translation with Bayes’ Rule (2019, methodology): “Rather than using a specially designed channel model (Yu et al., 2016) which is limited in processing long sequences like documents, our conditional sentence independence assumptions allow us to use any sequence-to-sequence model as the channel model, making it a better option for document-level tra”
Neural Machine Translation: A Review (2019, background): “Similar ideas have been explored for speech recognition [385], morphological inflection [386], text summarization [387, 388], and image caption generation [82].”
Sequence-to-Sequence Learning with Latent Neural Grammars (2021, unknown): “Since we marginalize over unobserved trees during learning, our work is also related to the line of work on marginalizing out latent variables/structures for sequence transduction tasks [46, 33, 11, 66, 128, 90, 57, 69, 108, 111, inter alia ].”
Neural Data-to-Text Generation via Jointly Learning the Segmentation and Correspondence (2020, methodology): “There have been quite a few neural alignment models applied to tasks like machine translation (Wang et al., 2018; Deng et al., 2018), character transduction (Wu et al., 2018; Shankar and Sarawagi, 2019) and summarization (Yu et al., 2016; Shen et al., 2019b).”
Review of end-to-end speech synthesis technology based on deep learning (2021, background): “The constrained alignment is conceptually borrowed from segment-to-segment neural transduction (SSNT) [244, 245].”
Benchmarking Hierarchical Script Knowledge (2019, methodology): “Therefore, in addition to using the global attention model of (Luong et al., 2015), we adapt the transducer model proposed by Yu et al. (2016), which uses learned latent discrete variables to model phrase-to-phrase alignments.”
Initial investigation of an encoder-decoder end-to-end TTS framework using marginalization of monotonic hard latent alignments (2019, methodology): “…the gradient can be computed with the following form: Then the back-propagation can be combined via back-ward operation of the forward-backward algorithms by introducing a backward variable β ( i, j ) and utilizing the relation ∂p ( y 1: J | x 1: I ; θ ) ∂α ( i, j ) = β ( i, j ) [12, 21].”
Learning Semantic Parsers from Denotations with Latent Structured Alignments and Abstract Programs (2019, methodology): “Besides semantic parsing, treating alignments as discrete latent variables has proved effective in other tasks like sequence transduction (Yu et al., 2016) and AMR parsing (Lyu and Titov, 2018).”
Learning to Translate in Real-time with Neural Machine Translation (2016, background): “In addition, Yu et al. (2016) recently proposed an online alignment model to help sentence compression and morphological in-ﬂection.”
Multi-Task Learning of Keyphrase Boundary Classification (2017, unknown): “A further instance of multi-task learning is to optimise a supervised training objective jointly with an unsupervised training objective, as shown in Yu et al. (2016) for natural language generation and auto-encoding, and in Rei (2017) for different sequence labelling tasks and language modelling.”
Selective Encoding for Abstractive Sentence Summarization (2017, background): “Yu et al. (2016) propose a segment to segment neural transduction model for sequence-to-sequence framework.”