Why Neural Translations are the Right Length

By Xing Shi, Kevin Knight, Deniz Yuret. 2016

69 citations recorded by the source corpus.

We investigate how neural, encoder-decoder translation systems output target strings of appropriate lengths, ﬁnding that a collection of hidden units learns to explicitly implement this functionality.

Subfields: Artificial Intelligence, Molecular Biology, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Molecular Biology

In Natural Language Processing

Citations in the corpus

Don’t Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization (2018, background): “Position embeddings have proved useful for convolutional sequence modeling (Gehring et al., 2017b), because, in contrast to RNNs, they do not observe the temporal positions of words (Shi et al., 2016).”
Analysis Methods in Neural Language Processing: A Survey (2018, methodology): “The attention mechanism that originated in work on NMT (Bahdanau et al., 2014) also lends itself to a natural visualization.”
Linguistic Knowledge and Transferability of Contextual Representations (2019, methodology): “…learned representations in neural models, such as directly examining the activations of individual neurons (Karpathy et al., 2015; Li et al., 2015; Shi et al., 2016a, inter alia ), ablating components of the model and dataset (Kuncoro et al., 2017; Gaddy et al., 2018; Khandelwal et al., 2018), or…”
Understanding Neural Networks through Representation Erasure (2016, background): “Shi et al. (2016) study the role of vector dimensions (for example to track sequence length) in sequence generation tasks;.”
Investigating Catastrophic Forgetting During Continual Training for Neural Machine Translation (2020, background): “Shi et al. (2016) investigates how NMT models output target strings of appropriate lengths.”
Identifying and Controlling Important Neurons in Neural Machine Translation (2018, background): “Previous work found speciﬁc neurons in NMT that capture position of words in the sentence (Shi et al., 2016a).”
What Is One Grain of Sand in the Desert? Analyzing Individual Neurons in Deep NLP Models (2018, background): “Some recent work on neural language models and machine translation analyzes speciﬁc neurons of length (Qian, Qiu, and Huang 2016a; Shi, Knight, and Yuret 2016) and sentiment (Radford, Jozefowicz, and Sutskever 2017).”
LSTMs Exploit Linguistic Attributes of Data (2018, methodology): “…that the LSTMs in an encoder-decoder machine translation model learn to increment and decrement a counter to generate translations of proper length (Shi et al., 2016a) and that representations produced by auto-encoding LSTMs contain information about the input sequence length (Adi et al., 2017).”
Interrogating the Explanatory Power of Attention in Neural Machine Translation (2019, methodology): “While these methods focus on explaining a model’s decision, Shi et al. (2016); Kádár et al. (2017); Calvillo and Crocker (2018) investigate how a particular concept is represented in the network.”
On the Relation between Position Information and Sentence Length in Neural Machine Translation (2019, background): “Shi et al. (2016) examine why RNN-based model generates translations of the right length without special mechanism for the length, and report how LSTM regulates the output length.”
Analyzing Redundancy in Pretrained Transformer Models (2020, methodology): “The former is commonly done using a probing classiﬁer (Shi et al., 2016a; Belinkov et al., 2017; Hupkes et al., 2018).”
The EOS Decision and Length Extrapolation (2020, background): “den states of LSTM sequence-to-sequence models trained to perform machine translation models track sequence length by implementing something akin to a counter that increments during encoding and decrements during decoding (Shi et al., 2016).”
Similarity Analysis of Contextual Word Representation Models (2020, background): “While neural language representations are thought to be densely distributed, several recent studies have pointed out the importance of individual neurons (Qian et al., 2016; Shi et al., 2016; Radford et al., 2017; Lakretz et al., 2019; Bau et al., 2019; Dalvi et al., 2019; Baan et al., 2019).”