Does String-Based Neural MT Learn Source Syntax?

By Xing Shi, Inkit Padhi, Kevin Knight. 2016

370 citations recorded by the source corpus.

We investigate whether a neural, encoder-decoder translation system learns syntactic information on the source side as a by-product of training. We propose two methods to detect whether the encoder has learned local and global source syntax. A ﬁne-grained analysis of the syntactic structure learned by the encoder reveals which kinds of syntax are learned and which are missing.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Neural Network Acceptability Judgments (2018, background): “Examples of such tasks are to determine whether the sentence is in active or passive voice (Shi et al., 2016), whether the subject is singular or plural (Conneau et al.”
SentEval: An Evaluation Toolkit for Universal Sentence Representations (2018, background): “In particular, tasks that probe for speciﬁc linguistic properties of the sentence embeddings (Shi et al., 2016; Adi et al., 2017) are interesting directions towards understanding how the encoder understands language.”
Learning General Purpose Distributed Sentence Representations via Large Scale Multi-task Learning (2018, methodology): “The syntactic properties tasks are setup in the same way as described in Shi et al. (2016).”
How Can We Know What Language Models Know? (2019, background): “Much work has focused on understanding the internal representations in neural NLP models (Belinkov and Glass, 2019), either by using extrinsic probing tasks to examine whether certain linguistic properties can be predicted from those representations (Shi et al., 2016; Linzen et al., 2016; Belinkov e”
Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned (2019, background): “Different levels of linguistic analysis have been considered including morphology (Belinkov et al., 2017a; Dalvi et al., 2017; Bisazza and Tump, 2018), syntax (Shi et al., 2016) and semantics (Hill et al., 2017; Belinkov et al., 2017b; Raganato and Tiedemann, 2018).”
Probing Classifiers: Promises, Shortcomings, and Advances (2021, background): “Other early work classiﬁed hidden states of a recurrent neural network machine translation system into morpho-syntactic properties (Shi, Padhi, and Knight 2016).”
What you can cram into a single $&!#* vector: Probing sentence embeddings for linguistic properties (2018, methodology): “TopConst was introduced by Shi et al. (2016).”
What do Neural Machine Translation Models Learn about Morphology? (2017, methodology): “Our work is most similar to Shi et al. (2016), who use hidden vectors from a neural MT encoder to predict syntactic properties on the English source side.”
What Does BERT Look at? An Analysis of BERT’s Attention (2019, neutral): “Like our work, some of these studies have also demonstrated models capturing aspects of syntax (Shi et al., 2016; Blevins et al., 2018) or coreference (Tenney et al., 2018, 2019; Liu et al., 2019) without explicitly being trained for the tasks.”
What do you learn from context? Probing for sentence structure in contextualized word representations (2019, methodology): “Recent work has explored many token-level properties of these representations, such as their ability to capture part-of-speech tags (Blevins et al., 2018; Belinkov et al., 2017b; Shi et al., 2016), morphology (Belinkov et al., 2017a;b), or word-sense disambiguation (Peters et al., 2018a).”
Designing and Interpreting Probes with Control Tasks (2019, methodology): “…a sentence vector (Shi et al., 2016; Adi et al., 2017; Con-neau et al., 2018), word properties like verb tense or part-of-speech using word vectors (Shi et al., 2016; Belinkov et al., 2017; Liu et al., 2019), or word-pair properties like syntactic relationships using pairs of vectors (Tenney et…”
Colorless Green Recurrent Networks Dream Hierarchically (2018, background): “…typically by training classiﬁers to decode various linguistic properties from hidden states of the network (e.g., Adi et al., 2017; Be-linkov et al., 2017; Shi et al., 2016), or looking at whether the end-to-end system correctly translates sentences with challenging constructions (Sen-nrich, 2017).”
Linguistic Knowledge and Transferability of Contextual Representations (2019, methodology): “…learned representations in neural models, such as directly examining the activations of individual neurons (Karpathy et al., 2015; Li et al., 2015; Shi et al., 2016a, inter alia ), ablating components of the model and dataset (Kuncoro et al., 2017; Gaddy et al., 2018; Khandelwal et al., 2018), or…”
Pre-Trained Models: Past, Present and Future (2021, methodology): “Representation probing have been widely applied to analyze NLP neural models from word embeddings to PTMs (Köhn, 2015; Ettinger et al., 2016; Shi et al., 2016; Adi et al., 2017; Conneau et al., 2018a; Hewitt and Manning, 2019; Glavaš and Vulić, 2021).”