Language Modeling Teaches You More Syntax than Translation Does: Lessons Learned Through Auxiliary Task Analysis

By Kelly W. Zhang, Samuel R. Bowman. 2018

71 citations recorded by the source corpus.

Recent work using auxiliary prediction task classifiers to investigate the properties of LSTM representations has begun to shed light on why pretrained representations, like ELMo (Peters et al., 2018) and CoVe (McCann et al., 2017), are so beneficial for neural language understanding models. We still, though, do not yet have a clear understanding of how the choice of pretraining objective affects the type of linguistic information that models learn. With this in mind, we compare four objectives---language modeling, translation, skip-thought, and autoencoding---on their ability to induce syntactic and part-of-speech information. We make a fair comparison between the tasks by holding constant the quantity and genre of the training data, as well as the LSTM architecture. We find that representations from language models consistently perform best on our syntactic auxiliary prediction tasks, even when trained on relatively small amounts of data. These results suggest that language modeling may be the best data-rich pretraining task for transfer learning applications requiring syntactic information. We also find that the representations from randomly-initialized, frozen LSTMs perform strikingly well on our syntactic auxiliary tasks, but this effect disappears when the amount of training data for the auxiliary tasks is reduced.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Dissecting Contextual Word Embeddings: Architecture and Representation (2018, background): “We also note that the faster architectures will allow training to scale to large unlabeled corpora, which has been shown to improve the quality of biLM representations for syntactic tasks (Zhang and Bowman, 2018).”
Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Tasks (2018, background): “Zhang and Bowman (2018) compare several pre-training tasks for syntactic target tasks, and ﬁnd that language model pretraining reliably performs well.”
Transfer Learning in Natural Language Processing (2019, methodology): “This review of pretraining approaches will provide recommendations and discuss trade-offs of pretraining tasks based on our own experiments and recent studies (Zhang and Bowman, 2018; Anonymous, 2019).”
Cloze-driven Pretraining of Self-attention Networks (2019, background): “which does not require bilingual data (Zhang and Bowman, 2018).”
Can You Tell Me How to Get Past Sesame Street? Sentence-Level Pretraining Beyond Language Modeling (2018, background): “This relative weakness on sentence structure is somewhat surprising given the ﬁnding in Zhang and Bowman (2018) that language model pretraining is helpful for tasks involving sentence structure.”
What Does BERT Look at? An Analysis of BERT’s Attention (2019, background): “Another line of work investigates the internal vector representations of the model (Adi et al., 2017; Giulianelli et al., 2018; Zhang and Bowman, 2018), often using probing classifiers.”
Designing and Interpreting Probes with Control Tasks (2019, background): “2740 We also examine the representations of an untrained BiLSTM run on the non-contextual character CNN word embeddings of ELMo, shown to be a strong baseline contextualization method, but without any linguistic knowledge learned from context (Zhang and Bowman, 2018; Hewitt and Manning, 2019).”
Linguistic Knowledge and Transferability of Contextual Representations (2019, background): “Zhang and Bowman (2018) compare the syntactic knowledge of language models and neural machine translation systems.”
What Does BERT Learn about the Structure of Language? (2019, methodology): “As random encoders can surprisingly encode a lot of lexical and structural information (Zhang and Bowman, 2018), we also evaluate the untrained version of BERT, obtained by setting all model weights to a random number.”
Looking for ELMo's friends: Sentence-Level Pretraining Beyond Language Modeling (2018, background): “This baseline works well, yielding scores far above those of a bag-of-words encoder.1 This surprising result matches results seen recently with ELMo-like models by Zhang & Bowman (2018) and earlier work on Reservoir Computing.”
Investigating Multilingual NMT Representations at Scale (2019, methodology): “…uses methods such as diagnostic tasks (Belinkov et al., 2017; Tenney et al., 2019; Belinkov et al., 2018), attention based methods (Raganato and Tiedemann, 2018) or task analysis (Zhang and Bowman, 2018) and is primarily focused on understanding the linguistic features encoded by a trained model.”
Information-Theoretic Probing for Linguistic Structure (2020, background): “, supervised models that are trained to extract the linguistic properties directly (Belinkov et al., 2017; Conneau et al., 2018; Peters et al., 2018b; Zhang and Bowman, 2018; Naik et al., 2018; Tenney et al., 2019).”
Conditional probing: measuring usable information beyond a baseline (2021, unknown): “(Zhang and Bowman, 2018); we call this baselined probing performance:”
Deep Contextualized Self-training for Low Resource Dependency Parsing (2019, background): “Indeed, in recent papers untrained LSTMs with random weights substantially enhanced model performance (Wang et al., 2019; Zhang and Bowman, 2018; Tenney et al., 2019; Wieting and Kiela, 2019).”
Correlating Neural and Symbolic Representations of Language (2019, result): “The non-random nature of the activation patterns of randomly initialized models (e.g., Zhang and Bowman, 2018) is also strongly in evidence in our results.”
Analyzing analytical methods: The case of phonology in neural models of spoken language (2020, background): “Previous work has shown that sometimes non-trivial representations can be found in the activation patterns of randomly initialized, untrained neural networks (Zhang and Bowman, 2018; Chrupała and Alishahi, 2019).”
The neural architecture of language: Integrative modeling converges on predictive processing (2020, background): “…visual processing (Geiger et al., 2020) and speech perception (Millet & King, 2021), as well as recent results in natural language processing (Merchant et al., 2020; Tenney et al., 2019; Zhang & Bowman, 2018), but it raises important questions of interpretation in the context of human language.”