Language Modeling Teaches You More than Translation Does: Lessons Learned Through Auxiliary Syntactic Task Analysis

By Kelly W. Zhang, Samuel R. Bowman. 2018

86 citations recorded by the source corpus.

Recently, researchers have found that deep LSTMs trained on tasks like machine translation learn substantial syntactic and semantic information about their input sentences, including part-of-speech. These findings begin to shed light on why pretrained representations, like ELMo and CoVe, are so beneficial for neural language understanding models. We still, though, do not yet have a clear understanding of how the choice of pretraining objective affects the type of linguistic information that models learn. With this in mind, we compare four objectives—language modeling, translation, skip-thought, and autoencoding—on their ability to induce syntactic and part-of-speech information, holding constant the quantity and genre of the training data, as well as the LSTM architecture.

Subfields: Artificial Intelligence, Large Language Models, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

In Natural Language Processing

Citations in the corpus

The Bottom-up Evolution of Representations in the Transformer: A Study with Machine Translation and Language Modeling Objectives (2019, neutral): “Previous work has made some interesting observations regarding these representations; for example, Zhang and Bowman (2018) show that untrained LSTMs outperform trained ones on a word identity prediction task; and Blevins et al. (2018) show that up to a certain layer performance of representations…”
Decoupling the Role of Data, Attention, and Losses in Multimodal Transformers (2021, methodology): “We focus on zero-shot evaluation as it enables us to examine the representations without confounding our ﬁndings with the side-effects of ﬁne-tuning (Yogatama et al., 2019) or probing classi-ﬁers ( e.g. , Zhang and Bowman, 2018; Hewitt and Liang, 2019).”
Probing Classifiers: Promises, Shortcomings, and Advances (2021, background): “Others advocate for random baselines, training the classi-ﬁer g on a randomized version of f (Conneau et al. 2018; Zhang and Bowman 2018; Tenney et al. 2019; Chrupała, Higy, and Alishahi 2020).”
What do you learn from context? Probing for sentence structure in contextualized word representations (2019, background): “Randomized neural networks have recently (Zhang & Bowman, 2018) shown surprisingly strong performance on many tasks, suggesting that architecture may play a signiﬁcant role in learning useful feature functions.”
Information-Theoretic Probing with Minimum Description Length (2020, methodology): “Probes using these representations show surprisingly strong performance for both token (Zhang and Bowman, 2018) and sentence (Wieting and Kiela, 2019) representations.”
Probing the Probing Paradigm: Does Probing Accuracy Entail Task Relevance? (2020, background): “…the sentence representations can directly be probed for linguistic properties of interest (Ettinger et al., 2016; Adi et al., 2017; Tenney et al., 2019; Hewitt and Manning, 2019; Warstadt et al., 2019; Zhang and Bowman, 2018), which can give insight into the competencies a model uses to do a task.”
Is neural language acquisition similar to natural? A chronological probing study (2022, unknown): “(Hewitt and Liang, 2019a; Zhang and Bowman, 2018; Voita and Titov, 2020; Pimentel et al., 2020).”
Towards Automated Circuit Discovery for Mechanistic Interpretability (2023, methodology): “Examples of gradient information being used more generally includes Michel, Levy, and Neubig (2019) who decide which heads should be pruned by using the absolute value of their gradients, while “movement pruning” (Sanh, Wolf, and Rush, 2020) removes parameters that have high velocity to a low…”
Contextual and Non-Contextual Word Embeddings: an in-depth Linguistic Investigation (2020, background): “…Vig and Belinkov, 2019) or designing speciﬁc probing tasks that a model can solve only if it captures a precise linguistic phenomenon using the contextual word/sentence embeddings of a pre-trained model as training features (Conneau et al., 2018; Zhang and Bowman, 2018; Hewitt and Liang, 2019).”
Post-hoc Interpretability for Neural NLP: A Survey (2021, methodology): “Zhang and Bowman [137] suggest learning a probe from an untrained model, as a baseline.”
Emergent Structures and Training Dynamics in Large Language Models (2022, methodology): “…and 1 their connection to the downstream performance linguistic structure (Belinkov et al., 2017a,b; Peters et al., 2018; Tenney et al., 2019; Conneau et al., 2018; Zhang and Bowman, 2018; Alain and Ben-134 gio, 2017; Hewitt and Manning, 2019; Hewitt and Liang, 2019) using probing methods.”
Asking without Telling: Exploring Latent Ontologies in Contextual Representations (2020, methodology): “An alternative is to control amount of supervision , whether by restricting training set size (Zhang and Bowman, 2018), comparing learning curves (Talmor et al., 2019), or using description length with online coding (Voita and Titov, 2020).”
Probing Classifiers: Promises, Shortcomings, and Alternatives (2021, background): “Others advocate for random baselines, training the classi-ﬁer g on a randomized version of f (Conneau et al. 2018; Zhang and Bowman 2018; Chrupała, Higy, and Alishahi 2020).”
Interpretability and Analysis in Neural NLP (2020, methodology): “…Kim et al., 2019; McCoy et al., 2019; Ross and Pavlick, 2019), psycholinguistic methods (Ettinger et al., 2018; Chrupała and Alishahi, 2019), layerwise analyses (Peters et al., 2018; Tenney et al., 2019a), among other methods (Hewitt and Manning, 2019; Zhang and Bowman, 2018; Shi et al., 2016).”
Low-Complexity Probing via Finding Subnetworks (2021, methodology): “962 First, we consider the non-parametric test of probing a random model (Zhang and Bowman, 2018).”