Pivot Based Language Modeling for Improved Neural Domain Adaptation

By Yftah Ziser, Roi Reichart. 2018

135 citations recorded by the source corpus.

Representation learning with pivot-based methods and with Neural Networks (NNs) have lead to significant progress in domain adaptation for Natural Language Processing. However, most previous work that follows these approaches does not explicitly exploit the structure of the input text, and its output is most often a single representation vector for the entire text. In this paper we present the Pivot Based Language Model (PBLM), a representation learning model that marries together pivot-based and NN modeling in a structure aware manner. Particularly, our model processes the information in the text with a sequential NN (LSTM) and its output consists of a representation vector for every input word. Unlike most previous representation learning models in domain adaptation, PBLM can naturally feed structure aware text classifiers such as LSTM and CNN. We experiment with the task of cross-domain sentiment classification on 20 domain pairs and show substantial improvements over strong baselines.

Subfields: Artificial Intelligence, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

CausaLM: Causal Model Explanation Through Counterfactual Language Models (2020, background): “tion of pre-trained word-embeddings, there have been an explosion of research on choosing pre-training tasks and understanding their effect (Jernite, Bowman, and Sontag 2017; Logeswaran and Lee 2018; Ziser and Reichart 2018; Dong et al. 2019; Chang et al. 2019; Sun et al. 2019; Rotman and Reichart 2”
Train No Evil: Selective Masking for Task-guided Pre-training (2020, background): “Intuitively, some tokens are more important than others for a specific task and the important tokens vary among different tasks (Ziser and Reichart, 2018; Feng et al., 2018; Rietzler et al., 2020).”
Predicting In-Game Actions from Interviews of NBA Players (2019, methodology): “DNNs have proven effective for many text classiﬁcation tasks (Kim 2014; Ziser and Reichart 2018).”
UDALM: Unsupervised Domain Adaptation through Language Modeling (2021, background): “Ziser and Reichart (2017, 2018, 2019) combine SCL with neural network architectures and language modeling.”
Iterative Network Pruning with Uncertainty Regularization for Lifelong Sentiment Classification (2021, methodology): “Subsequently, [42] proposed combining the denoising auto-encoder and conventional pivot-based methods so as to further boost the accuracy.”
Deep Contextualized Self-training for Low Resource Dependency Parsing (2019, methodology): “, 2019) and particularly for domain adaptation (Ziser and Reichart, 2018).”
PADA: A Prompt-based Autoregressive Approach for Adaptation to Unseen Domains (2021, methodology): “2 In order to generate effective prompts, we draw inspiration from previous work on pivot features (Blitzer et al., 2006; Ziser and Reichart, 2018a; Ben-David et al., 2020) to deﬁne sets of Domain Related Features (DRFs, §4.2).”
The Hitchhiker’s Guide to Computational Linguistics in Suicide Prevention (2021, unknown): “…scenarios, many efforts are directed today at domain adaptation research, that is, developing methods that can help adapt a model that was trained with data from one textual domain so that it can produce accurate predictions on data from other domains (e.g., Ziser & Reichart, 2017, 2018).”
Unsupervised Domain Adaptation of Contextualized Embeddings for Sequence Labeling (2019, methodology): “Finally, language modeling objectives have previously been used for domain adaptation of text classiﬁers (Ziser and Reichart, 2018), but this prior work has focused on representation learning from scratch, rather than adaptation of a pretrained contextualized embedding model.”
Neural Unsupervised Domain Adaptation in NLP—A Survey (2020, methodology): “To solve this problem, a pivot-based language modeling (PBLM) method has been proposed (Ziser and Reichart, 2018a; Ziser and Reichart, 2018b).”