Encoder-Agnostic Adaptation for Conditional Language Generation

By Zachary M. Ziegler, Luke Melas-Kyriazi, Sebastian Gehrmann, Alexander M. Rush. 2019

60 citations recorded by the source corpus.

Large pretrained language models have changed the way researchers approach discriminative natural language understanding tasks, leading to the dominance of approaches that adapt a pretrained model for arbitrary downstream tasks. However it is an open-question how to use similar techniques for language generation. Early results in the encoder-agnostic setting have been mostly negative. In this work we explore methods for adapting a pretrained language model to arbitrary conditional input. We observe that pretrained transformer models are sensitive to large parameter changes during tuning. We therefore propose an adaptation that directly injects arbitrary conditioning into self attention, an approach we call pseudo self attention. Through experiments on four diverse conditional text generation tasks we show that this encoder-agnostic technique outperforms strong baselines, produces coherent generations, and is data efficient.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures, Natural Language Generation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

In Natural Language Generation

Citations in the corpus

Multimodal Few-Shot Learning with Frozen Language Models (2021, background): “Similarly, [46] and [6] show that a large pre-trained language model as decoder can improve a captioning performance when training data is limited.”
Connecting Vision and Language with Localized Narratives (2019, neutral): “In Out - Image captioning [51, 55, 52], Paragraph generation [26, 56, 64] Out In - Text-to-image Generation [42, 47, 57] In Out - Out Dense image captioning [21, 58, 25], Dense relational captioning [25] In Out - In Controllable and Grounded Captioning [10] In In - Out Phrase grounding [13] In In + ”
Natural Language Rationales with Full-Stack Visual Reasoning: From Pixels to Semantic Frames to Commonsense Graphs (2020, methodology): “Ziegler et al. (2019b) augment GPT-2 with a feature vector for the entire image and evaluate this model on image paragraph captioning.”
Putting Machine Translation in Context with the Noisy Channel Model (2019, methodology): “…area where pretrained language models have been applied to, with existing work focusing on ﬁne-tuning approaches for repurposing an unconditional language model (Zhang et al., 2019; Edunov et al., 2019; Song et al., 2019; Dong et al., 2019; Ziegler et al., 2019; de Oliveira & Rodrigo, 2019).”
Multimodal Conditionality for Natural Language Generation (2021, methodology): “A TTN : Context attention adds a randomly initialized encoder-decoder layer on top of every pretrained decoder layer of GPT-2 (Ziegler et al., 2019).”
Facts2Story: Controlling Text Generation by Key Facts (2020, methodology): “Attributing this to vast difference in structure between the key facts and the consecutive preﬁxes the model has been exposed to during pre-training, we resort to the technique of pseudo self attention as proposed by Ziegler et al. (2019).”
RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learning (2022, background): “Current state-of-the-art models typically fine-tune entire pre-trained LMs (e.g., Ziegler et al., 2019a; Keskar et al., 2019; Ziegler et al., 2019b; Liu et al., 2021e).”
Generating Abstractive Summaries with Finetuned Language Models (2019, methodology): “Note that here, for the sake of simplicity, we do not consider the Transformer with pseudo self attention and a copy mechanism which was reported to give strong performance in Ziegler et al. (2019).”
Transformers in Medical Imaging: A Survey (2022, methodology): “These semantic features are used to condition GPT2 decoder [373], [374] to generate accurate medical reports.”
Better Document-Level Machine Translation with Bayes’ Rule (2019, methodology): “…is another area where pretrained language models have been applied, with existing work focusing on ﬁne-tuning for repurposing an unconditional language model (Zhang et al., 2019; Edunov et al., 2019; Song et al., 2019; Dong et al., 2019; Ziegler et al., 2019; de Oliveira and Ro-drigo, 2019).”