Large-Scale Transfer Learning for Natural Language Generation

By Sergey Golovanov, Rauf Kurbanov, Sergey I. Nikolenko, Kyryl Truskovskyi, Alexander Tselousov, Thomas Wolf. 2019

93 citations recorded by the source corpus.

Large-scale pretrained language models define state of the art in natural language processing, achieving outstanding performance on a variety of tasks. We study how these architectures can be applied and adapted for natural language generation, comparing a number of architectural and training schemes. We focus in particular on open-domain dialog as a typical high entropy generation task, presenting and comparing different architectures for adapting pretrained models with state of the art results.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

HuggingFace's Transformers: State-of-the-art Natural Language Processing (2019, positive): “The approach used to reach these performances is described in Wolf et al. (2019); Golovanov et al. (2019) and the code and pretrained models, based on the Transformers library, are available online 6 .”
HuggingFace's Transformers: State-of-the-art Natural Language Processing (2019, neutral): “-second Annual Conference on Neural Information Processing Systems (NIPS 2018). The approach used to reach 5https://transformer.huggingface.co 5 these performances is described in Wolf et al. (2019); Golovanov et al. (2019) and the code and pretrained models, based on the Transformers library, are a”
VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioning (2021, background): “Via self-supervised learning, these models acquire rich linguistic and semantic knowledge, which has been shown to inform downstream tasks in NLP [7, 21].”
Challenges in Building Intelligent Open-domain Dialog Systems (2019, methodology): “The fine-tuning process is often tailored to the dialog scenario via encoding with dialog state embeddings[155], classifying golden and negatively sampled responses given the same dialog context[39], designing dialog-specific pre-training tasks [11, 84], and so on.”
Hello, It’s GPT-2 - How Can I Help You? Towards the Use of Pretrained Language Models for Task-Oriented Dialogue Systems (2019, background): “This has been validated also for open-domain dialogue modeling (Wolf et al., 2019; Golovanov et al., 2019).”
ProphetNet-X: Large-Scale Pre-training Models for English, Chinese, Multi-lingual, Dialog, and Code Generation (2021, unknown): “…and Le, 2015) 0.336 0.238 0.03 0.128 0.183 0.448 0.353 0.004 0.016 0.205 iVAE MI (Fang et al., 2019) 0.309 0.249 0.029 0.25 0.209 - - - - - LIC (Golovanov et al., 2019) - - - - - 0.405 0 .320 0.019 0.113 0.214 PLATO w/o latent (Bao et al., 2020) Table 6: Results of ProphetNet-Dialog-En on…”
PLATO: Pre-trained Dialogue Generation Model with Discrete Latent Variable (2019, methodology): “In addition, it can be observed that the generation quality of transformed-based approaches (LIC and our method) is signiﬁcantly better than that of RNN-based methods (Seq2Seq and iVAE MI ).”
Pretrained Language Models for Dialogue Generation with Multiple Input Sources (2020, methodology): “Following the former work (Golovanov et al., 2019), we use the personalized dialogue generation task on PersonaChat (Zhang et al., 2018) as an example in our study.”
Pre-Trained Language Models for Text Generation: A Survey (2022, background): “[58] modified the architecture to better model multiple inputs including dialogue history, persona information, and current state.”
Encoder-Agnostic Adaptation for Conditional Language Generation (2019, background): “Concurrent with this work, Golovanov et al. (2019) propose a similar approach to pseudo self attention and report initial experiments with dialogue generation.”
Generating Counter Narratives against Online Hate Speech: Data and Strategies (2020, background): “Pre-trained Language Models (LMs) have achieved promising results when ﬁne-tuned on challenging generation tasks such as chit-chat dialog (Wolf et al., 2019; Golovanov et al., 2019).”
Recent advances in deep learning based dialogue systems: a systematic survey (2021, methodology): “Golovanov et al. (2019) used GPT as a sequence-to-sequence model to directly generate utterances and compared the performances under single- and multi-input settings.”
Exploring Prompt-based Few-shot Learning for Grounded Dialog Generation (2021, result): “effect of dialog grounding, as revealed in previous works (Wolf et al., 2019; Golovanov et al., 2019).”