Few-Shot NLG with Pre-Trained Language Model

By Zhiyu Chen, Harini Eavani, Yinyin Liu, William Yang Wang. 2019

150 citations recorded by the source corpus.

Neural-based end-to-end approaches to natural language generation (NLG) from structured data or knowledge are data-hungry, making their adoption for real-world applications difficult with limited data. In this work, we propose the new task of few-shot natural language generation. Motivated by how humans tend to summarize tabular data, we propose a simple yet effective approach and show that it not only demonstrates strong performance but also provides good generalization across domains. The design of the model architecture is based on two aspects: content selection from input data and language modeling to compose coherent sentences, which can be acquired from prior knowledge. With just 200 training examples, across multiple domains, we show that our approach achieves very reasonable performances and outperforms the strongest baseline by an average of over 8.0 BLEU points improvement. Our code and data can be found at https://github.com/czyssrs/Few-Shot-NLG

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Neural Generation for Czech: Data and Baselines (2019, methodology): “13 In future work, we will collect a large unannotated dataset and pretrain the generator (Chen et al., 2019).”
Structure-to-Text Generation with Self-Training, Acceptability Classifiers and Context-Conditioning for the GEM Shared Task (2021, background): “Pre-trained models like GPT2 (Radford et al., 2019) have shown great potential to address this challenge (Peng et al., 2020; Chen et al., 2020), and combining pre-trained models with self-training has been shown to improve data efﬁciency even further (Arun et al., 2020).”
Few-Shot Out-of-Domain Transfer Learning of Natural Language Explanations (2021, background): “In the more general area of natural language generation, few-shot learning is a growing topic (Chen et al., 2020), e.g., in dialog generation (Peng et al., 2020; Shalyminov et al., 2019).”
DART: Open-Domain Structured Data Record to Text Generation (2020, background): “Chen et al. (2020c); Baolin Peng (2020); Kale (2020) also ﬁnetune pre-trained language models on data-to-text tasks.”
Text-to-Text Pre-Training for Data-to-Text Tasks (2020, background): “4 Our baselines are • HDSA (Chen et al., 2019a) is a transformer based architecture that encodes the dialog acts into a multi-layer hierarchical graph, with individual attention heads modeling speciﬁc nodes in graph.”
Building Adaptive Acceptability Classifiers for Neural NLG (2021, background): “Recently, there have been several efforts to train NLG models in a few-shot setting (Chen et al., 2020; Peng et al., 2020; Arun et al., 2020; Heidari et al., 2021).”
Few-Shot Natural Language Generation by Rewriting Templates (2020, methodology): “Baselines Besides our proposed approaches, we also compare with the following baselines: • HDSA - Hierarchically Disentangled Self- Attention (Chen et al., 2019a), a transformer based architecture that exploits the structure of dialog acts to build a multi-layer hierarchical graph.”
ESPRIT: Explaining Solutions to Physical Reasoning Tasks (2020, methodology): “Chen et al. (2020b) use pre-trained language models to generate descriptions for tabular data in a few shot setting.”
Template Guided Text Generation for Task Oriented Dialogue (2020, background): “Domainadaptation and transfer learning in low resource settings has also been an extensively studied problem (Tran and Le Nguyen, 2018; Chen et al., 2020; Peng et al., 2020; Mi et al., 2019), with recently released datasets like SGD (Rastogi et al., 2019) and FewShotWOZ (Peng et al., 2020)…”
Neural Data-to-Text Generation with LM-based Text Augmentation (2021, methodology): “We compared four types of models: the seq2seq baseline, seq2seq plus cycle-consistency as in Qader et al. (2019), ﬁnetuned GPT-2 as in Chen et al. (2020) and our proposed model.”
Controlling hallucinations at word level in data-to-text generation (2021, methodology): “In this direction, adapting the work of [5, 22] to our model could bring improvements to the results presented in this paper.”
Disentangling the Properties of Human Evaluation Methods: A Classification System to Support Comparability, Meta-Evaluation and Reproducibility Testing (2020, background): “E.g. Chen et al. (2020) define Language Naturalness as “whether the generated text is grammatically correct and fluent, regardless of factual correctness”, while Juraska et al. (2019) give essentially the same definition (see preceding paragraph) for Coherence.”
Pre-Trained Language Models for Text Generation: A Survey (2022, methodology): “For data-to-text generation, the pointer generator [164] is usually adopted to copy the input facts into output for preserving factuality [29, 105].”
Towards Table-to-Text Generation with Numerical Reasoning (2021, methodology): “…sources in natural language tasks, such as table-to-text generation (Liang et al., 2009; Wiseman et al., 2017; Lebret et al., 2016; Parikh et al., 2020), table question answering (Pasupat and Liang, 2015; Wang et al., 2018), and table-based fact veriﬁcation (Chen et al., 2020b; Gupta et al., 2020).”
Logic2Text: High-Fidelity Natural Language Generation from Logical Forms (2020, methodology): “We conduct experiment without using the logical form, i.e., only feed the table caption, table header and the table content as input, which is the task setting of generating arbitrary logically correct descriptions in (Chen et al., 2020).”
Getting to Production with Few-shot Natural Language Generation Models (2021, methodology): “As noted above, Peng et al. (2020) and Chen et al. (2020) likewise explore pre-trained models for few-shot NLG in task-oriented dialog, but they do not investigate how to achieve acceptable latency while maintaining high quality.”
Plan-then-Generate: Controlled Data-to-Text Generation via Planning (2021, methodology): “The Switch-GPT model applies a copy mechanism to copy content from the source to the output.”