Survey of the State of the Art in Natural Language Generation: Core tasks, applications and evaluation

By Albert Gatt, Emiel Krahmer. 2017

912 citations recorded by the source corpus.

This paper surveys the current state of the art in Natural Language Generation (NLG), defined as the task of generating text or speech from non-linguistic input. A survey of NLG is timely in view of the changes that the field has undergone over the past two decades, especially in relation to new (usually data-driven) methods, as well as new applications of NLG technology. This survey therefore aims to (a) give an up-to-date synthesis of research on the core tasks in NLG and the architectures adopted in which such tasks are organised; (b) highlight a number of recent research topics that have arisen partly as a result of growing synergies between NLG and other areas of artifical intelligence; (c) draw attention to the challenges in NLG evaluation, relating them to similar challenges faced in other areas of nlp, with an emphasis on different evaluation methods and the relationships between them.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, methodology): “Summarization datasets thus have strong positional data biases that models pick up on and which lead to inflated results if the test set has the same biases (Gehrmann et al., 2018). If the claims made about such a model are specific to news summarization and resulting models are only used to summari”
The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics (2021, background): “They measure specific generation challenges, such as the content selection and planning (What to say?), and the surface realization (How to say it?) (Reiter and Dale, 2000; Gatt and Krahmer, 2018).”
Structure-to-Text Generation with Self-Training, Acceptability Classifiers and Context-Conditioning for the GEM Shared Task (2021, background): “In the research community, generative models (Wen et al., 2015; Duˇsek and Jurcıcek, 2016; Rao et al., 2019) have become popular for their data-driven scaling story and superior naturalness over typical template-based systems (Gatt and Krahmer, 2018; Dale, 2020).”
Survey of Hallucination in Natural Language Generation (2022, background): “The source in abstractive summarization is the input source text that is being summarized [165], while the source in data-totext is non-linguistic data [56, 157], and the source(s) in the dialogue system is dialogue history and/or the external knowledge sentences.”
Evaluating the State-of-the-Art of End-to-End Natural Language Generation: The E2E NLG Challenge (2019, background): “Traditionally, human evaluation aims to assess the naturalness (fluency, readability) and informativeness (relevance, correctness, adequacy) of an automatically generated output (Gatt and Krahmer, 2017).”
Adversarial Text Generation Without Reinforcement Learning (2018, background): “A recent 2018 survey paper (Gatt and Krahmer, 2018) includes the discussion of neural approaches to natural language generation (NLG) in three out of four chapters dedicated to the current state-of-the-art methods.”
Twenty Years of Confusion in Human Evaluation: NLG Needs Evaluation Sheets and Standardised Definitions (2020, background): “Human evaluations are commonly viewed as a more reliable way to evaluate NLG systems (Celikyilmaz et al., 2020; Gatt and Krahmer, 2018), but come with their own issues, such as cost and time involved, the need for domain expertise (Celikyilmaz et al., 2020), and the fact that the experimental setup…”
MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distance (2019, background): “Many natural language generation (NLG) tasks can benefit from robust and unbiased evaluation, including textto-text (machine translation and summarization), data-to-text (response generation), and image-totext (captioning) (Gatt and Krahmer, 2018).”
Multi-hop Reading Comprehension through Question Decomposition and Rescoring (2019, background): “Moreover, generating the question word-by-word is known to be a difficult task that requires substantial training data and is not straight-forward to evaluate (Gatt and Krahmer, 2018; Novikova et al., 2017).”
Findings of the E2E NLG Challenge (2018, background): “The aim of this challenge was to assess the capabilities of recent end-to-end, fully data-driven NLG systems, which can be trained from pairs of input MRs and texts, without the need for ﬁne-grained semantic alignments.”
Evaluation of Text Generation: A Survey (2020, background): “Many NLG surveys have been published in the last few years (Gatt & Krahmer, 2017; Zhu et al., 2018; Zhang et al., 2019a).”
e-ViL: A Dataset and Benchmark for Natural Language Explanations in Vision-Language Tasks (2021, background): “Evaluating NLG is challenging and a much-studied field [19].”
Few-Shot NLG with Pre-Trained Language Model (2019, background): “ng automatic NLG systems in a wide range of real-life applications. In order to tackle such a research problem, traditional pipeline-based NLG systems (Becker 2002; Deemter, Theune, and Krahmer 2005; Gatt and Krahmer 2018) have been 1https://github.com/czyssrs/Few-Shot-NLG proposed, which heavily re”
Data-to-Text Generation with Content Selection and Planning (2018, background): “1 Introduction Data-to-text generation broadly refers to the task of automatically producing text from non-linguistic input (Reiter and Dale 2000; Gatt and Krahmer 2018).”
Content Selection in Deep Learning Models of Summarization (2018, background): “Content selection is a central component in many natural language generation tasks, where, given a generation goal, the system must determine which information should be expressed in the output text (Gatt and Krahmer, 2018).”
Generate Natural Language Explanations for Recommendation (2021, background): “A well trained neural network could learn lower-dimension dense representations to capture grammatical and semantical generalizations [11].”
Data-to-text Generation with Entity Modeling (2019, background): “Data-to-text generation is the task of generating textual output from non-linguistic input (Reiter and Dale, 1997; Gatt and Krahmer, 2018).”