Neural Text Generation from Structured Data with Application to the Biography Domain

By Rémi Lebret, David Grangier, Michael Auli. 2016

586 citations recorded by the source corpus.

This paper introduces a neural model for concept-to-text generation that scales to large, rich domains. We experiment with a new dataset of biographies from Wikipedia that is an order of magnitude larger than existing resources with over 700k samples. The dataset is also vastly more diverse with a 400k vocabulary, compared to a few hundred words for Weathergov or Robocup. Our model builds upon recent work on conditional neural language model for text generation. To deal with the large vocabulary, we extend these models to mix a fixed vocabulary with copy actions that transfer sample-specific words from the input database to the generated output sentence. Our neural model significantly out-performs a classical Kneser-Ney language model adapted to this task by nearly 15 BLEU.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Generating Wikipedia by Summarizing Long Sequences (2018, background): “There is a rich body of work incorporating Wikipedia for machine learning tasks, including question-answering (Hewlett et al. (2016), Rajpurkar et al. (2016)) and information extraction (Lehmann et al., 2015), and text generation from structured data (Lebret et al., 2016).”
Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, background): “In WikiBio (Lebret et al., 2016), which has the communicative goal to provide a short biography grounded in key-value attributes about a person, less than half of the attributes are actually realized in the reference and over half of references score very low or low in faithfulness on a 5-point scal”
Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus (2021, methodology): “We examine contamination of target text in test sets of datasets for three generation tasks: (i) abstractive summarization (TIFU, Kim et al., 2019; XSum, Narayan et al., 2018), (ii) table-to-text generation (WikiBio, Lebret et al., 2016), and (iii) graph-to-text generation (AMR-to-text, LDC2017T10).”
The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics (2021, neutral): “WikiBio (Lebret et al., 2016) 31.”
Creating Training Corpora for NLG Micro-Planners (2017, methodology): “We ignore here (Lebret et al., 2016)’s dataset which was created fully automatically from Wikipedia by associating infoboxes with text because this dataset fails to ensure an adequate match between data and text.”
Aligning Language Models with Preferences through f-divergence Minimization (2023, methodology): “For the first experiment, we use GPT-2 small as the initial model a, additionally fine-tuned on the WikiBio dataset (Lebret et al., 2016), whereas for the last one we use vanilla GPT-2 small.”
On Reinforcement Learning and Distribution Matching for Fine-Tuning Language Models with no Catastrophic Forgetting (2022, methodology): “For tasks 7-10, a is the same pre-trained model additionally fine-tuned on the WikiBio (Lebret et al., 2016) dataset.”
On Faithfulness and Factuality in Abstractive Summarization (2020, background): “Dhingra et al. (2019) proposed a new automatic metric, PARENT, for data-to-text generation (Lebret et al., 2016; Wiseman et al., 2017) which aligns n -grams from the reference and generated texts to the source table to measure the accuracy of n -grams that are entailed from the source table.”
Survey of Hallucination in Natural Language Generation (2022, methodology): “For instance, when constructing WIKIBIO [78], a dataset for generating biography notes based on infobox, the authors took the Wikipedia infobox as the source and the first sentence of the Wikipedia page as the target ground-truth reference, which is not necessary to supported by the infobox.”
DART: Open-Domain Structured Data Record to Text Generation (2020, methodology): “First, they adopt a flat ontology structure of the data, such as slot-value pairs for data records (Lebret et al., 2016; Novikova et al., 2017b) or flat schema for tables (Wiseman et al.”
CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLP (2021, neutral): “…reading comprehension Xiong et al. 2019 web_questions qa/closed-book qa Berant et al. 2013 wiki_auto cls/other Jiang et al. 2020 wiki_bio cg/other Lebret et al. 2016 wiki_qa cls/other Yang et al. 2015 wiki_split cg/other Botha et al. 2018 wikisql cg/other Zhong et al. 2017 wino_grande…”
Challenges in Data-to-Document Generation (2017, methodology): “Within the world of neural text generation, some recent work has focused on conditioning language models on tables (Yang et al., 2016), and generating short biographies from Wikipedia Tables (Lebret et al., 2016; Chisholm et al., 2017).”
ToTTo: A Controlled Table-To-Text Generation Dataset (2020, methodology): “To prevent any overlap with the Wikibio dataset (Lebret et al., 2016), we do not use infobox tables.”
Controlling Linguistic Style Aspects in Neural Language Generation (2017, background): “Most of these research efforts focus on controlling the content of the generated text (Lipton et al., 2015; Kiddon et al., 2016; Lebret et al., 2016; Kiddon et al., 2016; Tang et al., 2016; Radford et al., 2017), while a few model more stylistic aspects of the generated text such as the identity of ”
Handling Divergent Reference Texts when Evaluating Table-to-Text Generation (2019, methodology): “Our main experiments are on the WikiBio dataset (Lebret et al., 2016), which is automatically constructed and contains many divergent references.”
Evaluating the State-of-the-Art of End-to-End Natural Language Generation: The E2E NLG Challenge (2019, methodology): “Apart from delexicalisation, which is most often used in the E2E NLG Challenge, various variants of the copy mechanism are the most prominent approach to address open vocabulary in NLG (Wiseman et al., 2017; Lebret et al., 2016; Bao et al., 2018; Kaffee et al., 2018; Wang et al., 2018).”
A Distributional Approach to Controlled Text Generation (2020, methodology): “For distributional and hybrid experiments, we fine-tune GPT-2 small (117M params) to produce biographies on a dataset of 700K Wikipedia biographies (Lebret et al., 2016) which we refer to as GPT-2bio.”
Barack’s Wife Hillary: Using Knowledge Graphs for Fact-Aware Language Modeling (2019, methodology): “, 2017), and Wikipedia info-boxes (Lebret et al., 2016).”
A Neural Knowledge Language Model (2016, background): “Similarly, Merity et al. (2016) proposed to copy from the context sentences and Lebret et al. (2016) from Wikipedia infobox.”