Challenges in Data-to-Document Generation

By Sam Wiseman, Stuart M. Shieber, Alexander M. Rush. 2017

655 citations recorded by the source corpus.

Recent neural models have shown significant progress on the problem of generating short descriptive texts conditioned on a small number of database records. In this work, we suggest a slightly more difficult data-to-text generation task, and investigate how effective current approaches are on this task. In particular, we introduce a new, large-scale corpus of data records paired with descriptive documents, propose a series of extractive evaluation methods for analyzing performance, and obtain baseline results using current neural generation methods. Experiments show that these models produce fluent text, but fail to convincingly approximate human-generated documents. Moreover, even templated baselines exceed the performance of these neural models on some metrics, though copy- and reconstruction-based extensions lead to noticeable improvements.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

The Curious Case of Neural Text Degeneration (2019, methodology): “(2) Computing the optimum argmax sequence from recurrent neural language models is not tractable, so consider two prominent decoding methods for approximating the argmax: Beam search is the most commonly used approximation in practice (Li et al., 2016c; Shen et al., 2017; Wiseman et al., 2017).”
Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, background): “For table-to-text, Thomson and Reiter (2020) report the performance of an information extraction-based metric (Wiseman et al., 2017) given different types of errors, and highlights typically problematic cases such as errors with names and numbers which are not detected by the metric.”
The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics (2021, background): “ROTOWIRE/MLB (Wiseman et al., 2017; Puduppully et al., 2019) 14https://www.kaggle.com/shishu1421/hindi-poetrydataset 22.”
Semantic Noise Matters for Neural Natural Language Generation (2019, background): “…reports on hallucinations (i.e. adding information not grounded in the input), as observed for image captioning (Rohrbach et al., 2018), sports report generation (Wiseman et al., 2017), machine translation (Koehn and Knowles, 2017; Lee et al., 2019), and question answering (Feng et al., 2018).”
Hierarchical Neural Story Generation (2018, background): “…generation technology: stories must remain thematically consistent across the complete document, requiring modeling very long range dependencies; stories require creativity; and stories need a high level plot, necessitating planning ahead rather than word-by-word generation (Wiseman et al., 2017).”
Measuring Attribution in Natural Language Generation Models (2021, background): “Wiseman et al. (2017) presented challenges in data-to-text generation where neural models generate hallucinated content not supported by source data; they proposed an automatic information extraction-based metric to evaluate generated text for that particular scenario and conducted a small human…”
On Faithfulness and Factuality in Abstractive Summarization (2020, background): “In contrast, tasks such as document summarization (Nenkova and McKeown, 2011; See et al., 2017; Paulus et al., 2018) and data-to-text generation (Lebret et al., 2016; Wiseman et al., 2017) which are not open-ended, require models to be factual and/or faithful to the source text.”
Survey of Hallucination in Natural Language Generation (2022, background): “Data2text [195] Intrinsic TEAM CITY WIN LOSS PTS FG_PCT BLK The Houston Rockets (18-4) defeated the Denver Nuggets (10-13) 108-96 on Saturday.”
DART: Open-Domain Structured Data Record to Text Generation (2020, methodology): “Furthermore, some of them only have loose alignments between data input and sentence due to the automatic generation procedure (e.g., RotoWire, Neural Wikipedian (Vougiouklis et al., 2018), and T-Rex (Elsahar et al., 2018)).”
BLEURT: Learning Robust Metrics for Text Generation (2020, background): “However, progress is increasingly impeded by the shortcomings of existing metrics (Wiseman et al., 2017; Ma et al., 2019; Tian et al., 2019).”
Learning to Write with Cooperative Discriminators (2018, background): “…automatic mea-5 sures such as BLEU (Papineni et al., 2002) and Meteor (Denkowski and Lavie, 2010) do not lead to meaningful evaluation when used for long or creative text generation where there can be high variance among acceptable generation outputs (Wiseman et al., 2017; Vedantam et al., 2015).”
ToTTo: A Controlled Table-To-Text Generation Dataset (2020, methodology): “Target Quality Target Source Content Selection Wikibio (Lebret et al., 2016) 583K Biographies Noisy Wikipedia Not speciﬁed Rotowire (Wiseman et al., 2017) 4.9K Basketball Noisy Rotowire Not speciﬁed WebNLG (Gardent et al., 2017b) 25.3K 15 DBPedia categories Clean Annotator Generated Fully speciﬁed E”
A Discourse-Aware Attention Model for Abstractive Summarization of Long Documents (2018, background): “Copying from source There has been a surge of recent works in sequence learning tasks to address the problem of unkown token prediction by allowing the model to occasionally copy words directly from source instead of generating a new token (Gu et al., 2016; See et al., 2017; Paulus et al., 2017; Wis”
Handling Divergent Reference Texts when Evaluating Table-to-Text Generation (2019, methodology): “Given this information extraction system, we consider the following metrics for evaluation, along the lines of Wiseman et al. (2017).”
Evaluating the State-of-the-Art of End-to-End Natural Language Generation: The E2E NLG Challenge (2019, methodology): “Apart from delexicalisation, which is most often used in the E2E NLG Challenge, various variants of the copy mechanism are the most prominent approach to address open vocabulary in NLG (Wiseman et al., 2017; Lebret et al., 2016; Bao et al., 2018; Kaffee et al., 2018; Wang et al., 2018).”
Hurdles to Progress in Long-form Question Answering (2021, neutral): “A possible explanation for this issue is high source-reference divergence, a common problem in table-to-text generation (Wiseman et al., 2017; Tian et al., 2019).”
Barack’s Wife Hillary: Using Knowledge Graphs for Fact-Aware Language Modeling (2019, methodology): “Our goals are much more aligned to the data-to-text task (Ahn et al., 2016; Lebret et al., 2016; Wiseman et al., 2017; Yang et al., 2017; Gardent et al., 2017; Ferreira et al., 2018), where a small table-sized KB is provided to generate a short piece of text; we are interested in language models…”
RankME: Reliable Human Ratings for Natural Language Generation (2018, methodology): “This can also be approximated automatically, as demonstrated by Wiseman et al. (2017).”