ToTTo: A Controlled Table-To-Text Generation Dataset

By Ankur P. Parikh, Xuezhi Wang, Sebastian Gehrmann, Manaal Faruqui, Bhuwan Dhingra, Diyi Yang, Dipanjan Das. 2020

455 citations recorded by the source corpus.

We present ToTTo, an open-domain English table-to-text dataset with over 120,000 training examples that proposes a controlled generation task: given a Wikipedia table and a set of highlighted table cells, produce a one-sentence description. To obtain generated targets that are natural but also faithful to the source table, we introduce a dataset construction process where annotators directly revise existing candidate sentences from Wikipedia. We present systematic analyses of our dataset and annotation process as well as results achieved by several state-of-the-art baselines. While usually fluent, existing methods often hallucinate phrases that are not supported by the table, suggesting that this dataset can serve as a useful research benchmark for high-precision conditional text generation.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, methodology): “(2018) demonstrated that we may be able to optimize ROUGE directly, and Pasunuru and Bansal (2018) explored alternative optimization targets such as maximizing entailment.”
ExT5: Towards Extreme Multi-Task Scaling for Transfer Learning (2021, neutral): “GEM We report test set results on all datasets except CommonGen and ToTTo, on which we report validation scores.”
Is Reinforcement Learning (Not) for Natural Language Processing?: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization (2022, background): “ToTTo (Parikh et al., 2020) is a controlled table-to-text generation task in which the goal is to produce one-sentence description of highlighted table cells.”
UL2: Unifying Language Learning Paradigms (2022, methodology): “We arbitrarily select XSUM (summarization), ToTTo (table-to-text generation) (Parikh et al., 2020) and Schema Guided Dialog (SGD) (Rastogi et al.”
Measuring Attribution in Natural Language Generation Models (2021, methodology): “More recently, Parikh et al. (2020) presented a larger human evaluation study in the context of a data-to-text generation dataset entitled ToTTo, and measured hallucinations in terms of faithfulness with respect to a source data table.”
Survey of Hallucination in Natural Language Generation (2022, background): “TOTTO [140] is an open-domain faithful table-to-text dataset, where each sample includes a Wikipedia table with several highlighted cells and a description.”
DART: Open-Domain Structured Data Record to Text Generation (2020, methodology): “DART’s hierarchical and structured format differentiates itself from other open-domain table-to-text corpora such as ToTTo.”
Input-Tuning: Adapting Unfamiliar Inputs to Frozen Pretrained Models (2022, methodology): “ToTTo is the most challenging task among them, as it increases the generalization challenge by reducing the vocabulary overlap between train set and dev/test set.”
UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text Language Models (2022, background): “A phrase is supported by the table if it is either directly stated in the cell contents or meta-data, or can be logically inferred by them (Parikh et al., 2020).”
What Makes Good In-Context Examples for GPT-3? (2021, methodology): “ToTTo (Parikh et al., 2020) is utilized for evaluation due to its popularity.”
Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence (2021, background): “Generating texts that match given facts is a known challenge (Fan et al., 2020; Kryscinski et al., 2020; Lewis et al., 2020b; Parikh et al., 2020; Shah et al., 2020; Tian et al., 2020) as language models tend to degenerate and hallucinate (Holtzman et al., 2020; Schuster et al., 2020; Zhou et al.,…”
AgreeSum: Agreement-Oriented Multi-Document Summarization (2021, background): “More broadly, there have been other recent works striving to develop techniques for high-precision generation (Malmi et al., 2019; Tian et al., 2019; Pang and He, 2020; Parikh et al., 2020; Dušek and Kasner, 2020).”
Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models (2023, unknown): “Besides, Parikh et al. (2020) instruct annotators to revise verified sentences from Wikipedia rather than directly creating new sentences when constructing table-to-text training data.”
SynthBio: A Case Study in Human-AI Collaborative Curation of Text Datasets (2021, methodology): “A hybrid approach was taken in the case of ToTTo [37], a table-to-text dataset, for which researchers first retrieved samples from Wikipedia, then asked crowdworkers to edit those references.”
A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions (2023, unknown): “…such as abstractive summarization (Kryscinski et al., 2020; Wang et al., 2020a; Maynez et al., 2020; Fabbri et al., 2021; Goyal and Durrett, 2021; Pagnoni et al., 2021; Tang et al., 2022), data-to-text(Tian et al., 2019; Parikh et al., 2020), and machine translation (Zhou et al., 2021).”