Text-to-Text Pre-Training for Data-to-Text Tasks

By Mihir Kale. 2020

220 citations recorded by the source corpus.

We study the pre-train + fine-tune strategy for data-to-text tasks. Our experiments indicate that text-to-text pre-training in the form of T5 (Raffel et al., 2019), enables simple, end-to-end transformer based models to outperform pipelined neural architectures tailored for data-to-text generation, as well as alternatives such as BERT and GPT-2. Importantly, T5 pre-training leads to better generalization, as evidenced by large improvements on out-ofdomain test sets. We hope our work serves as a useful baseline for future research, as transfer learning becomes ever more prevalent for data-to-text tasks.

Subfields: Artificial Intelligence, Information Systems, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Natural Language Processing

Citations in the corpus

Prefix-Tuning: Optimizing Continuous Prompts for Generation (2021, methodology): “There are several differences between XSUM and the three table-totext datasets which could account for why prefixtuning has comparative advantage in table-to-text: 7250K for E2E, 250K for WebNLG, and 500K for DART versus 345M GPT-2 parameters.”
DART: Open-Domain Structured Data Record to Text Generation (2020, background): “Chen et al. (2020c); Baolin Peng (2020); Kale (2020) also ﬁnetune pre-trained language models on data-to-text tasks.”
AggGen: Ordering and Aggregating while Generating (2021, methodology): “…we choose two types of previous works for comparison: (1) best-performing models reported on the WebNLG 2017 (seen) and E2E dataset, i.e. T5 (Kale and Rastogi, 2020), PlanEnc (Zhao et al., 2020), ADAPT (Gardent et al., 2017b), and TGen (Dušek and Jurčı́ček, 2016); (2) models with…”
Leveraging Large Pretrained Models for WebNLG 2020 (2020, background): “The optimizer is Adam where β 1 = 0 .”
Towards Table-to-Text Generation with Numerical Reasoning (2021, methodology): “T is converted to a ﬂat string P T = w 1 , ..., w | P T | , similar to that used in many prior work (Wang et al., 2020; Chen et al., 2020a; Kale and Rastogi, 2020b), where w i denotes the i -th word in para-graph P T with length | P T | .”
CINS: Comprehensive Instruction for Few-shot Learning in Task-oriented Dialog Systems (2021, methodology): “OOS For intent classification, we use a benchmark dataset from Larson et al. (2019). Apart from the single out-of-scope intent, it contains 150 intents in 15 domains.”
Investigating Pretrained Language Models for Graph-to-Text Generation (2020, methodology): “Recently, Harkous et al. (2020) and Kale (2020) demonstrate state-of-the-art re-sults in different data-to-text datasets, employing GPT-2 and T5 models respectively.”
Controlling hallucinations at word level in data-to-text generation (2021, unknown): “In this direction, adapting the work of [5, 22] to our model could bring improvements to the results presented in this paper.”
A Survey of Controllable Text Generation Using Transformer-based Pre-trained Language Models (2022, background): “Recent work has found that fine-tuning PLMs on the target data, such as AMR-to-text [56, 91, 111] for dialogue generation, can establish a new level of performance.”
GEMv2: Multilingual NLG Benchmarking in a Single Line of Code (2022, methodology): “For reproducibility, we implement linearization schemes following earlier work (e.g., Saleh et al., 2019; Kale and Rastogi, 2020; Pudup-pully and Lapata, 2021).”
Plan-then-Generate: Controlled Data-to-Text Generation via Planning (2021, methodology): “Because BERT-to-BERT and T5-3B do not take the content plan as input, thus we only measure the accuracy score for the results generated by our model and the reference sentence.”
Data-QuestEval: A Referenceless Metric for Data-to-Text Semantic Evaluation (2021, methodology): “We adopt the T5 (Kale and Rastogi, 2020) paradigm, where any task is considered as a Textto-Text task: we linearize the tables and encode them directly using T5.”
Unifying Large Language Models and Knowledge Graphs: A Roadmap (2023, methodology): “As pioneering research efforts in using LLMs for KG-to-Text generation, Ribeiro et al. [167] and Kale and Rastogi [231] directly fine-tune various LLMs, including BART and T5, with the goal of transferring LLMs knowledge for this task.”
Automatic Construction of Evaluation Suites for Natural Language Generation Datasets (2021, methodology): “Following [50], we employ a constant learning rate of 0.001.”