Best Practices for Data-Efficient Modeling in NLG:How to Train Production-Ready Neural Models with Less Data

By Ankit Arun, Soumya Batra, Vikas Bhardwaj, Ashwini Challa, Pinar E. Donmez, Peyman Heidari, Hakan Inan, Shashank Jain, Anuj Kumar, Shawn Mei and 2 more. 2020

10 citations recorded by the source corpus.

Natural language generation (NLG) is a critical component in conversational systems, owing to its role of formulating a correct and natural text response. Traditionally, NLG components have been deployed using template-based solutions. Although neural network solutions recently developed in the research community have been shown to provide several benefits, deployment of such model-based solutions has been challenging due to high latency, correctness issues, and high data needs. In this paper, we present approaches that have helped us deploy data-efficient neural solutions for NLG in conversational systems to production. We describe a family of sampling and modeling techniques to attain production quality with light-weight neural network models using only a fraction of the data that would be necessary otherwise, and show a thorough comparison between each. Our results show that domain complexity dictates the appropriate approach to achieve high data efficiency. Finally, we distill the lessons from our experimental findings into a list of best practices for production-level NLG model development, and present them in a brief runbook. Importantly, the end products of all of the techniques are small sequence-to-sequence models (~2Mb) that we can reliably deploy in production. These models achieve the same quality as large pretrained models (~1Gb) as judged by human raters.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Structure-to-Text Generation with Self-Training, Acceptability Classifiers and Context-Conditioning for the GEM Shared Task (2021, background): “Pre-trained models like GPT2 (Radford et al., 2019) have shown great potential to address this challenge (Peng et al., 2020; Chen et al., 2020), and combining pre-trained models with self-training has been shown to improve data efﬁciency even further (Arun et al., 2020).”
Getting to Production with Few-shot Natural Language Generation Models (2021, methodology): “Arun et al. (2020) do also consider this trade-off in their data efﬁciency study, ultimately recommending several sampling and modeling techniques to attain production quality with fast, light-weight neural network models.”
TaTa: A Multilingual Table-to-Text Dataset for African Languages (2022, background): “The setup has many applications ranging from virtual assistants (Arun et al., 2020; Mehri et al., 2022) to the generation of news articles (Washington Post, 2020) or weather reports (Sripada et al., 2004).”
Building Adaptive Acceptability Classifiers for Neural NLG (2021, methodology): “We conducted experiments on 5 datasets: WebNLG2 (Gardent et al., 2017), Cleaned E2E (Dušek et al., 2019, 2020), ViGGO (Juraska et al., 2019), a Conversational Weather dataset created following the method described in Arun et al. (2020), and an Alarm dataset released by Arun et al. (2020).”