Getting to Production with Few-shot Natural Language Generation Models

By Peyman Heidari, Arash Einolghozati, Shashank Jain, Soumya Batra, Lee F. Callender, Ankit Arun, Shawn Mei, Sonal Gupta, Pinar E. Donmez, Vikas Bhardwaj and 2 more. 2021

17 citations recorded by the source corpus.

In this paper, we study the utilization of pre-trained language models to enable few-shotNatural Language Generation (NLG) in task-oriented dialog systems. We introduce a system consisting of iterative self-training and an extensible mini-template framework that textualizes the structured input data into semi-natural text to fully take advantage of pre-trained language models. We compare var-ious representations of NLG models’ input and output and show that transforming the input and output to be similar to what the language model has seen before during pre-training improves the model’s few-shot performance substantially. We show that neural mod-els can be trained with as few as 300 annotated examples while providing high fidelity, considerably lowering the resource requirements for standing up a new domain or language.This level of data efficiency removes the need for crowd-sourced data collection resulting in higher quality data annotated by expert linguists. In addition, model maintenance and debugging processes will improve in this few-shot setting. Finally, we explore distillation and using a caching system to satisfy latency requirements of real-world systems.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Natural Language Generation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Michael. White
2. Shawn Mei
3. Pinar E. Donmez
4. Anuj Kumar
5. Ankit Arun
6. Vikas Bhardwaj
7. Soumya Batra
8. Shashank Jain

In Computer Vision and Pattern Recognition

1. Michael. White
2. Pinar E. Donmez
3. Anuj Kumar
4. Vikas Bhardwaj

In Natural Language Processing

1. Soumya Batra
2. Ankit Arun
3. Shashank Jain
4. Peyman Heidari
5. Michael. White
6. Shawn Mei
7. Pinar E. Donmez
8. Anuj Kumar

Citations in the corpus

• Building Adaptive Acceptability Classifiers for Neural NLG (2021, methodology): “Following Heidari et al. (2021), we delexicalize the meaning representations of the WebNLG dataset and pair them with existing delexicalized responses.”