QAmeleon: Multilingual QA with Only 5 Examples

By Priyanka Agrawal, Chris Alberti, Fantine Huot, Joshua Maynez, Ji Ma, Sebastian Ruder, Kuzman Ganchev, Dipanjan Das, Mirella Lapata. 2022

34 citations recorded by the source corpus.

Abstract The availability of large, high-quality datasets has been a major driver of recent progress in question answering (QA). Such annotated datasets, however, are difficult and costly to collect, and rarely exist in languages other than English, rendering QA technology inaccessible to underrepresented languages. An alternative to building large monolingual training datasets is to leverage pre-trained language models (PLMs) under a few-shot learning setting. Our approach, QAmeleon, uses a PLM to automatically generate multilingual data upon which QA models are fine-tuned, thus avoiding costly annotation. Prompt tuning the PLM with only five examples per language delivers accuracy superior to translation-based baselines; it bridges nearly 60% of the gap between an English-only baseline and a fully-supervised upper bound fine-tuned on almost 50,000 hand-labeled examples; and consistently leads to improvements compared to directly fine-tuning a QA model on labeled examples in low resource settings. Experiments on the TyDiqa-GoldP and MLQA benchmarks show that few-shot prompt tuning for data synthesis scales across languages and is a viable alternative to large-scale annotation.1

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Prompting & Reasoning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Thomas Scialom
2. Timo Schick
3. Yejin Choi
4. Or Honovich
5. Hinrich Schütze
6. Tomas Pfister
7. Omer Levy
8. Xianpei Han

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Dacheng Tao
3. Trevor Darrell
4. Andrew Zisserman
5. Xiaogang Wang
6. Mohit Bansal
7. Jitendra Malik
8. Cordelia Schmid

In Natural Language Processing

1. Thomas Scialom
2. Timo Schick
3. Or Honovich
4. Xianpei Han
5. Omer Levy
6. Hongyu Lin
7. Le Sun
8. Alexander J. Ratner

In Prompting & Reasoning

1. Or Honovich
2. Thomas Scialom
3. Timo Schick
4. Omer Levy
5. Tomas Pfister
6. Ann Yuan
7. Hinrich Schütze
8. Luke S. Zettlemoyer

Citations in the corpus

• Unnatural Instructions: Tuning Language Models with (Almost) No Human Labor (2022, methodology): “Agrawal et al. (2022) use pretrained language models to automatically construct multilingual QA data using only five examples per language.”
• Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes (2023, methodology): “It supports learning from limited labeled data, since the larger teacher model is often used to generate a training dataset with noisy pseudo labels (Chen et al., 2020; Il-iopoulos et al., 2022; Wang et al., 2021; Smith et al., 2022a; Arora et al., 2022; Agrawal et al., 2022).”
• ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Cases (2023, methodology): “LLMs for Data Generation Many research studies have employed LLMs for data generation, focusing on various tasks such as question answering (Wang et al. 2021; Agrawal et al. 2022; Chen, Chen, and tau Yih 2023), semantic similarity predictions (Schick and Schütze 2021), and instruction tuning (Honovi”
• Towards Agile Text Classifiers for Everyone (2023, background): “Recent work suggests that PET may also enable more data-efficient training of models (Liu et al., 2022; Agrawal et al., 2022).”