UnifiedQA: Crossing Format Boundaries With a Single QA System

By Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, Hannaneh Hajishirzi. 2020

827 citations recorded by the source corpus.

Question answering (QA) tasks have been posed using a variety of formats, such as extractive span selection, multiple choice, etc. This has led to format-specialized models, and even to an implicit division in the QA community. We argue that such boundaries are artificial and perhaps unnecessary, given the reasoning abilities we seek to teach are not governed by the format. As evidence, we use the latest advances in language modeling to build a single pre-trained QA model, UNIFIEDQA, that performs well across 19 QA datasets spanning 4 diverse formats. UNIFIEDQA performs on par with 8 different models that were trained on individual datasets themselves. Even when faced with 12 unseen datasets of observed formats, UNIFIEDQA performs surprisingly well, showing strong generalization from its outof-format training data. Finally, simply finetuning this pre trained QA model into specialized models results in a new state of the art on 10 factoid and commonsense question answering datasets, establishing UNIFIEDQA as a strong starting point for building QA systems.

Subfields: Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

Citations in the corpus

Few-shot Learning with Retrieval Augmented Language Models (2022, methodology): “, 2020) and UnifiedQA (Khashabi et al., 2020) from the MMLU leaderboard at https: //github.”
TruthfulQA: Measuring How Models Mimic Human Falsehoods (2021, methodology): “UnifiedQA (Khashabi et al., 2020) is a T5 model (Raffel et al., 2019) fine-tuned on diverse QA tasks.”
Measuring Massive Multitask Language Understanding (2020, background): “UniﬁedQA uses the T5 (Raffel et al., 2019) text-to-text backbone and is ﬁne-tuned on previously proposed question answering datasets (Lai et al., 2017), where the prediction is the class with the highest token overlap with UniﬁedQA’s text output.”
Self-Consistency Improves Chain of Thought Reasoning in Language Models (2022, methodology): “The previous SoTA baselines are obtained from: a : DeBERTaV3-large + KEAR (Xu et al., 2021b), b : Chowdhery et al. (2022), c : UniﬁedQA-FT (Khashabi et al., 2020).”
Finetuned Language Models Are Zero-Shot Learners (2021, methodology): “UnifiedQA (Khashabi et al., 2020) shows similar transfer learning gains as McCann et al.”
mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer (2020, methodology): “…for the “Text-to-Text Transfer Transformer” (T5) model released by Raﬀel et al. (2019) have been used to achieve state-of-the-art results on many benchmarks (Khashabi et al., 2020; Roberts et al., 2020; Kale, 2020; Izacard and Grave, 2020; Nogueira et al., 2020; Narang et al., 2020, etc.).”
OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization (2022, methodology): “This is achieved by unifying all the tasks into a common notion (Khashabi et al., 2020) via instructions, and training them together by sharing all the weights of the model across all tasks.”
Augmented Language Models: a Survey (2023, methodology): “After distilling knowledge from a larger GPT3 (Brown et al., 2020) model into a smaller T5 model (Raffel et al., 2020), RAINIER is finetuned using PPO (Schulman et al., 2017) with feedback provided by the pre-trained question answering model from Khashabi et al. (2020).”
Galactica: A Large Language Model for Science (2022, methodology): “And additionally there is the UnifiedQA approach (Khashabi et al., 2020).”
BBQ: A hand-built bias benchmark for question answering (2021, methodology): “Within the ambiguous contexts, UnifiedQA relies on social biases to different degrees in different categories, with biases related to physical appearance, age, and socio-economic status driving model response much moreso than biases related to race and sexual orientation.”