Pre-training Is (Almost) All You Need: An Application to Commonsense Reasoning

By Alexandre Tamborrino, Nicola Pellicanò, Baptiste Pannier, Pascal Voitot, Louise Naudin. 2020

67 citations recorded by the source corpus.

Fine-tuning of pre-trained transformer models has become the standard approach for solving common NLP tasks. Most of the existing approaches rely on a randomly initialized classifier on top of such networks. We argue that this fine-tuning procedure is sub-optimal as the pre-trained model has no prior on the specific classifier labels, while it might have already learned an intrinsic textual representation of the task. In this paper, we introduce a new scoring method that casts a plausibility ranking task in a full-text format and leverages the masked language modeling head tuned during the pre-training phase. We study commonsense reasoning tasks where the model must rank a set of hypotheses given a premise, focusing on the COPA, Swag, HellaSwag and CommonsenseQA datasets. By exploiting our scoring method without fine-tuning, we are able to produce strong baselines (e.g. 80% test accuracy on COPA) that are comparable to supervised approaches. Moreover, when fine-tuning directly on the proposed scoring function, we show that our method provides a much more stable training phase across random restarts (e.g x10 standard deviation reduction on COPA test accuracy) and requires less annotated data than the standard classifier approach to reach equivalent performances.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Large Language Models, Prompting & Reasoning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Large Language Models

In Prompting & Reasoning

Citations in the corpus

What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization? (2022, background): “However, this architecture is limited to producing the same number of tokens as it was fed as input, considerably limiting its applicability and making it only rarely used in the zero-shot setting [Tamborrino et al., 2020].”
Relational World Knowledge Representation in Contextual Language Models: A Review (2021, methodology): “Both categories have shown promise over comparable unsupervised (and, under some conditions, supervised) methods for tasks like multiple-choice QA (Tamborrino et al., 2020; Shwartz et al., 2020; Brown et al., 2020) and commonsense KB completion (Davison et al., 2019).”
Designing Templates for Eliciting Commonsense Knowledge from Pretrained Sequence-to-Sequence Models (2020, methodology): “Tamborrino et al. (2020) further present a hybrid method that combines full-text inputs, pretrained token representations, and learning-to-rank objectives.”
COMET-ATOMIC 2020: On Symbolic and Neural Commonsense Knowledge Graphs (2020, background): “Despite the conclusions of prior work (Petroni et al. 2019; Roberts, Raffel, and Shazeer 2020; Tamborrino et al. 2020), our results in Table 6 are clear that language models fail to express large varieties of knowledge when prompted for it in a zero-shot manner.”