Transfer Fine-Tuning: A BERT Case Study

By Yuki Arase, Junichi Tsujii. 2019

43 citations recorded by the source corpus.

A semantic equivalence assessment is defined as a task that assesses semantic equivalence in a sentence pair by binary judgment (i.e., paraphrase identification) or grading (i.e., semantic textual similarity measurement). It constitutes a set of tasks crucial for research on natural language understanding. Recently, BERT realized a breakthrough in sentence representation learning (Devlin et al., 2019), which is broadly transferable to various NLP tasks. While BERT’s performance improves by increasing its model size, the required computational power is an obstacle preventing practical applications from adopting the technology. Herein, we propose to inject phrasal paraphrase relations into BERT in order to generate suitable representations for semantic equivalence assessment instead of increasing the model size. Experiments on standard natural language understanding tasks confirm that our method effectively improves a smaller BERT model while maintaining the model size. The generated model exhibits superior performance compared to a larger BERT model on semantic equivalence assessment tasks. Furthermore, it achieves larger performance gains on tasks with limited training datasets for fine-tuning, which is a property desirable for transfer learning.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

GAN Memory with No Forgetting (2020, background): “Transfer learning Being general and effective, transfer learning has attracted increasing attention recently in various research ﬁelds, with a focus on discriminative tasks like classiﬁcation [6, 68, 88, 90, 16, 24, 44, 46, 72, 89] and those in natural language processing [3, 61, 53, 52, 2].”
DirectProbe: Studying Representations without Classifiers (2021, background): “Past work (Peters et al., 2019; Arase and Tsujii, 2019; Merchant et al., 2020) has shown that fine-tuning”
Efficient Intent Detection with Dual Sentence Encoders (2020, background): “, 2019b), and still requires sufficient taskspecific data (Arase and Tsujii, 2019; Sanh et al., 2019).”
A Primer in BERTology: What We Know About How BERT Works (2020, background): “, 2020; Arase and Tsujii, 2019; Pruksachatkun et al., 2020; Glavaš and Vulić, 2020). Ben-David et al. (2020) propose a pivot-based variant of MLM to fine-tune BERT for domain adaptation.”