How to Fine-Tune BERT for Text Classification?

By Chi Sun, Xipeng Qiu, Yige Xu, Xuanjing Huang. 2019

1,803 citations recorded by the source corpus.

Language model pre-training has proven to be useful in learning universal language representations. As a state-of-the-art language model pre-training model, BERT (Bidirectional Encoder Representations from Transformers) has achieved amazing results in many language understanding tasks. In this paper, we conduct exhaustive experiments to investigate different fine-tuning methods of BERT on text classification task and provide a general solution for BERT fine-tuning. Finally, the proposed solution obtains new state-of-the-art results on eight widely-studied text classification datasets.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference (2020, methodology): “To test this hypothesis, we simply further pretrain RoBERTa on in-domain data, a common technique for improving text classification accuracy (e.g., Howard and Ruder, 2018; Sun et al., 2019).”
BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generation (2021, background): “unfair outcomes and negative experiences to the end users of the downstream applications [11, 18, 20, 33, 34].”
CausaLM: Causal Model Explanation Through Counterfactual Language Models (2020, methodology): “Since the introduction of pre-trained word-embeddings, there has been an explosion of research on choosing pre-training tasks and understanding their effect (Jernite, Bowman, and Sontag 2017; Logeswaran and Lee 2018; Ziser and Reichart 2018; Dong et al. 2019; Chang et al. 2019; Sun et al. 2019; Rotm”
Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks (2020, background): “…cost effective, since we continue pretraining an already powerful LM. Task-adaptive pretraining Continued pretraining of a LM on the unlabeled data of a given task (TAPT) has been show to be beneficial for endtask performance (e.g. in Howard and Ruder, 2018; Phang et al., 2018; Sun et al., 2019).”
Revisiting Few-sample BERT Fine-tuning (2020, background): “[38] Chi Sun, Xipeng Qiu, Yige Xu, and Xuanjing Huang.”
Continual Training of Language Models for Few-Shot Learning (2022, background): “Researchers have applied post-training to many domains, e.g., reviews (Xu et al., 2019; Sun et al., 2019), news and academic papers (Gururangan et al., 2020b), and shown improved end-task results.”
TERA: Self-Supervised Learning of Transformer Encoder Representation for Speech (2020, background): “These approaches adapt the NLP pre-training technique to continuous speech.”
Pre-trained models for natural language processing: A survey (2020, background): “[161] showed that the “further pretraining” on the related-domain corpus can further improve the ability of BERT and achieved state-of-the-art performance on eight widely-studied text classification datasets.”
Deep Learning--based Text Classification (2020, methodology): “58 BERT-large [7, 149] 95.”
Bias in medical AI: Implications for clinical decision-making (2024, background): “Although recent LLMs have achieved unprecedented performance on NLP tasks in medicine and other domains [116–118], medical LLMs deployed for clinical decision-making pose unique limitations and risks with respect to bias [119–122].”
PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimization (2023, background): “Evaluation of Language Models Accurate evaluation of language models is crucial in determining optimal hyperparameters, thus improving the models’ overall performance [39, 38].”
Where to Go Next for Recommender Systems? ID- vs. Modality-based Recommender Models Revisited (2023, background): “Performing a second round of pre-training for ME using the downstream dataset often works well in much machine learning literature [16, 54].”
DNABERT-2: Efficient Foundation Model and Benchmark For Multi-Species Genome (2023, methodology): “Following Sun et al. (2020), we train the model with a batch size of 32, a maximum sequence length of 128, and a learning rate of 5 e - 5 for 100,000 steps.”
MeLL: Large-scale Extensible User Intent Classification for Dialogue Systems with Meta Lifelong Learning (2021, methodology): “• Lifelong-freeze: We first uses the multi-task fine-tuning approach [45] on the 𝑁 base tasks.”
FinBERT: Financial Sentiment Analysis with Pre-trained Language Models (2019, background): “One such recent work is Sun et al. (2019) [27].”
CoDA: Contrast-enhanced and Diversity-promoting Data Augmentation for Natural Language Understanding (2020, background): “, 2020) or require special finetuning techniques (Sun et al., 2019; Hao et al., 2019).”
ERNIE-Gram: Pre-Training with Explicitly N-Gram Masked Language Modeling for Natural Language Understanding (2020, methodology): “To inject n-gram information into pre-training, many works (Sun et al., 2019b; Cui et al., 2019; Joshi et al., 2020) extend BERT’s masked language modeling (MLM) from masking individual tokens to contiguous sequences of n tokens.”