DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing

By Pengcheng He, Jianfeng Gao, Weizhu Chen. 2021

1,986 citations recorded by the source corpus.

This paper presents a new pre-trained language model, DeBERTaV3, which improves the original DeBERTa model by replacing mask language modeling (MLM) with replaced token detection (RTD), a more sample-efficient pre-training task. Our analysis shows that vanilla embedding sharing in ELECTRA hurts training efficiency and model performance. This is because the training losses of the discriminator and the generator pull token embeddings in different directions, creating the"tug-of-war"dynamics. We thus propose a new gradient-disentangled embedding sharing method that avoids the tug-of-war dynamics, improving both training efficiency and the quality of the pre-trained model. We have pre-trained DeBERTaV3 using the same settings as DeBERTa to demonstrate its exceptional performance on a wide range of downstream natural language understanding (NLU) tasks. Taking the GLUE benchmark with eight tasks as an example, the DeBERTaV3 Large model achieves a 91.37% average score, which is 1.37% over DeBERTa and 1.91% over ELECTRA, setting a new state-of-the-art (SOTA) among the models with a similar structure. Furthermore, we have pre-trained a multi-lingual model mDeBERTa and observed a larger improvement over strong baselines compared to English models. For example, the mDeBERTa Base achieves a 79.8% zero-shot cross-lingual accuracy on XNLI and a 3.6% improvement over XLM-R Base, creating a new SOTA on this benchmark. We have made our pre-trained models and inference code publicly available at https://github.com/microsoft/DeBERTa.

Subfields: Artificial Intelligence, Large Language Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

Citations in the corpus

BBQ: A hand-built bias benchmark for question answering (2021, background): “, 2019), and DeBERTaV3 (He et al., 2021) models on BBQ and find that in under-informative contexts, the models generally select unsupported answers rather than answers that express uncertainty, often in ways that align with social biases.”
Adversarial Training for High-Stakes Reliability (2022, methodology): “We trained a classiﬁer by ﬁne-tuning deberta-v3-large [44] from HuggingFace [45].”
Human Parity on CommonsenseQA: Augmenting Self-Attention with External Attention (2021, methodology): “DeBERTaV3-large (He et al., 2021) is the base encoder model.”
LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusion (2023, methodology): “We use the DeBERTa (He et al., 2021) (400m) as the backbone for P AIR R ANKER , and G EN F USER is based on Flan-T5-XL (3b).”
SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models (2023, methodology): “Similar to the grey-box approach, we can also use the maximum of the negative log probabilities, For SelfCheck-NLI, we use DeBERTa-v3-large (He et al., 2023) fine-tuned to MNLI as the NLI model.”
ChatGPT: Jack of all trades, master of none (2023, background): “8%) 638 2 319 /319 8 TextEntail S EN Natural language inference No Binary sentence pair classification RTE [73] / [72] 3 Yes 277 277 0 0 (0%) 277 2 146 /131 9 WNLI S EN Natural language inference No Binary sentence pair classification WNLI [74] / [75] 3 Yes 71 71 0 0 (0%) 71 2 40/31 10 SQuAD S EN Qu”
ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems (2023, methodology): “We compare RAGAS and the few-shot judge against ARES, which as discussed before uses synthetically fine-tuned LLM (DeBERTa-v3-Large).”
PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models (2024, neutral): “We also evaluate PiSSA’s natural language understanding (NLU) capability on the GLUE benchmark [60] with DeBERTa-v3-base [61].”
AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning (2023, methodology): “We evaluate the fine-tuning performance of DeBERTaV3-base (He et al., 2021a) using the proposed algorithm.”
ChatGPT in Healthcare: A Taxonomy and Systematic Review (2023, background): “consultation [42],[66],[20],[41] clinical workflow [69],[68],[4],[54]”
Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructions (2023, methodology): “We use this dataset to evaluate the effect of safety tuning on general model capabilities.”
Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference (2024, neutral): “As mentioned in Section 3.1.1, we follow standard practice (Liu et al., 2019a; Portes et al., 2023; He et al., 2023) and conduct an hyperparameter search on each subset.”
Dataless Knowledge Fusion by Merging Weights of Language Models (2022, methodology): “The dominant paradigm for solving NLP tasks ranging from classification to sequence tagging involves fine-tuning a pretrained language model (PLM) using task-specific labeled data (Devlin et al., 2019; He et al., 2021).”
Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning (2023, positive): “We evaluate the fine-tuning performance of DeBERTaV3-base (He et al., 2021a) using the proposed algorithm.”
CONDAQA: A Contrastive Reading Comprehension Dataset for Reasoning about Negation (2022, methodology): “In our experiments, we use DeBERTa-v2-XLarge and DeBERTa-v3-Large.”
LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders (2024, methodology): “We compare the LLM2Vec-transformed models to DeBERTa-v3-large (He et al., 2023), the current state-of-the-art encoder-only model.”
Large Language Model as Attributed Training Data Generator: A Tale of Diversity and Bias (2023, background): “They are TinyBERT [19], DistillBERT [45], DeBERTa-V3-base , and DeBERTa-V3-Large [18] with parameter size from 14M to 435M 8 .”
ProtoSiTex: Learning Semi-Interpretable Prototypes for Multi-label Text Classification (2025, unknown): “1) Comparison with Black-box Models: As observed from Table III, compared with traditional, graph-based, and advanced black-box models, ProtoSiTex demonstrates consistent performance improvements, highlighting the limitations of static embeddings, recurrent encoders, and global representation…”
BITEM at the NTCIR-19 R2C2 Task: Predicting Confidence from Agentic RAG Pipeline Signals (2026, unknown): “…then asks whether the span really says what the claim says, reading the pair with MiniCheck-Flan-T5-Large and a DeBERTa-v3-large entailment model [2]. list_nuggets lists what has been recorded, and flag_ nugget_bogus was available to retract a nugget that later reading contradicts, though no…”
Latent Trajectory Discrimination for AI-Generated Text Detection (2026, unknown): “Most supervised methods rely on encoder-only transformer architectures, such as RoBERTa [8] or DeBERTaV3 [12], which are fine-tuned with a sequence-level classification head.”