Dice Loss for Data-imbalanced NLP Tasks

By Xiaoya Li, Xiaofei Sun, Yuxian Meng, Junjun Liang, Fei Wu, Jiwei Li. Annual Meeting of the Association for Computational Linguistics, 2019

765 citations recorded by the source corpus.

Many NLP tasks such as tagging and machine reading comprehension are faced with the severe data imbalance issue: negative examples significantly outnumber positive examples, and the huge number of easy-negative examples overwhelms the training. The most commonly used cross entropy (CE) criteria is actually an accuracy-oriented objective, and thus creates a discrepancy between training and test: at training time, each training instance contributes equally to the objective function, while at test time F1 score concerns more about positive examples. In this paper, we propose to use dice loss in replacement of the standard cross-entropy objective for data-imbalanced NLP tasks. Dice loss is based on the Sørensen--Dice coefficient or Tversky index , which attaches similar importance to false positives and false negatives, and is more immune to the data-imbalance issue. To further alleviate the dominating influence from easy-negative examples in training, we propose to associate training examples with dynamically adjusted weights to deemphasize easy-negative examples. Theoretical analysis shows that this strategy narrows down the gap between the F1 score in evaluation and the dice loss in training. With the proposed training objective, we observe significant performance boost on a wide range of data imbalanced NLP tasks. Notably, we are able to achieve SOTA results on CTB5, CTB6 and UD1.4 for the part of speech tagging task; SOTA results on CoNLL03, OntoNotes5.0, MSRA and OntoNotes4.0 for the named entity recognition task; along with competitive results on the tasks of machine reading comprehension and paraphrase identification.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/5487dadb5b4b8b240ab4ae28705acc0b9f138db0

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Cross-City Matters: A Multimodal Remote Sensing Benchmark Dataset for Cross-City Semantic Segmentation using High-Resolution Domain Adaptation Networks (2023, background): “Embedding Dice loss [58] into networks, HighDAN is capable of weakening the class imbalance effects that tend to be generated in the case of cross-city image interpretation, e.g., semantic segmentation.”
DeepStruct: Pretraining of Language Models for Structure Prediction (2022, methodology): “Comparison Methods We compare our method D EEP S TRUCT on the four datasets to the following NER methods: (i) BERT-MRC (Li et al., 2020a): this method formulates NER as a machine reading comprehension problem, (ii) BERT-MRC+DSC (Li et al., 2020b): this model is a dice-loss enhanced version of BERT-M”
A Survey on Deep Learning for Named Entity Recognition (2018, methodology): “The work with BERT and dice loss [138] achieves the state-of-the-art performance (92.”
GPT-NER: Named Entity Recognition via Large Language Models (2023, methodology): “• MRC-NER+DSC (Li et al., 2019b) is the current SOTA model on the OntoNotes5.”
AnomalyCLIP: Object-agnostic Prompt Learning for Zero-shot Anomaly Detection (2023, neutral): “Let S ∈ R H image × W image be the ground-truth segmentation mask, with S jk = 1 if the pixel is as an anomaly and S jk = 0 otherwise, then we have where Focal ( · , · ) and Dice ( · , · ) denote a focal loss (Lin et al., 2017) and a Dice loss (Li et al., 2019) respectively.”
Pick and Choose: A GNN-based Imbalanced Learning Approach for Fraud Detection (2021, background): “Re-weighting methods assign different weights to different classes or even different samples by cost-sensitive adjustments [4, 9, 19, 21] or meta-learning based methods [14, 35, 37].”
Structured Prediction as Translation between Augmented Natural Languages (2021, methodology): “In this line of work, BERT MRC (Li et al., 2019a) performs NER using two classiﬁcation modules to predict respectively the ﬁrst and the last tokens corresponding to an entity for a given input sentence.”
Bio-Megatron: Larger Biomedical Domain Language Model (2020, background): “Recently, Li et al. (2019) proposed dice-loss for data-imbalance issues in NLP, with SOTA results on NER and QA, which could be a future avenue to explore for domain LMs. Transfer learning showed effectiveness in the biomedical QA task.”
Zero-Shot Information Extraction via Chatting with ChatGPT (2023, result): “For the ACE05 dataset, the predicted argument results are matched with the manually marked argument results at the entity level and evaluated by the micro F1.”