Towards Discriminability and Diversity: Batch Nuclear-Norm Maximization Under Label Insufficient Situations

By Shuhao Cui, Shuhui Wang, Junbao Zhuo, Liang Li, Qingming Huang, Qi Tian. 2020

443 citations recorded by the source corpus.

The learning of the deep networks largely relies on the data with human-annotated labels. In some label insufficient situations, the performance degrades on the decision boundary with high data density. A common solution is to directly minimize the Shannon Entropy, but the side effect caused by entropy minimization, \it i.e., reduction of the prediction diversity, is mostly ignored. To address this issue, we reinvestigate the structure of classification output matrix of a randomly selected data batch. We find by theoretical analysis that the prediction discriminability and diversity could be separately measured by the Frobenius-norm and rank of the batch output matrix. Besides, the nuclear-norm is an upperbound of the Frobenius-norm, and a convex approximation of the matrix rank. Accordingly, to improve both discriminability and diversity, we propose Batch Nuclear-norm Maximization (BNM) on the output matrix. BNM could boost the learning under typical label insufficient learning scenarios, such as semi-supervised learning, domain adaptation and open domain recognition. On these tasks, extensive experimental results show that BNM outperforms competitors and works well with existing well-known methods. The code is available at https://github.com/cuishuhao/BNM

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jian Liang
2. Jiashi Feng
3. Dapeng Hu
4. Ran He
5. Yunbo Wang
6. Qi Tian
7. Shuhao Cui
8. Chi Su

Citations in the corpus

• Margin-Based Few-Shot Class-Incremental Learning with Class-Level Overfitting Mitigation (2022, background): “For example, [7] studied this problem for adversarial domain adaptation, [8] studied this problem from the aspect of information-bottleneck theory, and [9] investigated this problem under the label insufficient situations.”
• Source Data-Absent Unsupervised Domain Adaptation Through Hypothesis Transfer and Labeling Transfer (2020, background): “For vanilla unsupervisedDA in digit recognition, we compare SHOT with ADDA [5], ADR [56], CDAN [42], CyCADA [8], CAT [98], SWD [99] and STAR [100]; for object recognition, we compare ours with DANN [24], DAN [4], SAFN [82], BSP [101], MDD [102], TransNorm [55], DSBN [25], BNM [103] and GVB-GD [104].”
• CDTrans: Cross-domain Transformer for Unsupervised Domain Adaptation (2021, unknown): “We compare our method with state-of-the-art methods on UDA tasks, including MinEnt (Grandvalet et al., 2005), DANN (Ganin & Lempitsky, 2015), CDAN+E (Long et al., 2018), CDAN+BSP (Chen et al., 2019), CDAN+TN (Wang et al., 2019), rRGrad+CAT (Deng et al., 2019), MCD (Saito et al., 2018), SWD (Lee et a”
• Gradually Vanishing Bridge for Adversarial Domain Adaptation (2020, background): “Visual domain adaptation, first proposed by [34], receives increasing attention and fruitful results [45, 43], especially with deep learning techniques [8, 9, 15, 18, 19, 6].”
• A Comprehensive Survey on Test-Time Adaptation Under Distribution Shifts (2023, unknown): “, inputlevel translation [28], [29], feature-level alignment [30], [31], [32]), output-level regularization [33], [34], [35], and prior estimation [17], [36], [37].”