Efficient Test-Time Model Adaptation without Forgetting

By Shuaicheng Niu, Jiaxiang Wu, Yifan Zhang, Yaofo Chen, S. Zheng, Peilin Zhao, Mingkui Tan. 2022

647 citations recorded by the source corpus.

Test-time adaptation (TTA) seeks to tackle potential distribution shifts between training and testing data by adapting a given model w.r.t. any testing sample. This task is particularly important for deep models when the test environment changes frequently. Although some recent attempts have been made to handle this task, we still face two practical challenges: 1) existing methods have to perform backward computation for each test sample, resulting in unbearable prediction cost to many applications; 2) while existing TTA solutions can significantly improve the test performance on out-of-distribution data, they often suffer from severe performance degradation on in-distribution data after TTA (known as catastrophic forgetting). In this paper, we point out that not all the test samples contribute equally to model adaptation, and high-entropy ones may lead to noisy gradients that could disrupt the model. Motivated by this, we propose an active sample selection criterion to identify reliable and non-redundant samples, on which the model is updated to minimize the entropy loss for test-time adaptation. Furthermore, to alleviate the forgetting issue, we introduce a Fisher regularizer to constrain important model parameters from drastic changes, where the Fisher importance is estimated from test samples with generated pseudo labels. Extensive experiments on CIFAR-10-C, ImageNet-C, and ImageNet-R verify the effectiveness of our proposed method.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Mingkui Tan
2. Shuaicheng Niu
3. Yifan Zhang
4. Chuang Gan
5. Hongbin Lin
6. Zhen Qiu
7. Yanxia Liu
8. Song Han

In Computer Vision and Pattern Recognition

1. Mingkui Tan
2. Shuaicheng Niu
3. Yifan Zhang
4. Chuang Gan
5. Hongbin Lin
6. Zhen Qiu
7. Yanxia Liu
8. Luc Van Gool

In Radiology, Nuclear Medicine and Imaging

1. Shuaicheng Niu
2. Chuang Gan
3. Yifan Zhang
4. Mingkui Tan

Citations in the corpus

• Prototype-Guided Continual Adaptation for Class-Incremental Unsupervised Domain Adaptation (2022, background): “Step 4 [30,31,32,33,34,35,36,37,38,39] Sneakers, Notebook, Backpack, Alarm Clock, Push Pin, Paper Clip, Batteries, Radio, Fan, Ruler”
• A Comprehensive Survey on Test-Time Adaptation Under Distribution Shifts (2023, neutral): “Previous studies [483], [484] find that the model optimized by TTA methods suffers from severe performance degradation (named forgetting) on original training samples.”