Generative Negative Text Replay for Continual Vision-Language Pretraining

By Shipeng Yan, Lanqing Hong, Hang Xu, Jianhua Han, Tinne Tuytelaars, Zhenguo Li, Xuming He. 2022

30 citations recorded by the source corpus.

Vision-language pre-training (VLP) has attracted increasing attention recently. With a large amount of image-text pairs, VLP models trained with contrastive loss have achieved impressive performance in various tasks, especially the zero-shot generalization on downstream datasets. In practical applications, however, massive data are usually collected in a streaming fashion, requiring VLP models to continuously integrate novel knowledge from incoming data and retain learned knowledge. In this work, we focus on learning a VLP model with sequential chunks of image-text pair data. To tackle the catastrophic forgetting issue in this multi-modal continual learning setting, we first introduce pseudo text replay that generates hard negative texts conditioned on the training images in memory, which not only better preserves learned knowledge but also improves the diversity of negative samples in the contrastive loss. Moreover, we propose multi-modal knowledge distillation between images and texts to align the instance-wise prediction between old and new models. We incrementally pre-train our model on both the instance and class incremental splits of the Conceptual Caption dataset, and evaluate the model on zero-shot image classification and image-text retrieval tasks. Our method consistently outperforms the existing baselines with a large margin, which demonstrates its superiority. Notably, we realize an average performance boost of $4.60\%$ on image-classification downstream datasets for the class incremental split.

Subfields: Artificial Intelligence, Cancer Research, Computer Vision and Pattern Recognition, Multimodal Learning, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Han-Jia Ye
  2. De-chuan Zhan
  3. Da-Wei Zhou
  4. Hang Su
  5. Qiwen Wang
  6. Jun Zhu
  7. Ziwei Liu
  8. Liyuan Wang

In Cancer Research

  1. Jun Zhu
  2. De-chuan Zhan
  3. Han-Jia Ye

In Computer Vision and Pattern Recognition

  1. Han-Jia Ye
  2. De-chuan Zhan
  3. Jun Zhu
  4. Da-Wei Zhou
  5. Hang Su
  6. Liyuan Wang
  7. Ziwei Liu
  8. Qiwen Wang

In Multimodal Learning

  1. Jun Zhu
  2. Hang Su
  3. Ziwei Liu
  4. Xingxing Zhang

In Meta & Continual Learning

  1. Han-Jia Ye
  2. De-chuan Zhan
  3. Da-Wei Zhou
  4. Xingxing Zhang
  5. Liyuan Wang
  6. Hang Su
  7. Qiwen Wang
  8. Jun Zhu

Citations in the corpus