Class-Incremental Domain Adaptation with Smoothing and Calibration for Surgical Report Generation

By Mengya Xu, Mobarak Islam Hoque, Chwee Ming Lim, Hongliang Ren. 2021

35 citations recorded by the source corpus.

Generating surgical reports aimed at surgical scene understanding in robot-assisted surgery can contribute to documenting entry tasks and post-operative analysis. Despite the impressive outcome, the deep learning model degrades the performance when applied to different domains encountering domain shifts. In addition, there are new instruments and variations in surgical tissues appeared in robotic surgery. In this work, we propose class-incremental domain adaptation (CIDA) with a multi-layer transformer-based model to tackle the new classes and domain shift in the target domain to generate surgical reports during robotic surgery. To adapt incremental classes and extract domain invariant features, a class-incremental (CI) learning method with supervised contrastive (SupCon) loss is incorporated with a feature extractor. To generate caption from the extracted feature, curriculum by one-dimensional gaussian smoothing (CBS) is integrated with a multi-layer transformer-based caption prediction model. CBS smoothes the features embedding using anti-aliasing and helps the model to learn domain invariant features. We also adopt label smoothing (LS) to calibrate prediction probability and obtain better feature representation with both feature extractor and captioning model. The proposed techniques are empirically evaluated by using the datasets of two surgical domains, such as nephrectomy operations and transoral robotic surgery. We observe that domain invariant feature learning and the well-calibrated network improves the surgical report generation performance in both source and target domain under domain shift and unseen classes in the manners of one-shot and few-shot learning. The code is publicly available at https://github.com/XuMengyaAmy/CIDACaptioning.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Mingkui Tan
2. Shipeng Yan
3. Shuaicheng Niu
4. Yifan Zhang
5. Xuming He
6. Chuang Gan
7. Hongbin Lin
8. Zhen Qiu

In Computer Vision and Pattern Recognition

1. Mingkui Tan
2. Chuang Gan
3. Shipeng Yan
4. Yifan Zhang
5. Shuaicheng Niu
6. Xuming He
7. Jiangwei Xie
8. Luc Van Gool

Citations in the corpus

• Prototype-Guided Continual Adaptation for Class-Incremental Unsupervised Domain Adaptation (2022, methodology): “However, in practice, target data often come in a streaming manner with different categories [21,31,50].”
• General Incremental Learning with Domain-aware Categorical Representations (2022, background): “Several recent studies tackle the class-incremental domain adaptation problem [17, 36], which aims at adapting the model trained on a source domain to a target domain including novel classes.”
• Class-Incremental Learning: A Survey (2023, unknown): “…[239], [240], [241], object detection [137], [242], [243], [244], [245], [246], [247], video understanding [91], [161], medical surgery [248], [249], language model [250], [251], [252], [253], language-vision model [217], [254], etc. Specifically, object detection and semantic segmentation…”
• Deep Class-Incremental Learning: A Survey (2023, unknown): “, semantic segmentation [164], [165], [166], [167], [174], object detection [175], [176], [177], [178], [179], [180], [181], video understanding [95], [133], medical surgery [182], [183], language model [184], [185], [186], [187], language-vision model [188], [189], etc.”