Scaling Vision Transformers to Gigapixel Images via Hierarchical Self-Supervised Learning

By Richard J. Chen, Chengkuan Chen, Yicong Li, Tiffany Y. Chen, Andrew D. Trister, Rahul G. Krishnan, Faisal Mahmood. Computer Vision and Pattern Recognition, 2022

716 citations recorded by the source corpus.

Vision Transformers (ViTs) and their multi-scale and hierarchical variations have been successful at capturing image representations but their use has been generally studied for low-resolution images (e.g. 256 × 256, 384 × 384). For gigapixel whole-slide imaging (WSI) in computational pathology, WSIs can be as large as 150000 × 150000 pixels at 20 × magnification and exhibit a hierarchical structure of visual tokens across varying resolutions: from 16 × 16 images capturing individual cells, to 4096 × 4096 images characterizing interactions within the tissue microenvironment. We introduce a new ViT architecture called the Hierarchical Image Pyramid Transformer (HIPT), which leverages the natural hierarchical structure inherent in WSIs using two levels of self-supervised learning to learn high-resolution image representations. HIPT is pretrained across 33 cancer types using 10,678 gigapixel WSIs, 408,218 4096 × 4096 images, and 104M 256 × 256 images. We benchmark HIPT representations on 9 slide-level tasks, and demonstrate that: 1) HIPT with hierarchical pretraining outperforms current state-of-the-art methods for cancer subtyping and survival prediction, 2) self-supervised ViTs are able to model important inductive biases about the hierarchical structure of phenotypes in the tumor microenvironment.

Subfields: Artificial Intelligence, Biophysics, Radiology, Nuclear Medicine and Imaging

View Paper: https://www.semanticscholar.org/paper/68cda2cfefe8c21dc64fee55deab87672a517d39

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Biophysics

In Radiology, Nuclear Medicine and Imaging

Citations in the corpus

A foundation model for clinical-grade computational pathology and rare cancers detection (2024, positive): “Now self-supervised learning is enabling in-domain transfer by removing the label requirement, driving a second wave of scaling to tens of thousands of WSIs to inform image representation 35–39 , 56 .”
Artificial intelligence for digital and computational pathology (2023, background): “Segmentation is crucial for assisting clinical diagnosis by objectively”
CellViT: Vision Transformers for Precise Cell Segmentation and Classification (2023, methodology): “Our solution demonstrates exceptional performance on the PanNuke [17] dataset by leveraging transfer learning and pre-trained models [18, 19].”
A whole-slide foundation model for digital pathology from real-world data (2024, neutral): “Following the default settings in HIPT, we trained the comparison models with a learning rate of 2 × 10 −4 .”
V*: Guided Visual Search as a Core Mechanism in Multimodal LLMs (2023, methodology): “This brute-force strategy is often used in aerial image detection and whole slide image analysis [6, 36].”
FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision (2024, neutral): “…new capabilities (modeling and reasoning over multiple long documents [24, 43, 49] and ﬁles in large codebases [30, 47]), new modalities (high-resolution images [10], audio [23], video [25]), and new applications (user interaction with long history [52], agent workﬂow with long horizon [61]).”