Multimodal Co-Attention Transformer for Survival Prediction in Gigapixel Whole Slide Images

By Richard J. Chen, Ming Y. Lu, Wei-Hung Weng, Tiffany Y. Chen, Drew F. K. Williamson, Trevor Manz, Maha Shady, Faisal Mahmood. IEEE International Conference on Computer Vision, 2021

406 citations recorded by the source corpus.

Survival outcome prediction is a challenging weakly-supervised and ordinal regression task in computational pathology that involves modeling complex interactions within the tumor microenvironment in gigapixel whole slide images (WSIs). Despite recent progress in formulating WSIs as bags for multiple instance learning (MIL), representation learning of entire WSIs remains an open and challenging problem, especially in overcoming: 1) the computational complexity of feature aggregation in large bags, and 2) the data heterogeneity gap in incorporating biological priors such as genomic measurements. In this work, we present a Multimodal Co-Attention Transformer (MCAT) framework that learns an interpretable, dense co-attention mapping between WSIs and genomic features formulated in an embedding space. Inspired by approaches in Visual Question Answering (VQA) that can attribute how word embed-dings attend to salient objects in an image when answering a question, MCAT learns how histology patches attend to genes when predicting patient survival. In addition to visualizing multimodal interactions, our co-attention trans-formation also reduces the space complexity of WSI bags, which enables the adaptation of Transformer layers as a general encoder backbone in MIL. We apply our proposed method on five different cancer datasets (4,730 WSIs, 67 million patches). Our experimental results demonstrate that the proposed method consistently achieves superior performance compared to the state-of-the-art methods.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging

View Paper: https://www.semanticscholar.org/paper/5d1aadd4053ba4fdcd71fe1c87a53b7540ef4eea

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Radiology, Nuclear Medicine and Imaging

Citations in the corpus

Artificial intelligence for digital and computational pathology (2023, background): “P a t c h i m p o r t a n c e”
Multimodal Learning With Transformers: A Survey (2022, methodology): “SQL database schema table node, column node GNN embedding SpeechSQLNet [25] textual question-graph word node GNN embedding SADGA [24] sketch key point of stroke linear projection and learnable embedding Multi-Graph Transformer [163] sketch patch of picture linear projection RVT [182] table cell lear”
Scaling Vision Transformers to Gigapixel Images via Hierarchical Self-Supervised Learning (2022, background): “The difference is pronounced in context-aware tasks such as survival prediction in which larger context is appreciated in characterizing broader prognostic features in the tissue microenvironment [1, 17, 60, 63].”