Classifying and segmenting microscopy images with deep multiple instance learning

By Oren Z. Kraus, Jimmy Ba, Brendan J. Frey. 2015

422 citations recorded by the source corpus.

Motivation: High-content screening (HCS) technologies have enabled large scale imaging experiments for studying cell biology and for drug screening. These systems produce hundreds of thousands of microscopy images per day and their utility depends on automated image analysis. Recently, deep learning approaches that learn feature representations directly from pixel intensity values have dominated object recognition challenges. These tasks typically have a single centered object per image and existing models are not directly applicable to microscopy datasets. Here we develop an approach that combines deep convolutional neural networks (CNNs) with multiple instance learning (MIL) in order to classify and segment microscopy images using only whole image level annotations. Results: We introduce a new neural network architecture that uses MIL to simultaneously classify and segment microscopy images with populations of cells. We base our approach on the similarity between the aggregation function used in MIL and pooling layers used in CNNs. To facilitate aggregating across large numbers of instances in CNN feature maps we present the Noisy-AND pooling function, a new MIL operator that is robust to outliers. Combining CNNs with MIL enables training CNNs using whole microscopy images with image level labels. We show that training end-to-end MIL CNNs outperforms several previous methods on both mammalian and yeast datasets without requiring any segmentation steps. Availability and implementation: Torch7 implementation available upon request. Contact: oren.kraus@mail.utoronto.ca

Subfields: Artificial Intelligence, Biophysics, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Biophysics

In Computer Vision and Pattern Recognition

Citations in the corpus

Unmasking Clever Hans predictors and assessing what machines really learn (2019, methodology): “Explanation methods were successfully applied to a wide set of complex real-world problems such as the analysis of faces [81, 123, 124], EEG and fMRI data [23, 50], human gait [48], videos [125], speech [126], reinforcement learning [25], biological data [127], text [26,128,129], or comparing human ”
Attention-based Deep Multiple Instance Learning (2018, neutral): “, 2017) and microscopy cell detection (Kraus et al., 2016).”
A Theoretical Explanation for Perplexing Behaviors of Backpropagation-based Visualizations (2018, background): “This class of backpropagation-based visualizations, in particular GBP and DeconvNet, has attracted a lot of attention in both the deep learning community and other fields (Szegedy et al., 2013; Dosovitskiy & Brox, 2016; Selvaraju et al., 2016; Fong & Vedaldi, 2017; Kraus et al., 2016).”
CytoGAN: Generative Modeling of Cell Images (2017, background): “Other related work on this dataset include supervised classiﬁcation [10], transfer learning on CellProﬁler features [9], and dimensionality reduction using autoencoders [19].”
Applications of Deep Learning and Reinforcement Learning to Biological Data (2017, methodology): “integrated both the segmentation as well as classification in a model which can be utilized to classify the microscopy images of the yeast [113].”
Automating Morphological Profiling with Generic Deep Convolutional Networks (2016, methodology): “Kraus et al. [6] evaluated their method on the same BBBC021 image set as used in this paper.”
Data-analysis strategies for image-based cell profiling (2017, background): “It is an exciting time for the field of image-based cell profiling, as methods are rapidly evolving and applications leading to major biological discoveries are beginning to be published.”
Machine learning and computer vision approaches for phenotypic profiling (2017, methodology): “Deep learning has also been used to classify protein localization in yeast and mechanisms of action in a publicly available drug screen (Kraus et al., 2016; Pärnamaa and Parts, 2016 Preprint ).”
State-of-the-art review on deep learning in medical imaging. (2019, background): “(2016) [52] Breast cancer (340) Yeast (2480) Cellular Segmentation CNN+MIL Jacobian Maps + loopy belief propagation Breast cancer Acc: 0.”
Explaining Deep Neural Networks and Beyond: A Review of Methods and Applications (2021, background): “Further applications of DNN explanation techniques for medical images include the detection of lesions in diabetic retinopathy data [145], pixelwise analysis of microscopy images from global image annotations [98], the validation of predictions in dermatology [199], and analysis of X-ray images [48]”
Deep learning for cellular image analysis (2019, methodology): “Deep learning has also been used to classify spatial patterns in fluorescence images and to determine protein localization in large datasets from yeast 82–84 and humans 57 .”
Toward Interpretable Machine Learning: Transparent Deep Neural Networks and Beyond (2020, background): “For instance, [79] use deep multiple instance learning to classify and segment microscopy images using only whole image level annotations.”
Multiple Instance Learning for Heterogeneous Images: Training a CNN for Histopathology (2018, methodology): “Even with a small number of labeled samples, our model was successful in ﬁne-tuning the AlexNet CNN because of the large size of the images providing plenty of opportunity for MI augmentation.”