Similar image search for histopathology: SMILY

By Narayan Hegde, J. Hipp, Yun Liu, M. Emmert-Buck, Emily Reif, Daniel Smilkov, Michael Terry, Carrie J. Cai, M. Amin, Craig H. Mermel and 4 more. 2019

154 citations recorded by the source corpus.

The increasing availability of large institutional and public histopathology image datasets is enabling the searching of these datasets for diagnosis, research, and education. Although these datasets typically have associated metadata such as diagnosis or clinical notes, even carefully curated datasets rarely contain annotations of the location of regions of interest on each image. As pathology images are extremely large (up to 100,000 pixels in each dimension), further laborious visual search of each image may be needed to find the feature of interest. In this paper, we introduce a deep-learning-based reverse image search tool for histopathology images: Similar Medical Images Like Yours (SMILY). We assessed SMILY’s ability to retrieve search results in two ways: using pathologist-provided annotations, and via prospective studies where pathologists evaluated the quality of SMILY search results. As a negative control in the second evaluation, pathologists were blinded to whether search results were retrieved by SMILY or randomly. In both types of assessments, SMILY was able to retrieve search results with similar histologic features, organ site, and prostate cancer Gleason grade compared with the original query. SMILY may be a useful general-purpose tool in the pathologist’s arsenal, to improve the efficiency of searching large archives of histopathology images, without the need to develop and implement specific tools for each application.

Subfields: Artificial Intelligence, Biophysics, Computer Vision and Pattern Recognition, Medical Imaging

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Carrie J. Cai
2. Michael Terry
3. Tongshuang Sherry Wu
4. Lauren Wilcox
5. Samantha Winter
6. Rodney Hull
7. F. Francies
8. David F. Steiner

In Computer Vision and Pattern Recognition

1. Lauren Wilcox
2. Michael Terry
3. Carrie J. Cai
4. Tongshuang Sherry Wu

Citations in the corpus

• AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model Prompts (2021, background): “Contrary to recent work in human-AI interaction, which primarily examined how to increase AI controllability through exposing knobs within a model [44, 49], our work opens up the possibility of steering AI using the model itself as units to control.”
• Artificial intelligence (AI) and big data in cancer and precision oncology (2020, methodology): “This algorithm called SMILY (Similar image search for histopathology) uses database of unla-belled images to find similar images [70].”
• "Hello AI": Uncovering the Onboarding Needs of Medical Practitioners for Human-AI Collaborative Decision-Making (2019, background): “Prostate cancer is one of the most common types of cancer, and the subject of significant attention in recent work on machine learning-powered cancer diagnosis [11, 15, 35, 56, 57, 69].”