Automatic Instrument Segmentation in Robot-Assisted Surgery Using Deep Learning

By Alexey A. Shvets, Alexander Rakhlin, Alexandr A Kalinin, Vladimir I. Iglovikov. 2018

388 citations recorded by the source corpus.

Semantic segmentation of robotic instruments is an important problem for the robot-assisted surgery. One of the main challenges is to correctly detect an instrument’s position for the tracking and pose estimation in the vicinity of surgical scenes. Accurate pixel-wise instrument segmentation is needed to address this challenge. In this paper we describe our deep learning-based approach for robotic instrument segmentation. Our approach demonstrates an improvement over the state-of-the-art results using several novel deep neural network architectures. It addressed the binary segmentation problem, where every pixel in an image is labeled as an instrument or background from the surgery video feed. In addition, we solve a multi-class segmentation problem, in which we distinguish between different instruments or different parts of an instrument from the background. In this setting, our approach outperforms other methods for automatic instrument segmentation thereby providing state-of-the-art results for these problems. The source code for our solution is made publicly available.

Subfields: Aerospace Engineering, Biomedical Engineering, Surgery

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

1. Vladimir I. Iglovikov
2. Alexey A. Shvets
3. Alexander V. Buslaev
4. In-So Kweon

In Biomedical Engineering

1. Vladimir I. Iglovikov
2. Alexey A. Shvets
3. Alexander V. Buslaev
4. In-So Kweon

In Surgery

1. Vladimir I. Iglovikov
2. Alexey A. Shvets
3. Alexander V. Buslaev

Citations in the corpus

• Albumentations: fast and flexible image augmentations (2018, background): “In medical computer vision tasks that deal with color images or videos, color transformations have also been shown to help deep networks to generalize better, for example, such for surgical [16] or endoscopic [17] video analysis.”
• U-NetPlus: A Modified Encoder-Decoder U-Net Architecture for Semantic and Instance Segmentation of Surgical Instruments from Laparoscopic Images (2019, neutral): “1) over the state-of-the-art TernausNet framework [24].”
• PIDNet: A Real-time Semantic Segmentation Network Inspired by PID Controllers (2022, background): “Even though these models achieve encouraging segmentation accuracy, too much computational cost are required, which significantly hinder their application in real-time scenarios, such as autonomous vehicle [16] and robot surgery [44].”
• Angiodysplasia Detection and Localization Using Deep Convolutional Neural Networks (2018, methodology): “…In this work we evaluate 4 different deep architectures for segmentation: U-Net (Ronneberger et al., 2015; Iglovikov et al., 2017a), 2 modifications of TernausNet (Iglovikov and Shvets, 2018), and AlbuNet34, a modifications of LinkedNet (Chaurasia and Culurciello, 2017; Shvets et al., 2018).”
• Camera Model Identification Using Convolutional Neural Networks (2018, background): “Problems in satellite [1], [2], [3], [4], medical [5], [6], [7], [8] or any other imagery are successfully tackled with these techniques, routinely beating human performance in many tasks, including classification, segmentation, and detection.”
• Feature Pyramid Network for Multi-class Land Segmentation (2018, methodology): “This architecture shows significant improvement as a generic feature extractor in several applications such as object detection and instance object segmentation [15, 7, 21, 20].”
• TernausNetV2: Fully Convolutional Network for Instance Segmentation (2018, background): “This modification was called TernausNet [14] that we naturally extend in the current work (see also [20, 19]).”
• Unsupervised Intra-Domain Adaptation for Semantic Segmentation Through Self-Supervision (2020, background): “Recently, convolutional neural network-based segmentation models [13, 33] have achieved remarkable progresses, leading to various applications in computer vision systems, such as autonomous driving [14, 31], robotics [15, 23], and disease diagnosis [35, 32].”
• Fully Convolutional Network for Automatic Road Extraction from Satellite Imagery (2018, background): “This type of network architecture proven themselves in binary image segmentation competitions for satellite image analyses [6, 7, 8, 15, 16] and other applications [13, 12].”
• CHAOS Challenge - Combined (CT-MR) Healthy Abdominal Organ Segmentation (2020, neutral): “TernausNet (U-Net with VGG11 backbone [34]), LinkNet34 [35], and two with ResNet-50 and SE-Resnet50.”
• 2017 Robotic Instrument Segmentation Challenge (2019, neutral): “TernausNet is a U-Net-like architecture that uses relatively simple pre-trained VGG11 or VGG16 networks as an encoder [16], [17].”