The Lovasz-Softmax Loss: A Tractable Surrogate for the Optimization of the Intersection-Over-Union Measure in Neural Networks

By Maxim Berman, Amal Rannen Triki, Matthew B. Blaschko. 2017

997 citations recorded by the source corpus.

The Jaccard index, also referred to as the intersection-over-union score, is commonly employed in the evaluation of image segmentation results given its perceptual qualities, scale invariance - which lends appropriate relevance to small objects, and appropriate counting of false negatives, in comparison to per-pixel losses. We present a method for direct optimization of the mean intersection-over-union loss in neural networks, in the context of semantic image segmentation, based on the convex LovÃ¡sz extension of submodular losses. The loss is shown to perform better with respect to the Jaccard index measure than the traditionally used cross-entropy loss. We show quantitative and qualitative differences between optimizing the Jaccard index per image versus optimizing the Jaccard index taken over an entire dataset. We evaluate the impact of our method in a semantic segmentation pipeline and show substantially improved intersection-over-union segmentation scores on the Pascal VOC and Cityscapes datasets using state-of-the-art deep learning segmentation architectures.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Optimization & Training

Citations in the corpus

Generalized Intersection Over Union: A Metric and a Loss for Bounding Box Regression (2019, methodology): “Optimizing IoU using an approximate or a surrogate function: In the semantic segmentation task, there have been some efforts to optimize IoU using either an approximate function [18] or a surrogate loss [17].”
Instance Segmentation by Jointly Optimizing Spatial Embeddings and Clustering Bandwidth (2019, methodology): “ary classiﬁcation loss with the binary foreground/background map of each instance as ground-truth. As opposed to using the standard cross-entropy loss function, we opt for using the Lovasz-hinge loss [4,28] instead. Since this loss function is a (piecewise linear) convex surrogate to the Jaccard los”
Semi-Supervised Segmentation of Salt Bodies in Seismic Images using an Ensemble of Convolutional Neural Networks (2019, methodology): “ learning rate starts from 0:001 and decays to 0:0001 every cycle. Model weights are ”warmed-up” using binary crossentropy loss during the ﬁrst 50 epochs. After that, we minimize Lovasz loss function [4] for 150 epochs, which allows a direct optimization of the IoU metric. The warm-up phase is neces”
Salient Object Detection in the Deep Learning Era: An In-Depth Survey (2019, background): “Another recent work [138] proposes to directly optimize the mean intersection-over-union loss, which brings impact to semantic segmentation as well as its binary case, i.”
(AF)2-S3Net: Attentive Feature Fusion with Adaptive Feature Selection for Sparse Semantic Segmentation Network (2021, methodology): “We leveraged a linear combination of geo-aware anisotrophic [20], Exponential-log loss [35] and Lovász loss [4] to optimize our network.”
SuperFusion: A Versatile Image Registration and Fusion Network with Semantic Awareness (2022, methodology): “More details about the implementation of the Lovasz-Softmax loss can be found in the original paper [74].”
Tri-Perspective View for Vision-Based 3D Semantic Occupancy Prediction (2023, methodology): “Following Cylinder3D [51], we employ both cross entropy loss and lovasz-softmax [3] loss to optimize our network.”
Exploring Cross-Image Pixel Contrast for Semantic Segmentation (2021, methodology): “These two issues were rarely noticed; only a few structure-aware losses are designed to address 1), by considering pixel affinity [40], optimizing intersection-over-union measurement [2], or maximizing the mutual information between the groundtruth and prediction map [86].”
CycleSegNet: Object Co-Segmentation With Cycle Refinement and Region Correspondence (2021, methodology): “Please refer to [3] for more details about Lovász-Softmax loss function.”
Optimization for Medical Image Segmentation: Theory and Practice When Evaluating With Dice Score or Jaccard Index (2020, methodology): “To investigate whether the aforementioned properties hold in practice, we investigate the performance of segmentation networks trained with different loss functions: cross-entropy (CE), weighted cross-entropy (wCE), soft Dice (sDice), soft Jaccard (sJaccard), Lovász-sigmoid (Lovász; this is the bina”
Confidence Calibration and Predictive Uncertainty Estimation for Deep Medical Image Segmentation (2019, background): “of Dice loss and cross-entropy loss, as well as the recently proposed Lovász-Softmax loss [54] that we think is promising for medical image segmentation.”
ChangeMamba: Remote Sensing Change Detection With Spatiotemporal State Space Model (2024, methodology): “During training, we optimized the network using the AdamW optimizer [57] with a learning rate of 1 e − 4 and weight decay of 5 e − 3 , without any learning rate decay strategy.”
Optimizing the Dice Score and Jaccard Index for Medical Image Segmentation: Theory and Practice (2019, background): “soft Jaccard [20,16] or its more recent convex extension Lovász-softmax [5]) in order to incorporate it into gradientbased training schemes, such as stochastic gradient descent (SGD).”
Breast Tumor Cellularity Assessment using Deep Neural Networks (2019, methodology): “[33] used the Resnet-34 network [34] as the encoder and the LovászSoftmax loss function [35] along with Stochastic Weight Averaging (SWA) [36] for training.”