Not Using the Car to See the Sidewalk — Quantifying and Controlling the Effects of Context in Classification and Segmentation

By Rakshith Shetty, Bernt Schiele, Mario Fritz. 2018

87 citations recorded by the source corpus.

Importance of visual context in scene understanding tasks is well recognized in the computer vision community. However, to what extent the computer vision models are dependent on the context to make their predictions is unclear. A model overly relying on context will fail when encountering objects in different contexts than in training data and hence it is important to identify these dependencies before we can deploy the models in the real-world. We propose a method to quantify the sensitivity of black-box vision models to visual context by editing images to remove selected objects and measuring the response of the target models. We apply this methodology on two tasks, image classification and semantic segmentation, and discover undesirable dependency between objects and context, for example that ``sidewalk'' segmentation is very sensitive to the presence of ``cars'' in the image. We propose an object removal based data augmentation solution to mitigate this dependency and increase the robustness of classification and segmentation models to contextual variations. Our experiments show that the proposed data augmentation helps these models improve the performance in out-of-context scenarios, while preserving the performance on regular data.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Big Transfer (BiT): General Visual Representation Learning (2019, background): “It has been shown that CNNs can lack robustness when classifying objects outof-context [2,6,7].”
Toward Causal Representation Learning (2021, background): “In computer vision [76], [228], changes in the test distribution may, for instance, come from aberrations, such as camera blur, noise, or compression quality [107], [129], [170], [206], or from shifts, rotations, or viewpoints [7], [12], [64], [282].”
Towards Causal Representation Learning (2021, background): “In computer vision [75, 228], changes in the test distribution may, for instance, come from aberrations like camera blur, noise or compression quality [106, 129, 170, 206], or from shifts, rotations, or viewpoints [7, 11, 63, 282].”
Object-Contextual Representations for Semantic Segmentation (2019, positive): “4 We use “object” to represent both “things” and “stuff” following [14,54].”
WILDS: A Benchmark of in-the-Wild Distribution Shifts (2020, neutral): “First, several works have shown that object classifiers often rely on the background rather than the object to make its classification (Rosenfeld et al., 2018; Shetty et al., 2019; Xiao et al., 2020).”
OCNet: Object Context Network for Scene Parsing (2018, neutral): “The importance of context for semantic segmentation is also verified in the recent works (Chen et al., 2017; Liu et al., 2015; Shetty et al., 2019; Zhao et al., 2017).”
Learning De-biased Representations with Biased Representations (2019, result): “Such a data augmentation strategy is indeed a valid solution adopted by many prior studies (Panda et al., 2018; Geirhos et al., 2019; Shetty et al., 2019).”
Last Layer Re-Training is Sufficient for Robustness to Spurious Correlations (2022, background): “Prior work has demonstrated that computer vision models such as ImageNet classifiers can rely on background to make their predictions [105, 81, 89, 100, 90].”
OCNet: Object Context for Semantic Segmentation (2021, background): “The recent study (Hoyer et al. 2019; Shetty et al. 2019) has verified that the overuse of the noisy context information based on PPM suffers from poor generalization ability.”