Benchmarking Robustness in Object Detection: Autonomous Driving when Winter is Coming

By Claudio Michaelis, Benjamin Mitzkus, Robert Geirhos, Evgenia Rusak, Oliver Bringmann, Alexander S. Ecker, Matthias Bethge, Wieland Brendel. 2019

574 citations recorded by the source corpus.

The ability to detect objects regardless of image distortions or weather conditions is crucial for real-world applications of deep learning like autonomous driving. We here provide an easy-to-use benchmark to assess how object detection models perform when image quality degrades. The three resulting benchmark datasets, termed Pascal-C, Coco-C and Cityscapes-C, contain a large variety of image corruptions. We show that a range of standard object detection models suffer a severe performance loss on corrupted images (down to 30--60\% of the original performance). However, a simple data augmentation trick---stylizing the training images---leads to a substantial increase in robustness across corruption type, severity and dataset. We envision our comprehensive benchmark to track future progress towards building robust object detection models. Benchmark, code and data are publicly available.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Evaluation & Benchmarks

Citations in the corpus

Shortcut learning in deep neural networks (2020, background): “This has been reported in various cases such as shifting the image by a few pixels [47], rotating the object [49], adding a bit of random noise or blur [51, 50, 52, 53] or (as discussed earlier) by changing background [9] or texture while keeping the shape intact [38] (see Figure 3 for examples).”
Toward Causal Representation Learning (2021, background): “…and detection methods with respect to simple algorithmically generated interventions, such as spatial shifts, blur, changes in brightness or contrast [107], [170], time consistency [95], [227], control over background and rotation [12], as well as images collected in multiple environments [20].”
Towards Causal Representation Learning (2021, background): “In computer vision [75, 228], changes in the test distribution may, for instance, come from aberrations like camera blur, noise or compression quality [106, 129, 170, 206], or from shifts, rotations, or viewpoints [7, 11, 63, 282].”
SWAD: Domain Generalization by Seeking Flat Minima (2021, background): “For example, a self-driving car should adapt to adverse weather or day-to-night shifts [1, 2].”
A Simple Way to Make Neural Networks Robust Against Diverse Image Corruptions (2020, methodology): “Similarly, common corruptions have been applied to and evaluated on COCO-C, Pascal-C, Cityscapes-C [25] and MNIST-C [29].”
Bayesian Few-Shot Classification with One-vs-Each Pólya-Gamma Augmented Gaussian Processes (2020, methodology): “We used the Imagecorruptions package [17] to apply Gaussian noise, impulse noise, and defocus blur to both the support set and query sets of episodes at test time and evaluated both accuracy and calibration.”
Increasing the robustness of DNNs against image corruptions by playing the Game of Noise (2020, methodology): “Concurrent work to ours demonstrates that having more training data [Xie et al., 2019a, Mahajan et al., 2018] or using stronger backbones [Xie et al., 2019a, Michaelis et al., 2019] can signiﬁcantly improve model performance on common corruptions.”
Improving robustness against common corruptions by covariate shift adaptation (2020, methodology): “The IN-C benchmark [2] has been extended to MNIST [41], several object detection datasets [42] and image segmentation [43] reflecting the interest of the robustness community.”
Multi-Sensor Fusion in Automated Driving: A Survey (2020, methodology): “In reference [66], defaced images produced based on the existing datasets of Pascal, Coco, and Cityscapes were used to evaluate the most advanced target detection algorithms.”
Wavelet Convolutions for Large Receptive Fields (2024, methodology): “We also evaluate object detection on COCO under corruption [40].”