ObjectNet: A large-scale bias-controlled dataset for pushing the limits of object recognition models

By Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, D. Gutfreund, Josh Tenenbaum, Boris Katz. 2019

692 citations recorded by the source corpus.

Subfields: Computer Vision and Pattern Recognition, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Evaluation & Benchmarks

Citations in the corpus

Learning Transferable Visual Models From Natural Language Supervision (2021, background): “Curiously, accuracy on ObjectNet also increases by 2.3%.”
AI and the Everything in the Whole Wide World Benchmark (2021, background): “Specifically, a well-critiqued limitation of ImageNet is that the objects tend to be centered within the images, which does not reflect how “natural” images actually appear [Barbu et al., 2019].”
Shortcut learning in deep neural networks (2020, background): “benchmarking, allowing to disentangle the influence of background, rotation and viewpoint [98].”
On Robustness and Transferability of Convolutional Neural Networks (2020, background): “Metrics For robustness, we measure the model accuracy on the IMAGENET, IMAGENET-V2 (the matched frequency variant) and OBJECTNET datasets.”
Are we done with ImageNet? (2020, background): “ObjectNet [25] is another work that collects a new validation set that can be used to evaluate ImageNet models, focusing on gathering challenging “real-life” images in diverse contexts.”
Underspecification Presents Challenges for Credibility in Modern Machine Learning (2020, methodology): “The variability of predictor accuracies on ObjectNet is large compared to this baseline (p = 0.002 for ResNet-50 and p = 0.000 for BiT).”
Measuring Robustness to Natural Distribution Shifts in Image Classification (2020, methodology): “[2]; and, (iii) ImageNetVid-Robust-anchor and YTBB-Robust-anchor [76], which are the datasets constructed from only the anchor frames in the consistency datasets described above.”
Robust Speech Recognition via Large-Scale Weak Supervision (2022, background): “…works have shown and continually reiterated how despite high performance on IID test sets, machine learning models can still make many mistakes when evaluated in even slightly different settings (Lake et al., 2017; Jia & Liang, 2017; Alcorn et al., 2019; Barbu et al., 2019; Recht et al., 2019).”
Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time (2022, background): “Only on ReaL and ObjectNet does there exist an individual model that performs statistically signiﬁcantly better than the soup, and the best model differs between those two datasets.”
Combined Scaling for Zero-shot Transfer Learning (2021, methodology): “For instance, on 5 test sets with natural distribution shifts such as ImageNet-{A,R,V2,Sketch} and ObjectNet, the model achieves an average of 83.7% top-1 accuracy, only a small drop from its original ImageNet accuracy (see Table 1).”
LiT: Zero-Shot Transfer with Locked-image text Tuning (2021, methodology): “Notably, the LiT model sets a new state-of-the-art 81.1% accuracy on the ObjectNet test set.”
ORBIT: A Real-World Few-Shot Dataset for Teachable Object Recognition (2021, background): “Datasets captured by users in real-world settings are naturally high-variation [1, 12, 7, 21, 27, 18, 41, 13], but none collected thus far explicitly target few-shot object recognition.”
Reinforce Data, Multiply Impact: Improved Model Accuracy and Robustness with Dataset Reinforcement (2023, methodology): “We also evaluate ImageNet models on other distribution shift datasets, ImageNet-A [24], ImageNet-R [22], ImageNet-Sketch [61], ObjectNet [4], and ImageNet-C [23].”
LAION-5B: An open large-scale dataset for training next generation image-text models (2022, methodology): “Some datasets such as ObjectNet [3] are likely not contained in Common Crawl because ObjectNet was not assembled from web images.”
Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data (2020, background): “In the vision community, Barbu et al. (2019) offer a novel dataset which explicitly tries to achieve a more realistic distribution of task data; it would be interesting to explore similar ideas for language.”
Toward Causal Representation Learning (2021, background): “…and detection methods with respect to simple algorithmically generated interventions, such as spatial shifts, blur, changes in brightness or contrast [107], [170], time consistency [95], [227], control over background and rotation [12], as well as images collected in multiple environments [20].”
Towards Causal Representation Learning (2021, background): “In computer vision [75, 228], changes in the test distribution may, for instance, come from aberrations like camera blur, noise or compression quality [106, 129, 170, 206], or from shifts, rotations, or viewpoints [7, 11, 63, 282].”
CoCa: Contrastive Captioners are Image-Text Foundation Models (2022, neutral): “7 Table 4: Zero-shot image classification results on ImageNet [9], ImageNet-A [64], ImageNet-R [65], ImageNet-V2 [66], ImageNet-Sketch [67] and ObjectNet [68].”