Recognition in Terra Incognita

By Sara Beery, Grant Van Horn, Pietro Perona. 2018

1,098 citations recorded by the source corpus.

It is desirable for detection and classification algorithms to generalize to unfamiliar environments, but suitable benchmarks for quantitatively studying this phenomenon are not yet available. We present a dataset designed to measure recognition generalization to novel environments. The images in our dataset are harvested from twenty camera traps deployed to monitor animal populations. Camera traps are fixed at one location, hence the background changes little across images; capture is triggered automatically, hence there is no human bias. The challenge is learning recognition in a handful of locations, and generalizing animal detection and classification to new locations where no training data is available. In our experiments state-of-the-art algorithms show excellent performance when tested at the same location where they were trained. However, we find that generalization to new locations is poor, especially for classification systems.(The dataset is available at https://beerys.github.io/CaltechCameraTraps/)

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Big Transfer (BiT): General Visual Representation Learning (2019, background): “It has been shown that CNNs can lack robustness when classifying objects out-of-context [4,42,47].”
Toward Causal Representation Learning (2021, methodology): “Motivated by this, new benchmarks were proposed to specifically test a generalization of classification and detection methods with respect to simple algorithmically generated interventions, such as spatial shifts, blur, changes in brightness or contrast [107], [170], time consistency [95], [227], co”
Towards Causal Representation Learning (2021, background): “were proposed to specifically test generalization of classification and detection methods with respect to simple algorithmically generated interventions like spatial shifts, blur, changes in brightness or contrast [106, 170], time consistency [94, 227], control over background and rotation [11], as ”
Invariant Risk Minimization (2019, background): “As a thought experiment, consider the problem of classifying images of cows and camels [4].”
SWAD: Domain Generalization by Seeking Flat Minima (2021, positive): “It takes about 4 hours for PACS and VLCS , 8 hours for OfficeHome , 8.5 hours for TerraIncognita , and 56 hours for DomainNet on average, when using a single NVIDIA Tesla P40 GPU.”
In Search of Lost Domain Generalization (2020, methodology): “Examples of spurious correlations commonly absorbed by learning machines include racial biases [Stock and Cisse, 2018], texture statistics [Geirhos et al., 2018], and object backgrounds [Beery et al., 2018].”
WILDS: A Benchmark of in-the-Wild Distribution Shifts (2020, neutral): “, 2018), images taken by different cameras (Beery et al., 2018), bioassays from different cell types (Li et al.”
Inductive biases for deep learning of higher-level cognition (2020, neutral): “, 2016), spurious correlations (Krueger et al., 2021; Beery et al., 2018; Arjovsky et al., 2019) etc.”
Background Subtraction in Real Applications: Challenges, Current Models and Future Directions (2019, methodology): “In these application, the detection done by background subtraction is followed by a tracking phase or a recognition phase [32][398][385].”
SelfReg: Self-supervised Contrastive Regularization for Domain Generalization (2021, background): “This testbed currently provides seven multi-domain datasets (i.e. ColoredMNIST [1], RotatedMNIST [16], VLCS [13], PACS [26], Ofﬁce-Home [42], and TerraIncognita [2], DomainNet [34]) and provides benchmarks results of 14 baseline approaches (i.e. ERM [41], IRM [1], GroupDRO [36], Mixup [45], MLDG [27”
Gradient Matching for Domain Generalization (2021, positive): “…Rotated MNIST (Ghifary et al., 2015), VLCS (Fang et al., 2013), PACS (Li et al., 2017), OfﬁceHome (Venkateswara et al., 2017), Terra Incognita (Beery et al., 2018) and DomainNet (Peng et al., 2019), and offers comparison to a variety of SOTA domain generalization methods, including IRM…”
Out-of-Distribution Generalization via Risk Extrapolation (REx) (2020, background): “Most pictures of cows appear in pastures, and so common machine learning approaches struggle to classify pictures of cows on the beach (Beery et al., 2018).”
Last Layer Re-Training is Sufficient for Robustness to Spurious Correlations (2022, background): “Deep neural networks rely on these spurious features, and consequently degrade in performance when tested on examples where the spurious correlations break, for example, on images with unusual background contexts [25, 82, 7].”
Detecting semantic anomalies (2019, background): “A possibly related finding is that classification-trained CNNs find it much easier to overfit to low-level properties such as texture [2], canonical pose [3], or contextual cues [4] rather than learning globally coherent characteristics of objects.”
Efficient Pipeline for Camera Trap Image Review (2019, result): “Previous work has shown good results on automated species classification in camera trap data [8], but further analysis has shown that these results do not generalize to new cameras or new geographical regions [3].”
The iWildCam 2021 Competition Dataset (2021, background): “This is central to the 2018 [11], 2019 [9], 2020 [7] and 2021 iWildCam challenges, all of which split train and test data by camera location, so no images from the test cameras are included in the training set to avoid overfitting to one set of backgrounds [12].”
Bugs in the Data: How ImageNet Misrepresents Biodiversity (2022, positive): “The iWildCam Competition is another notable example of extensive biodiversity datasets, leveraging camera traps to gather hundreds of thousands of images across the world [3, 4]; these datasets have been used to host competitions, establish benchmarks, and assess model generalization to novel ecosys”