Object-Proposal Evaluation Protocol is ‘Gameable’

By Neelima Chavali, Harsh Agrawal, Aroma Mahendru, Dhruv Batra. 2015

84 citations recorded by the source corpus.

Object proposals have quickly become the de-facto preprocessing step in a number of vision pipelines (for object detection, object discovery, and other tasks). Their performance is usually evaluated on partially annotated datasets. In this paper, we argue that the choice of using a partially annotated dataset for evaluation of object proposals is problematic - as we demonstrate via a thought experiment, the evaluation protocol is 'gameable', in the sense that progress under this protocol does not necessarily correspond to a "better" category independent object proposal algorithm. To alleviate this problem, we: (1) Introduce a nearly-fully annotated version of PASCAL VOC dataset, which serves as a test-bed to check if object proposal techniques are overfitting to a particular list of categories. (2) Perform an exhaustive evaluation of object proposal methods on our introduced nearly-fully annotated PASCAL dataset and perform cross-dataset generalization experiments, and (3) Introduce a diagnostic experiment to detect the bias capacity in an object proposal algorithm. This tool circumvents the need to collect a densely annotated dataset, which can be expensive and cumbersome to collect. Finally, we have released an easy-to-use toolbox which combines various publicly available implementations of object proposal algorithms which standardizes the proposal generation and evaluation so that new methods can be added and evaluated on different datasets. We hope that the results presented in the paper will motivate the community to test the category independence of various object proposal methods by carefully choosing the evaluation protocol.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Evaluation & Benchmarks

Citations in the corpus

Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks (2015, background): “[19], [20], [21] related to the ultimate detection accuracy.”
Learning to Segment Object Candidates (2015, methodology): “Generalization: To see if our approach can generalize to unseen classes [2, 19], we train two additional versions of our model, DeepMask20 and DeepMask20∗.”
Segment Anything (2023, methodology): “We note that this “baseline” corresponds to the “Detector Masquerading as Proposal generator” (DMP) method [16] that was shown to game AR, making it a truly demanding comparison.”
Open-World Instance Segmentation: Exploiting Pseudo Ground Truth From Learned Pairwise Affinity (2022, background): “Evaluating open-world segmentation on such datasets may fail to capture performance difference across methods due to punishing precision and not rewarding recall [8].”
Learning Open-World Object Proposals without Learning to Classify (2021, methodology): “[19] report the vulnerability of current evaluation protocol for object proposals when evaluating on a partially-annotated dataset such as COCO, because it fails to capture the performance (e.”
Attend Refine Repeat: Active Box Proposal Generation via In-Out Localization (2016, methodology): “[4], we evaluate our AttractioNet model on two extra datasets that are labelled with object categories that are not present in its training set ("unseen" object categories).”
ME R-CNN: Multi-Expert Region-based CNN for Object Detection (2017, background): “Several more approaches [5, 6, 18, 19, 33, 37] to generate RoIs based on objectness characteristics have been introduced afterwards.”
Image‐based post‐disaster inspection of reinforced concrete bridge systems using deep learning with Bayesian optimization (2018, methodology): “Comprehensive surveys and comparisons of object proposal methods can be found in Hosang, Benenson, and Schiele (2014), Hosang, Benenson, Dollár, and Schiele (2015), and Chavali, Agrawal, Mahendru, and Batra (2015).”
Object Detection in 20 Years: A Survey (2019, background): “object proposal detection has evolved from the bottom-up vision to “overfitting to a specific set of object classes”, and the distinction between detectors and proposal generators is becoming blurred [132].”
What leads to generalization of object proposals? (2020, methodology): “study of different methods can be found in [13] and a study of proposal evaluation metrics can be found in [2].”
Cascade RPN: Delving into High-Quality Region Proposal Network with Adaptive Convolution (2019, background): “Region proposals have become the de-facto paradigm for high-quality object detectors [6, 19, 20].”
Zoom Out-and-In Network with Map Attention Decision for Region Proposal and Object Detection (2017, background): “…algorithms (Uijlings et al., 2013; Man´en et al., 2013; Arbel ´ aez et al., 2014; Hayder et al., 2016; Kong et al., 2016; Ghodrati et al., 2016; Chavali et al., 2016; Sun et al., 2016; Li et al., 2017a) have signiﬁcantly boosted the development of many vision tasks, (Liu et al., 2017a,b; Li et…”
Semantic Image Interpretation - Integration of Numerical Data and Logical Knowledge for Cognitive Vision (2018, unknown): “There is a huge literature on object proposal methods and comprehensive surveys can be found in [49, 14].”
Active Object Localization with Deep Reinforcement Learning (2015, methodology): “We use this scoring function instead of the classiﬁcation scores for fairness in the evaluation [4], since other methods rank proposals using an estimated objectness score, and high Q-values can be interpreted in the same way.”
BING: Binarized normed gradients for objectness estimation at 300fps (2014, unknown): “com/ [36] , RPN [30] , Endres [21], Objectness [2], GOP [48], LPO [49], Rahtu [45], RandomPrim [46], Rantalankila [47], and SelectiveSearch [20], using publicly available code [82] downloaded from https: //github.”
Deep Learning for Generic Object Detection: A Survey (2018, methodology): “view of object proposal algorithms is outside the scope of this paper, because object proposals have applications beyond object detection [6,72,252]. We refer interested readers to the recent surveys [86,23] which provides an in-depth 3 We use the terminology detection proposals, object proposals an”