Scaling Open-Vocabulary Object Detection

By Matthias Minderer, Alexey A. Gritsenko, Neil Houlsby. Neural Information Processing Systems, 2023

426 citations recorded by the source corpus.

Open-vocabulary object detection has benefited greatly from pretrained vision-language models, but is still limited by the amount of available detection training data. While detection training data can be expanded by using Web image-text pairs as weak supervision, this has not been done at scales comparable to image-level pretraining. Here, we scale up detection data with self-training, which uses an existing detector to generate pseudo-box annotations on image-text pairs. Major challenges in scaling self-training are the choice of label space, pseudo-annotation filtering, and training efficiency. We present the OWLv2 model and OWL-ST self-training recipe, which address these challenges. OWLv2 surpasses the performance of previous state-of-the-art open-vocabulary detectors already at comparable training scales (~10M examples). However, with OWL-ST, we can scale to over 1B examples, yielding further large improvement: With an L/14 architecture, OWL-ST improves AP on LVIS rare classes, for which the model has seen no human box annotations, from 31.2% to 44.6% (43% relative improvement). OWL-ST unlocks Web-scale training for open-world localization, similar to what has been seen for image classification and language modelling.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/2bf1b63d4222a14f78bfcb481378c3caf9c39528

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Sergey Levine
2. Chelsea Finn
3. Yuke Zhu
4. Dieter Fox
5. Pieter Abbeel
6. Karl Pertsch
7. Li Fei-Fei
8. Trevor Darrell

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Xinggang Wang
3. Sergey Levine
4. Trevor Darrell
5. Dacheng Tao
6. Jitendra Malik
7. Li Fei-Fei
8. Xiaogang Wang

Citations in the corpus

• YOLO-World: Real-Time Open-Vocabulary Object Detection (2024, methodology): “Inspired by vision-language pre-training [19, 39], recent works [35, 36, 56, 57, 60, 63] formulate open-vocabulary object detection as image-text matching and exploit large-scale image-text data to increase the training vocabulary at scale.”
• Robotic Control via Embodied Chain-of-Thought Reasoning (2024, positive): “We use OWLv2 [60] and SAM [61] to detect 2D end effector positions in the training images ( GRIPPER ) paired with 3D positions extracted from the robot state to fit a robust estimate of the projection matrix using RANSAC [65].”
• GR00T N1: An Open Foundation Model for Generalist Humanoid Robots (2025, positive): “Specifically, for each frame in a trajectory segment, we annotate the bounding box of the target object using the OWL-v2 object detector (Minderer et al., 2023).”