ORBIT: A Real-World Few-Shot Dataset for Teachable Object Recognition

By Daniela Massiceti, Luisa M. Zintgraf, John F. Bronskill, Lida Theodorou, M. Harris, Edward Cutrell, Cecily Morrison, Katja Hofmann, Simone Stumpf. 2021

55 citations recorded by the source corpus.

Object recognition has made great advances in the last decade, but predominately still relies on many high-quality training examples per object category. In contrast, learning new objects from only a few examples could enable many impactful applications from robotics to user personalization. Most few-shot learning research, however, has been driven by benchmark datasets that lack the high variation that these applications will face when deployed in the real-world. To close this gap, we present the ORBIT dataset and benchmark, grounded in the real-world application of teachable object recognizers for people who are blind/low-vision. The dataset contains 3,822 videos of 486 objects recorded by people who are blind/low-vision on their mobile phones. The benchmark reflects a realistic, highly challenging recognition problem, providing a rich playground to drive research in robustness to few-shot, high-variation conditions. We set the benchmark’s first state-of-the-art and show there is massive scope for further innovation, holding the potential to impact a broad range of real-world vision applications including tools for the blind/low-vision community. We release the dataset at https://doi.org/10.25383/city.14294597 and benchmark code at https://github.com/microsoft/ORBIT-Dataset.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Evaluation & Benchmarks, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Richard E. Turner
2. Sebastian Nowozin
3. John F. Bronskill
4. Katja Hofmann
5. Massimiliano Patacchiola
6. Sergey Levine
7. Daniela Massiceti
8. Aliaksandra Shysheya

In Computer Vision and Pattern Recognition

1. Richard E. Turner
2. John F. Bronskill
3. Sebastian Nowozin
4. Massimiliano Patacchiola
5. Katja Hofmann
6. Luisa M. Zintgraf
7. Cecily Morrison
8. Aliaksandra Shysheya

In Evaluation & Benchmarks

1. Aliaksandra Shysheya
2. Katja Hofmann
3. Richard E. Turner
4. Simone Stumpf
5. Sebastian Nowozin
6. Massimiliano Patacchiola
7. John F. Bronskill

In Meta & Continual Learning

1. Katja Hofmann
2. Massimiliano Patacchiola
3. Richard E. Turner
4. John F. Bronskill
5. Sebastian Nowozin
6. Luisa M. Zintgraf
7. Daniela Massiceti
8. Chelsea Finn

Citations in the corpus

• FiT: Parameter Efficient Few-shot Transfer Learning for Personalized and Federated Image Classification (2022, methodology): “Third, we show how FIT can be used in a personalization scenario on the ORBIT (Massiceti et al., 2021) dataset, where a smaller updateable model is an important evaluation metric.”
• Disability-first Dataset Creation: Lessons from Constructing a Dataset for Teachable Object Recognition with Blind and Low Vision Data Collectors (2021, background): “Our work, which resulted in a disability-first dataset [38] is a significant step in contributing to driving ML innovation for the disabled community [39].”
• Memory Efficient Meta-Learning with Large Images (2021, methodology): “We use LITE to train key meta-learning methods and show that our best performing instantiation – Simple CNAPs [5] with LITE – achieves state-of-the-art results relative to all meta-learners on two challenging few-shot benchmarks: VTAB+MD [11], an extensive suite of both meta-learning and transfer le”