The Omniglot Challenge: A 3-Year Progress Report

By Brenden M. Lake, Ruslan Salakhutdinov, Josh Tenenbaum. 2019

148 citations recorded by the source corpus.

Three years ago, we released the Omniglot dataset for one-shot learning, along with five challenge tasks and a computational model that addresses these tasks. The model was not meant to be the final word on Omniglot; we hoped that the community would build on our work and develop new approaches. In the time since, we have been pleased to see wide adoption of the dataset. There has been notable progress on one-shot classification, but researchers have adopted new splits and procedures that make the task easier. There has been less progress on the other four tasks. We conclude that recent approaches are still far from human-like concept learning on Omniglot, a challenge that requires performing many tasks with a single model.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Evaluation & Benchmarks, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Evaluation & Benchmarks

In Meta & Continual Learning

Citations in the corpus

Omnidata: A Scalable Pipeline for Making Multi-Task Mid-Level Vision Datasets from 3D Scans (2021, background): “MTL datasets often take different shades of classification as the core problem of interest [34, 64, 39].”
A Comprehensive Survey of Few-shot Learning: Evolution, Applications, Challenges, and Opportunities (2022, unknown): “5, during 2017-2021, 898 papers used the CUB-200-2011 [28] dataset, accounting for 46.6% of the total number of statistics; 587 papers used the Mini-ImageNet [29] dataset, accounting for 30.5%; and 335 papers using the Omniglot [30] dataset, accounting for 17.4% ; 44 papers used the PASCAL-5i [31] d”
SketchEmbedNet: Learning Novel Concepts by Imitating Drawings (2020, unknown): “(Snell et al., 2017; Lake et al., 2019) Conv4 Omniglot - - 86.30 - RCN (Supervised) (George et al., 2017; Lake et al., 2019) N/A Omniglot - - 92.70 - VHE (Supervised) (Hewitt et al., 2018; Lake et al., 2019) N/A Omniglot - - 81.30 -”
Opening up Open World Tracking (2021, background): “The task is related to many other fields pushing towards similar goals such as few-shot learning [85, 94, 49, 50], open-world recognition [7, 58, 70] and zero-shot learning [111].”
Data Distributional Properties Drive Emergent Few-Shot Learning in Transformers (methodology): “Although we were inspired by the properties of natural language, we hypothesized that these eﬀects are general, and therefore performed our experiments using a standard image-based few-shot dataset (the Omniglot dataset; Lake et al., 2019).”
Data Distributional Properties Drive Emergent In-Context Learning in Transformers (2022, methodology): “We performed our experiments over data sequences sampled from a standard image-based few-shot dataset (the Omniglot dataset; Lake et al., 2019).”
Compositional generalization through meta sequence-to-sequence learning (2019, methodology): “Meta learning has been successfully applied to few-shot image classification [36, 32, 7, 19] including sequential versions that require external memory [31].”