One Pass ImageNet
By Huiyi Hu, Ang Li, Daniele Calandriello, Dilan Gorur. 2021
21 citations recorded by the source corpus.
We present the One Pass ImageNet (OPIN) problem, which aims to study the effectiveness of deep learning in a streaming setting. ImageNet is a widely known benchmark dataset that has helped drive and evaluate recent advancements in deep learning. Typically, deep learning methods are trained on static data that the models have random access to, using multiple passes over the dataset with a random shuffle at each epoch of training. Such data access assumption does not hold in many real-world scenarios where massive amounts of data are collected from a stream and storing and accessing all the data becomes impractical due to storage costs and privacy concerns. For OPIN, we treat the ImageNet data as arriving sequentially, and there is limited memory budget to store a small subset of the data. We observe that training a deep network in a single pass with the same training settings used for multi-epoch training results in a huge drop in prediction accuracy. We show that the performance gap can be significantly decreased by paying a small memory cost and utilizing techniques developed for continual learning, despite the fact that OPIN differs from typical continual problem settings. We propose using OPIN to study resource-efficient deep learning.
Subfields: Artificial Intelligence
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Youngjae Min
- Oncel Tuzel
- Mehrdad Farajtabar
- Hadi Pouransari
- Kwangjun Ahn
- Navid Azizan
- Sachin Mehta
- Fartash Faghri
Citations in the corpus
- One-Pass Learning via Bridging Orthogonal Gradient Descent and Recursive Least-Squares (2022, positive): “In particular, [7] studied learning the ImageNet dataset in a single pass by revisiting some “important” previous datapoints at each learning step, and [18] investigated learning incremental ‘batches’ on a large scale by correcting the classifier’s bias towards new data.”
- TiC-CLIP: Continual Training of CLIP Models (2023, methodology): “9 zero-shot on Imagenet, whereas, without warm up ViT-B/16 achieves only 24 .”