Online Learning: A Comprehensive Survey

By Steven C. H. Hoi, Doyen Sahoo, Jing Lu, Peilin Zhao. 2018

832 citations recorded by the source corpus.

Online learning represents an important family of machine learning algorithms, in which a learner attempts to resolve an online prediction (or any type of decision-making) task by learning a model/hypothesis from a sequence of data instances one at a time. The goal of online learning is to ensure that the online learner would make a sequence of accurate predictions (or correct decisions) given the knowledge of correct answers to previous prediction or learning tasks and possibly additional information. This is in contrast to many traditional batch learning or offline machine learning algorithms that are often designed to train a model in batch from a given collection of training data instances. This survey aims to provide a comprehensive survey of the online machine learning literatures through a systematic review of basic ideas and key principles and a proper categorization of different algorithms and techniques. Generally speaking, according to the learning type and the forms of feedback information, the existing online learning works can be classified into three major categories: (i) supervised online learning where full feedback information is always available, (ii) online learning with limited feedback, and (iii) unsupervised online learning where there is no feedback available. Due to space limitation, the survey will be mainly focused on the first category, but also briefly cover some basics of the other two categories. Finally, we also discuss some open issues and attempt to shed light on potential future research directions in this field.

Subfields: Artificial Intelligence, Management Science and Operations Research

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

Citations in the corpus

Adaptive Deep Models for Incremental Learning: Considering Capacity Scalability and Sustainability (2019, neutral): “Incremental learning aims to update the models from data stream sequentially, and has achieved many successes in both application and theory [17, 37].”
A review of ensemble learning and data augmentation models for class imbalanced problems: combination, implementation and evaluation (2023, methodology): “In machine learning, online learning is a paradigm where data is used to train the model as it becomes available in a sequential order [284].”
Toward clinical digital phenotyping: a timely opportunity to consider purpose, quality, and safety (2019, methodology): “Or, alternatively, digital phenotyping data could be used to drive online optimization, 115 where intervention tailoring models are continuously updated (Fig.”
Continual Learning in Linear Classification on Separable Data (2023, methodology): “Forgetting is different from the regret used to analyze online learning algorithms (Crammer et al., 2006; Shalev-Shwartz et al., 2012; Hoi et al., 2021).”
Understanding Continual Learning Settings with Data Distribution Drift Analysis (2021, background): “Learning in a sequence of tasks where the labels change for a given sample has been less explored in continual learning recent literature, but many research studied it in the online learning literature (Hoi et al., 2018).”
ReFT: Reasoning with Reinforced Fine-Tuning (2024, neutral): “As mentioned in §4.2, the Online-ST baseline tries to mimic the same setting as in ReFT.”
Towards Stable Test-Time Adaptation in Dynamic Wild World (2023, background): “OL (Hoi et al., 2021; Chowdhury & Gopalan, 2017; Zhao et al., 2011) conducts model learning from a sequence of data samples one by one at a time, which is common in many real-world applications ( e.g. , social web recommendation).”
Locality Sensitive Sparse Encoding for Learning World Models Online (2024, methodology): “…representational advantage of our encoding over other non-linear features, and demonstrate that our method outperforms NNs in the online supervised learning setting (Orabona, 2019; Hoi et al., 2021) as well as the model-based reinforcement learning setting (Sutton, 1990) with models learned online.”
Continual Backprop: Stochastic Gradient Descent with Persistent Randomness (2021, methodology): “Although deep learning methods have been successful in various challenging problems, they do not work well in non-stationary problems (McCloskey and Cohen, 1989; French, 1997; Sahoo et al., 2018; Hoi et al., 2018).”
Cost-Effective Incremental Deep Model: Matching Model Capacity With the Least Sampling (2023, neutral): “Incremental modeling aims to update the models from data stream sequentially, and has achieved many successes in both application and theory [25], [26].”
A survey on deep learning tools dealing with data scarcity: definitions, challenges, solutions, tips, and applications (2023, methodology): “While avoiding inconsistency, online learning dictates model performance based on vast available knowledge [119].”
Confidence Weighted Multitask Learning (2019, background): “In contrast to batch learning, which often suffers from expensive re-training costs whenever new training data come, online learning avoids re-training and learns incrementally from sequential data, which is much more efﬁcient (Hoi et al. 2018; Zhao et al. 2011).”
Machine Learning and Integrative Analysis of Biomedical Big Data (2019, background): “Online algorithms are also useful in big data applications, especially when it is computationally infeasible to train models on the entire dataset all at once [200].”
A Survey on Deep Learning for Named Entity Recognition (2018, background): “Specifically, the RL problem can be formulated as follows [163]: the environment is modeled as a stochastic finite state machine with inputs (actions from agent) and outputs (observations and rewards to the agent).”