Drinking From a Firehose: Continual Learning With Web-Scale Natural Language
By Hexiang Hu, Ozan Sener, Fei Sha, Vladlen Koltun. 2020
32 citations recorded by the source corpus.
Continual learning systems will interact with humans, with each other, and with the physical world through time – and continue to learn and adapt as they do. An important open problem for continual learning is a large-scale benchmark which enables realistic evaluation of algorithms. In this paper, we study a natural setting for continual learning on a massive scale. We introduce the problem of personalized online language learning (POLL), which involves fitting personalized language models to a population of users that evolves over time. To facilitate research on POLL, we collect massive datasets of Twitter posts. These datasets, Firehose10 M and Firehose100 M, comprise 100 million tweets, posted by one million users over six years. Enabled by the Firehose datasets, we present a rigorous evaluation of continual learning algorithms on an unprecedented scale. Based on this analysis, we develop a simple algorithm for continual gradient descent (ConGraD) that outperforms prior continual learning methods on the Firehose datasets as well as earlier benchmarks. Collectively, the POLL problem setting, the Firehose datasets, and the ConGraD algorithm enable a complete benchmark for reproducible research on web-scale continual learning.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Infectious Diseases
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Joelle Pineau
- Yoshua Bengio
- Sergey Levine
- Mohit Bansal
- Jianfeng Gao
- Trevor Darrell
- Vladlen Koltun
- Tinne Tuytelaars
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Andrea Vedaldi
- Dacheng Tao
- Trevor Darrell
- Andrew Zisserman
- Jitendra Malik
- Xiaogang Wang
- Ming-Hsuan Yang
In Infectious Diseases
- Joelle Pineau
- Lucas Caccia
- Tinne Tuytelaars
- Eugene Belilovsky
- Vladlen Koltun
- Massimo Caccia
- Fei Ye
- Adrian G. Bors
Citations in the corpus
- Online Continual Learning with Natural Distribution Shifts: An Empirical Study with Visual Data (2021, background): “Recently, the Firehose benchmark [13] proposed online continual learning of natural language models.”
- The CLEAR Benchmark: Continual LEArning on Real-World Imagery (2022, background): “al [23], who assemble a CL dataset of Tweet messages that naturally evolve over time.”
- Reducing Representation Drift in Online Continual Learning (2021, background): “Multiple iterations and evaluating computation constraints Considered in (Caccia et al., 2019; Hu et al., 2020) we also evaluate the performance when multiple iterations for each incoming batch is allowed.”
- StreamingQA: A Benchmark for Adaptation to New Knowledge over Time in Question Answering Models (2022, background): “Lazaridou et al. (2021) and Hu et al. (2020) release large scale benchmarks for studying temporal adaptation in the language modeling task.”