Never-Ending Learning

By Tom Michael Mitchell, William W. Cohen, Estevam R. Hruschka, Partha P. Talukdar, Bo Yang, Justin Betteridge, Andrew Carlson, Bhavana Dalvi, Matt Gardner, B. Kisiel and 16 more. 2015

1,135 citations recorded by the source corpus.

Whereas people learn many different types of knowledge from diverse experiences over many years, most current machine learning systems acquire just a single function or data model from just a single data set. We propose a neverending learning paradigm for machine learning, to better reflect the more ambitious and encompassing type of learning performed by humans. As a case study, we describe the Never-Ending Language Learner (NELL), which achieves some of the desired properties of a never-ending learner, and we discuss lessons learned. NELL has been learning to read the web 24 hours/day since January 2010, and so far has acquired a knowledge base with over 80 million confidenceweighted beliefs (e.g., servedWith(tea, biscuits)). NELL has also learned millions of features and parameters that enable it to read these beliefs from the web. Additionally, it has learned to reason over these beliefs to infer new beliefs, and is able to extend its ontology by synthesizing new relational predicates. NELL can be tracked online at http://rtw.ml.cmu.edu, and followed on Twitter at @CMUNELL.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Representation & Self-Supervised Learning

Citations in the corpus

Adversarial NLI: A New Benchmark for Natural Language Understanding (2019, methodology): “The process can be iteratively repeated in a never-ending learning (Mitchell et al., 2018) setting, with the model getting stronger and the test set getting harder in each new round.”
Growing a Brain: Fine-Tuning by Increasing Model Capacity (2017, background): “In a broad sense, our approach is related to developmental learning [26, 16, 36] and lifelong learning [41, 25, 39, 29].”
Open-Domain Conversational Agents: Current Progress, Open Problems, and Future Directions (2020, background): “The general challenges of never-ending learning (Mitchell et al., 2018; Carlson et al., 2010) and avoiding catastrophic forgetting (French, 1999; Kirkpatrick et al., 2017) have a particular flavor in the domain of conversation: the general task is always to talk to people about any subject, but the…”
Learning without Forgetting (2016, methodology): “Second, in contrast to methods such as Never Ending Learning [25], LwF requires all new task training data to be present before computing their old task responses.”
Knowledge Flow: Improve Upon Your Teachers (2019, background): “Other related work includes policy distillation (Rusu et al., 2016a), domain adaptation (Pan & Yang, 2010; Long et al., 2015; Tzeng et al., 2015) or lifelong learning (Chen & Liu, 2016; Thrun, 1998; Mitchell et al., 2015; Ruvolo & Eaton, 2013).”
A Survey on Data Collection for Machine Learning: A Big Data - AI Integration Perspective (2018, background): “The Never-Ending Language Learner (NELL) system [141], [142] continuously extracts structured information from the unstructuredWeb and constructs a knowledge base that consists of entities and relationships.”
The Challenges of Continuous Self-Supervised Learning (2022, background): “Continual and lifelong learning: The ability to continuously learn new concepts or tasks over time is often referred to as lifelong learning [ 74 ] or never-ending learning [ 14 , 48 ].”
IIRC: Incremental Implicitly-Refined Classification (2020, background): “Lifelong Learning is a broad, multi-disciplinary, and expansive research domain with several synonyms: Incremental Learning [42], Continual Learning [48], and Never End-ing Learning [35].”
Psycholinguistics Meets Continual Learning: Measuring Catastrophic Forgetting in Visual Question Answering (2019, background): “Early work on life-long learning (Chen et al., 2015; Mitchell et al., 2015) is related to ours, but typically concerns a single task (e.g., relation extraction).”
End-to-End Incremental Learning (2018, neutral): “Another relevant approach is learning concepts over time, in the form of lifelong [33] or never-ending [5, 7, 20] learning.”
Continuous Adaptation via Meta-Learning in Nonstationary and Competitive Environments (2017, background): “The problem of continuous adaptation considered in this work is related to continual or life-long [11, 22] and never-ending [23] learning problems.”
Efficient Parametrization of Multi-domain Deep Neural Networks (2018, background): “Another important research direction in MTL is sequential learning of multiple tasks [21, 30].”
Continual Learning for Robotics (2019, background): “We consider continual learning a synonym of Incremental Learning [50, 125], Lifelong Learning [24, 159] and Never Ending Learning [19, 107].”