Neural Topic Modeling with Continual Lifelong Learning
By Pankaj Gupta, Yatin Chaudhary, Thomas A. Runkler, Hinrich Schütze. 2020
56 citations recorded by the source corpus.
Lifelong learning has recently attracted attention in building machine learning systems that continually accumulate and transfer knowledge to help future learning. Unsupervised topic modeling has been popularly used to discover topics from document collections. However, the application of topic modeling is challenging due to data sparsity, e.g., in a small collection of (short) documents and thus, generate incoherent topics and sub-optimal document representations. To address the problem, we propose a lifelong learning framework for neural topic modeling that can continuously process streams of document collections, accumulate topics and guide future topic modeling tasks by knowledge transfer from several sources to better deal with the sparse data. In the lifelong process, we particularly investigate jointly: (1) sharing generative homologies (latent topics) over lifetime to transfer prior knowledge, and (2) minimizing catastrophic forgetting to retain the past learning via novel selective data augmentation, co-training and topic regularization approaches. Given a stream of document collections, we apply the proposed Lifelong Neural Topic Modeling (LNTM) framework in modeling three sparse document collections as future tasks and demonstrate improved performance quantified by perplexity, topic coherence and information retrieval task.
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
Citations in the corpus
- Achieving Forgetting Prevention and Knowledge Transfer in Continual Learning (2021, background): “Several papers also studied lifelong topic modeling [5, 13].”
- A Continual Learning Survey: Defying Forgetting in Classification Tasks (2019, background): “Albeit a challenging problem, progress in continual learning has led to real-world applications starting to emerge [21], [22], [23]. figuration, with an exclusive output layer or head for each task.”
- How to Reuse and Compose Knowledge for a Lifetime of Tasks: A Survey on Continual Learning and Functional Composition (2022, methodology): “These same techniques have been used with a meta-learning objective function, whereby the agent trains directly to optimize the network’s feature representation to avoid gradient conflicts between future and previous tasks (Riemer et al., 2019; Gupta et al., 2020a).”
- Continual Pre-training of Language Models (2023, methodology): “In NLP, CL has been used for slot filling (Shen et al., 2019), language learning (Li et al., 2019), sentiment analysis (Ke et al., 2021a), topic modeling (Gupta et al., 2020), question answering (Greco et al., 2019) and text classification (Sun et al., 2020; Huang et al., 2021; Chuang et al., 2020).”
- Pre-training is a Hot Topic: Contextualized Document Embeddings Improve Topic Coherence (2020, background): “Even for neural topic models, there exists work on incorporating external knowledge, e.g., via word embeddings (Gupta et al., 2019, 2020; Dieng et al., 2020).”