Pre-training is a Hot Topic: Contextualized Document Embeddings Improve Topic Coherence
By Federico Bianchi, Silvia Terragni, Dirk Hovy. Annual Meeting of the Association for Computational Linguistics, 2020
408 citations recorded by the source corpus.
Topic models extract groups of words from documents, whose interpretation as a topic hopefully allows for a better understanding of the data. However, the resulting word groups are often not coherent, making them harder to interpret. Recently, neural topic models have shown improvements in overall coherence. Concurrently, contextual embeddings have advanced the state of the art of neural models in general. In this paper, we combine contextualized representations with neural topic models. We find that our approach produces more meaningful and coherent topics than traditional bag-of-words topic models and recent neural models. Our results indicate that future improvements in language models will translate into better topic models.
Subfields: Artificial Intelligence, General Social Sciences
View Paper: https://www.semanticscholar.org/paper/07d85a6c8a31f43ee6e128f7ef0a1bf1494567cd
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Pascale Fung
- Andrea Madotto
- Genta Indra Winata
- Maarten Grootendorst
- Federico Bianchi
- Dirk Hovy
- Zihan Liu
- Yan Xu
In General Social Sciences
- Maarten Grootendorst
- Dirk Hovy
- Federico Bianchi
- Zihan Liu
- Andrea Madotto
- Etsuko Ishii
- Pascale Fung
- Genta Indra Winata
Citations in the corpus
- Retrieval-Free Knowledge-Grounded Dialogue Response Generation with Adapters (2021, methodology): “While any sort of topic models can be used, we adopt contextual topic model (CTM) which outperforms traditional topic models (Bianchi, Terragni, and Hovy 2021).”
- BERTopic: Neural topic modeling with a class-based TF-IDF procedure (2022, methodology): “CTM, for example, demonstrates the advantage of relying on pre-trained language models, namely that future improvements in language models may translate into better topic models (Bianchi et al., 2020a).”
- On the Gap between Adoption and Understanding in NLP (2021, methodology): “…is not exhibiting what we as humans would call “intelligence” (Marcus and Davis, 2020; Floridi and Chiriatti, 2020) and have suggested that learning just from text and without including external experience might be a difficult task (Bisk et al., 2020; Bender and Koller, 2020; Bianchi et al., 2021a).”