Topic Modeling for Short Texts with Auxiliary Word Embeddings

By Chenliang Li, Haoran Wang, Zhiqian Zhang, Aixin Sun, Zongyang Ma. Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, 2016

318 citations recorded by the source corpus.

For many applications that require semantic understanding of short texts, inferring discriminative and coherent latent topics from short texts is a critical and fundamental task. Conventional topic models largely rely on word co-occurrences to derive topics from a collection of documents. However, due to the length of each document, short texts are much more sparse in terms of word co-occurrences. Data sparsity therefore becomes a bottleneck for conventional topic models to achieve good results on short texts. On the other hand, when a human being interprets a piece of short text, the understanding is not solely based on its content words, but also her background knowledge (e.g., semantically related words). The recent advances in word embedding offer effective learning of word semantic relations from a large corpus. Exploiting such auxiliary word embeddings to enrich topic modeling for short texts is the main focus of this paper. To this end, we propose a simple, fast, and effective topic model for short texts, named GPU-DMM. Based on the Dirichlet Multinomial Mixture (DMM) model, GPU-DMM promotes the semantically related words under the same topic during the sampling process by using the generalized Polya urn (GPU) model. In this sense, the background knowledge about word semantic relatedness learned from millions of external documents can be easily exploited to improve topic modeling for short texts. Through extensive experiments on two real-world short text collections in two languages, we show that GPU-DMM achieves comparable or better topic representations than state-of-the-art models, measured by topic coherence. The learned topic representation leads to the best accuracy in text classification task, which is used as an indirect evaluation.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/b3625e35fe7de551d628dd4b3875d2ba847f1097

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Yuan Yunhao
2. Apurva Shah
3. Li Yun
4. Jipeng Qiang
5. Uttam Chauhan
6. Wu Xindong
7. Qian Zhenyu

Citations in the corpus

• Short Text Topic Modeling Techniques, Applications, and Performance: A Survey (2019, methodology): “Recently, more work incorporates word embeddings into DMM [29], [31].”
• Topic Modeling Using Latent Dirichlet allocation (2021, background): “Li et al. exploited neural network-based language modeling to fit word embedding for modeling topics over the short text [73].”