Document clustering using word clusters via the information bottleneck method

By Noam Slonim, Naftali Tishby. Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, 2000

554 citations recorded by the source corpus.

We present a novel implementation of the recently introduced information bottleneck method for unsupervised document clustering. Given a joint empirical distribution of words and documents, p(x, y), we first cluster the words, Y, so that the obtained word clusters, Ytilde;, maximally preserve the information on the documents. The resulting joint distribution. p(X, Ytilde;), contains most of the original information about the documents, I(X; Ytilde;) ≈ I(X; Y), but it is much less sparse and noisy. Using the same procedure we then cluster the documents, X, so that the information about the word-clusters is preserved. Thus, we first find word-clusters that capture most of the mutual information about to set of documents, and then find document clusters, that preserve the information about the word clusters. We tested this procedure over several document collections based on subsets taken from the standard 20Newsgroups corpus. The results were assessed by calculating the correlation between the document clusters and the correct labels for these documents. Finding from our experiments show that this double clustering procedure, which uses the information bottleneck method, yields significantly superior performance compared to other common document distributional clustering algorithms. Moreover, the double clustering procedure improves all the distributional clustering methods examined here.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/4164fb501998682d56d54e2a1fdfc6d46aee83a3

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Information-theoretic co-clustering (2003, methodology): “We applied the same pre-processing steps as in [19] to all the subsets, i.”
Information Bottleneck for Gaussian Variables (2003, methodology): “It has been proved useful for a range of applications from documents clustering (Slonim and Tishby, 2000) through neural code analysis (Dimitrov and Miller, 2001) to gene expression analysis (Friedman et al.”
Caveats for information bottleneck in deterministic scenarios (2018, unknown): “There are also numerous application domains of IB beyond supervised learning, including clustering (Slonim & Tishby, 2000), coding theory and quantization (Cardinal, 2003; Zeitler et al., 2008; Courtade & Wesel, 2011), and cognitive science (Zaslavsky et al., 2018).”
The Information Bottleneck : Theory and Applications (2006, methodology): “The probability values were estimated to be proportional to thenumber of times a specific verb occurred with the specific noun in the same sentence.”
Multivariate Information Bottleneck (2001, background): “at the same time identify cluster of words that characterize these topics [11].”
Disambiguating Web appearances of people in a social network (2005, methodology): “Slonim and Tishby [19] propose a greedy agglomerative al-gorithm for document clustering based on the Information Bottleneck method, where X stands for documents and Y stands for words in the documents.”
PAC-Bayesian Analysis of Co-clustering and Beyond (2010, methodology): “A typical example of this kind of problem is the analysis of word-document co-occurrence matrices in text mining (Slonim and Tishby, 2000; El-Yaniv and Souroujon, 2001; Dhillon et al., 2003).”
Solving cluster ensemble problems by bipartite graph partitioning (2004, methodology): “First, document clustering with keyword clustering has been shown to yield better performance than without keyword clustering (Slonim & Tishby, 2000).”
Unsupervised document classification using sequential information maximization (2002, methodology): “• The AIB results are significantly better than those reported in [14] (P(T)=45.”
/spl delta/-clusters: capturing subspace correlation in a large data set (2002, background): “Despite other differences in the problem formation, this approach significantly differs from our model on the following aspect: our model clusters the attributes (words) and objects (documents) simultaneously while [10] clusters the attributes (words) and objects (documents) sequentially.”
Using Topic Modeling Methods for Short-Text Data: A Comparative Analysis (2020, methodology): “• PCA is an essential tool for text processing tasks, and it has been used since the early 1990s (Jolliﬀe, 1986; Slonim and Tishby, 2000; Gomez et al., 2012).”
Learning human actions via information maximization (2008, methodology): “We adopt the Information Bottleneck [18].”
Data clustering: 50 years beyond K-means (2008, unknown): “We will highlight and illustrate some of these challenges below.”
An Evaluation on Feature Selection for Text Clustering (2003, methodology): “Feature extraction is a process that extracts a set of new features from the original features through some functional mapping (Wyse et al., 1980), such as principal component analysis (PCA) (Jolliffe, 1986) and word clustering ( Slonim & Tishby, 2000 ).”