Self organization of a massive document collection

By Teuvo Kohonen, Samuel Kaski, Krista Lagus, J. Salojärvi, Jukka Honkela, V. Paatero, Antti Saarela. IEEE Trans. Neural Networks Learn. Syst., 2000

1,057 citations recorded by the source corpus.

This article describes the implementation of a system that is able to organize vast document collections according to textual similarities. It is based on the self-organizing map (SOM) algorithm. As the feature vectors for the documents statistical representations of their vocabularies are used. The main goal in our work has been to scale up the SOM algorithm to be able to deal with large amounts of high-dimensional data. In a practical experiment we mapped 6,840,568 patent abstracts onto a 1,002,240-node SOM. As the feature vectors we used 500-dimensional vectors of stochastic figures obtained as random projections of weighted word histograms.

Subfields: Artificial Intelligence, Information Systems

View Paper: https://www.semanticscholar.org/paper/132095e9df1b990fed7e624ac57ac8833dfb16ac

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

Citations in the corpus

Overview and comparative study of dimensionality reduction techniques for high dimensional data (2020, neutral): “Real life applications of SOM include automatic organization of a massive collection of documents [230] , intrusion detection [231] , development of risk-based prioritization for stagnation [229] , recognition of protein folds [232] , market data analysis [233] , the classiﬁcation of fMRI [4] ,…”
Review of "Mining the Web: discovering knowledge from hypertext data" by Soumen Chakrabati. Morgan Kaufman 2003. (2002, neutral): “For an extensive treatment of self-organizing maps together with applications to Web and newsgroup data see Kohonen et al [121] and visit http://websom.”
A Survey of Dimension Reduction Techniques (2002, methodology): “It has been shown empirically that results with the random projection method are comparable with results obtained with PCA, and take a fraction of the time PCA requires [33, 35].”
Bibliography of Self-Organizing Map SOM) Papers: 1998-2001 Addendum (2003, neutral): “1726] counter-propagation [491, 1206, 1305] cross-validation [69, 173, 223, 1684, 1811] cursive character [167, 610, 1227] curvilinear component analysis (CCA) [530, 1013, 1875] data fusion [159, 589, 1017, 1061, 1545, 1603, 1746, 1750, 1917, 2065, 2066, 2068, 2069] data mining [17, 16, 18, 215, 304”
Spoken Content Retrieval—Beyond Cascading Speech Recognition with Text Retrieval (2015, methodology): “Global semantic structuring has been widely studied for text retrieval and document archives, with WebSOM [342] and ProbMap [343] as good examples, in which the relationships among document clusters are visualized as a two-dimensional map.”
The growing hierarchical self-organizing map: exploratory analysis of high-dimensional data (2002, neutral): “The model also has a strong tradition in the text mining area where a number of research groups described work based on the SOM [ 10 ]‐[14].”
Essentials of the self-organizing map (2013, methodology): “The size of the text corpus in this application (Kohonen et al., 2000) was about 20 times that of Encyclopaedia Britannica.”
Survey of clustering algorithms (2005, neutral): “An application for clustering of a large set of documental data is illustrated in [170], in which 6 840 568 patent abstracts were”
A Brief Survey of Text Mining (2005, methodology): “A reﬁned model, the WEBSOM approach, extended this idea to a web based interface applied to newsgroup data that provides simple zooming techniques and coloring methods [HKLK96, Hon97, KKL + 00].”
Generalizing self-organizing map for categorical data (2006, background): “The applications initially focused on engineering, including image processing [4]–[7], process monitoring and control [8], [9], speech recognition [10], [11], and flaw detection in machinery [12].”
Evidence for the network perspective on organizational learning (2008, background): “There is a wide spectrum of available informationvisualization techniques and applications, the main examples of which are Pathfinder associative networks (Chen, 1999; Fowler, Fowler, & Wilson, 1991; Schvaneveldt, Durso, & Dearholt, 1989;), self-organizing maps (Kohonen, 2001; Kohonen et al., 2000; ”