A re-examination of text categorization methods

By Yiming Yang, Xin Liu. 1999

This paper reports a controlled study with statistical signi cance tests on ve text categorization methods: the Support Vector Machines (SVM), a k-Nearest Neighbor (kNN) classi er, a neural network (NNet) approach, the Linear Leastsquares Fit (LLSF) mapping and a Naive Bayes (NB) classier. We focus on the robustness of these methods in dealing with a skewed category distribution, and their performance as function of the training-set category frequency. Our results show that SVM, kNN and LLSF signi cantly outperform NNet and NB when the number of positive training instances per category are small (less than ten), and that all the methods perform comparably when the categories are su ciently common (over 300 instances).

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Text Categorization as a Graph Classification Problem (2015, background): “Because we are dealing with single-label classiﬁcation, the micro-average F1-score corresponds to the accuracy and is a measure of the overall prediction effectiveness (Yang and Liu, 1999).”
Question classification using support vector machines (2003, methodology): “The statistical sign test can be applied for comparing two methods A and B [22].”
Geolocation Prediction in Social Media Data by Finding Location Indicative Words (2012, background): “We 13Although there are many other classifiers we could consider, when sufficient training data for each class is available, the performance of different methods is comparable (Yang and Liu, 1999).”
TEXT CLASSIFICATION AND CLASSIFIERS: A SURVEY (2012, methodology): “The works in [5] [54] compare the most common method in most cases support machine and K-nearest neighbor have better effect neural network is after then and then naïve bays is last and its evaluation index is again break –even point”
Personalized Web search for improving retrieval effectiveness (2004, unknown): “Based on our experiments with users, for almost all queries, each query is related to only one or two categories.”
Self-taught hashing for fast similarity search (2010, methodology): “SVM in its simplest form, linear SVM f(x) = sgn(wx) consistently provides state-of-the-art performance for text classification tasks [10, 22, 49].”
Think about it! Improving defeasible reasoning by first modeling the question scenario. (2021, methodology): “We perform two statistical tests for verifying our results: i) The micro-sign test (s-test) (Yang and Liu, 1999), and ii) McNermar’s test (Dror et al., 2018).”
An evaluation of statistical spam filtering techniques (2004, background): “Several attempts have been made to evaluate the performance of machine-learning methods on general text categorization task where there are ten or more categories [Yang and Liu 1999; Yang and Pedersen 1997].”
A Survey of Text Classification Algorithms (2012, methodology): “For example, it is possible to create non-linear decision surfaces with the SVM classifier, just as it is possible to create non-linear separation boundaries by using layered neurons in a neural network [132].”
Analogical Inference for Multi-relational Embeddings (2017, methodology): “We used one-sample proportion test (Yang & Liu, 1999) at the 5% p-value level for testing the statistical signiﬁcances 3 .”
Mining Text Data (2012, methodology): “For example, it is possible to create non-linear decision surfaces with the SVM classifier, just as it is possible to create non-linear separation boundaries by using layered neurons in a neural network [132].”
Sentiment analysis: A combined approach (2009, methodology): “Hence, it is common that macro averaged performance is lower than micro averaged performance, as shown in a classification performance evaluation conducted by Yang & Liu (1999) and Calvo & Ceccatto (2000).”
Ensemble of feature sets and classification algorithms for sentiment classification (2011, background): “Due to the dimension of feature space is quite large in text classification tasks, the classification problem is always linearly separable [39], therefore linear kernel is commonly used.”
A novel feature selection algorithm for text categorization (2007, methodology): “For example, k-nearest neighbor (kNN) (Cover & Hart, 1967; Yang, 1997; Yang & Lin, 1999; Tan, 2005), Naive Bayes (Lewis, 1998), decision tree (Lewis & Ringuette, 1994), support vector machines (SVM) (Joachims, 1998), linear least squares ﬁt, neural network, SWAP-1, and Roc-chio are all such kinds…”
Hierarchical document categorization with support vector machines (2004, methodology): “In all our experiments on WIPO-alpha, the learned solutions were very sparse, usually with an active ratio in the range of [0.005,0.1].”