Large-Scale Multi-label Text Classification - Revisiting Neural Networks

By Jinseok Nam, Jungi Kim, Eneldo Loza Mencía, Iryna Gurevych, Johannes Fürnkranz. 2013

384 citations recorded by the source corpus.

Neural networks have recently been proposed for multi-label classification because they are able to capture and model label dependencies in the output layer. In this work, we investigate limitations of BP-MLL, a neural network (NN) architecture that aims at minimizing pairwise ranking error. Instead, we propose to use a comparably simple NN approach with recently proposed learning techniques for large-scale multi-label text classification tasks. In particular, we show that BP-MLL's ranking loss minimization can be efficiently and effectively replaced with the commonly used cross entropy error function, and demonstrate that several advances in neural network training that have been developed in the realm of deep learning can be effectively employed in this setting. Our experimental results show that simple NN models equipped with advanced techniques such as rectified linear units, dropout, and AdaGrad perform as well as or even outperform state-of-the-art approaches on six large-scale textual datasets with diverse characteristics.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Elucidating image-to-set prediction: An analysis of models, losses and datasets (2019, background): “Multi-label classification has been a long lasting problem in computer vision [72, 21, 73, 67] tackled by a wide variety of methods, such as, decomposing the problem into independent single-label classification problems [44, 72], exploiting label co-occurrences [1, 56, 34], introducing priors such a”
AdapterFusion: Non-Destructive Task Composition for Transfer Learning (2020, background): “In this section, we formalize our goal of transfer learning (Pan and Yang, 2010; Torrey and Shavlik, 2010; Ruder, 2019), highlight its key challenges, and provide a brief overview of common methods that can be used to address them.”
Maximizing Subset Accuracy with Recurrent Neural Networks in Multi-label Classification (2017, methodology): “For preparing the train and the test set of Reuters-21578 and RCV1-v2, we follow [21].”
Multi-label Zero-Shot Learning with Structured Knowledge Graphs (2017, methodology): “As non-linear embedding approaches, deep neural networks have also been utilized for multi-label classification [50, 34, 18, 46, 45, 49].”
Deep Streaming Label Learning (2020, methodology): “For example, a document may belong to various topics (Zhang & Zhou, 2006; Nam et al., 2014); an image can contain several individuals’ faces (Xiao et al.”
CU-Net: Component Unmixing Network for Textile Fiber Identification (2019, methodology): “Nam et al. (2014) treated each label independently and use a binary logistic loss to predict each label.”
Probabilistic Label Tree for Streaming Multi-Label Learning (2021, methodology): “• DNN-BCE [16] makes use of ReLU, AdaGrad, Dropout, and other deep learning techniques to train a DNN-based model.”
The Emerging Trends of Multi-Label Learning (2020, methodology): “To address the issue, based on BP-MLL, [136] proposes to use a comparably simple NN approach that can achieve the state-of-the-art performance in large-scale multi-label text classification.”
Learning to Predict Charges for Criminal Cases with Legal Basis (2017, background): “Another difference is the multi-label nature of our task, where, rather than optimizing as multiple binary classiﬁcation tasks (Nam et al., 2014), we convert the multi-label target to label distribution during training with cross entropy as loss function (Ku-rata et al., 2016), and use a threshold…”
Segmenting Retinal Blood Vessels With Deep Neural Networks (2016, background): “The recent extension of this approach [36] shows that more efficient and more effective training can be attained by replacing BP-MLL's pairwise ranking loss with cross entropy (CE) loss defined as:”