On label dependence and loss minimization in multi-label classification

By Krzysztof Dembczynski, Willem Waegeman, Weiwei Cheng, Eyke Hüllermeier. Machine-mediated learning, 2012

330 citations recorded by the source corpus.

Most of the multi-label classification (MLC) methods proposed in recent years intended to exploit, in one way or the other, dependencies between the class labels. Comparing to simple binary relevance learning as a baseline, any gain in performance is normally explained by the fact that this method is ignoring such dependencies. Without questioning the correctness of such studies, one has to admit that a blanket explanation of that kind is hiding many subtle details, and indeed, the underlying mechanisms and true reasons for the improvements reported in experimental studies are rarely laid bare. Rather than proposing yet another MLC algorithm, the aim of this paper is to elaborate more closely on the idea of exploiting label dependence, thereby contributing to a better understanding of MLC. Adopting a statistical perspective, we claim that two types of label dependence should be distinguished, namely conditional and marginal dependence. Subsequently, we present three scenarios in which the exploitation of one of these types of dependence may boost the predictive performance of a classifier. In this regard, a close connection with loss minimization is established, showing that the benefit of exploiting label dependence does also depend on the type of loss to be minimized. Concrete theoretical results are presented for two representative loss functions, namely the Hamming loss and the subset 0/1 loss. In addition, we give an overview of state-of-the-art decomposition algorithms for MLC and we try to reveal the reasons for their effectiveness. Our conclusions are supported by carefully designed experiments on synthetic and benchmark data.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/a819ad955c83f2707b1d5ad4e5d45eeb9de43bdf

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Binary relevance for multi-label learning: an overview (2018, background): “Therefore, the actual multi-label metric being optimized depends on the binary loss, which is minimized by the binary learning algorithmB [46,47]; • Finally, binary relevance can be easily adapted to learn from multi-label examples with missing labels, where the labeling information for training exa”
DeepCoder: Learning to Write Programs (2016, methodology): “(Dembczy´nski et al., 2012) Training the neural network with the negative cross entropy loss function as the training objective is precisely a method for properly estimating the marginal probabilities of labels (functions appearing in source code).”
Explainable and Interpretable Models in Computer Vision and Machine Learning (2017, background): “As already explained, one commonly distinguishes between conditional and unconditional label dependencies (Dembczyński et al. 2012), where the former is of a global nature and holds (unconditionally) in the entire instance space (regardless of any features of the instances), whereas the latter is of”
Remarks on multi-output Gaussian process regression (2018, background): “Besides, some theoretical insights [68, 69, 70] pointed out that the consideration of additional features to transform the training data helps SST and ERC reduce the model bias at the cost of increasing the model variance.”