Learning Classifiers with Fenchel-Young Losses: Generalized Entropies, Margins, and Algorithms

By Mathieu Blondel, André F. T. Martins, Vlad Niculae. 2018

55 citations recorded by the source corpus.

This paper studies Fenchel-Young losses, a generic way to construct convex loss functions from a regularization function. We analyze their properties in depth, showing that they unify many well-known loss functions and allow to create useful new ones easily. Fenchel-Young losses constructed from a generalized entropy, including the Shannon and Tsallis entropies, induce predictive probability distributions. We formulate conditions for a generalized entropy to yield losses with a separation margin, and probability distributions with sparse support. Finally, we derive efficient algorithms, making Fenchel-Young losses appealing both in theory and practice.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Efficient Content-Based Sparse Attention with Routing Transformers (2020, neutral): “This formulation however does require instantiating a full dense attention matrix prior to sparsification through variants of L0-sparsity or sparsemax approximations (Blondel et al., 2019).”
DSelect-k: Differentiable Selection in the Mixture of Experts with Applications to Multi-Task Learning (2021, neutral): “, the sparsemax [24] and its generalizations [27, 6, 3].”
Adaptively Sparse Transformers (2019, background): “We focus on a recently-introduced ﬂexible family of transformations, α -entmax (Blondel et al., 2019; Peters et al., 2019), deﬁned as: where d := { p ∈ R d : i p i = 1 } is the probability simplex , and, for α ≥ 1 , H T α is the Tsallis continuous family of entropies (Tsallis, 1988): (5) This…”
Fast Differentiable Sorting and Ranking (2020, background): “Proposition 1 of Blondel et al. (2019) shows that µ?”
SparseMAP: Differentiable Sparse Structured Inference (2018, background): “This family, studied in more detail in (Blondel et al., 2018), includes the commonly-used structured losses:”
Sparse Sequence-to-Sequence Models (2019, methodology): “(11) The motivation for this loss function resides in the fact that it is a Fenchel-Young loss (Blondel et al., 2019), as we briefly explain in Appendix A.2.”