An Exploration of Softmax Alternatives Belonging to the Spherical Loss Family

By Alexandre de Brébisson, Pascal Vincent. 2015

105 citations recorded by the source corpus.

In a multi-class classification problem, it is standard to model the output of a neural network as a categorical distribution conditioned on the inputs. The output must therefore be positive and sum to one, which is traditionally enforced by a softmax. This probabilistic mapping allows to use the maximum likelihood principle, which leads to the well-known log-softmax loss. However the choice of the softmax function seems somehow arbitrary as there are many other possible normalizing functions. It is thus unclear why the log-softmax loss would perform better than other loss alternatives. In particular Vincent et al. (2015) recently introduced a class of loss functions, called the spherical family, for which there exists an efficient algorithm to compute the updates of the output weights irrespective of the output size. In this paper, we explore several loss functions from this family as possible alternatives to the traditional log-softmax. In particular, we focus our investigation on spherical bounds of the log-softmax loss and on two spherical log-likelihood losses, namely the log-Spherical Softmax suggested by Vincent et al. (2015) and the log-Taylor Softmax that we introduce. Although these alternatives do not yield as good results as the log-softmax loss on two language modeling tasks, they surprisingly outperform it in our experiments on MNIST and CIFAR-10, suggesting that they might be relevant in a broad range of applications.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Management Science and Operations Research

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Management Science and Operations Research

Citations in the corpus

Secure Evaluation of Quantized Neural Networks (2019, background): “More MPC friendly solutions exist, such as the spherical Softmax [18], which replaces e x with x 2 .”
From Softmax to Sparsemax: A Sparse Model of Attention and Multi-Label Classification (2016, background): “…1952; Zadrozny, 2001; Menke & Martinez, 2008), the multinomial probit (Albert & Chib, 1993), the spherical softmax (Ollivier, 2013; Vincent, 2015; de Br´ebisson & Vincent, 2015), or softmax approximations (Bouchard, 2007), while theoretically or computationally advantageous for certain…”
Sigsoftmax: Reanalysis of the Softmax Bottleneck (2018, methodology): “The study of de Brébisson and Vincent [7] explored spherical family functions: the spherical softmax and Taylor softmax.”
Sampled Softmax with Random Fourier Features (2019, methodology): “This includes different modified loss functions: hierarchical softmax [5] partitions the classes into a tree based on class similarities, allowing for O(d log n) training and inference time; spherical softmax [6, 7] replaces the exponential function by a quadratic function, enabling efficient algori”