Training Deep Neural Networks via Direct Loss Minimization

By Yang Song, Alexander Gerhard Schwing, Richard S. Zemel, Raquel Urtasun. 2015

101 citations recorded by the source corpus.

Supervised training of deep neural nets typically relies on minimizing cross-entropy. However, in many domains, we are interested in performing well on metrics specific to the application. In this paper we propose a direct loss minimization approach to train deep neural networks, which provably minimizes the application-specific loss function. This is often non-trivial, since these functions are neither smooth nor decomposable and thus are not amenable to optimization with standard gradient-based methods. We demonstrate the effectiveness of our approach in the context of maximizing average precision for ranking problems. Towards this goal, we develop a novel dynamic programming algorithm that can efficiently compute the weight updates. Our approach proves superior to a variety of baselines in the context of action classification and object detection, especially in the presence of label noise.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Optimization & Training

Citations in the corpus

Few-Shot Learning Through an Information Retrieval Lens (2017, methodology): “Theorem 1 (General Loss Gradient Theorem from [20]).”
Towards Accurate One-Stage Object Detection With AP-Loss (2019, methodology): “Third, approximate gradient methods [31, 9] are proposed for optimizing the AP-loss, which are less efficient and easy to fall into local optimum even for the case of linear models due to the non-convexity and non-quasiconvexity of the AP-loss.”
Differentiation of Blackbox Combinatorial Solvers (2019, background): “In fact, the update rule from (5) (but not the function fλ or any of its properties) was already proposed in a different context in [20, 45] and was later used in [29, 33].”
Task-based End-to-end Model Learning (2017, unknown): “…both Stoyanov et al. (2011) and Hazan et al. (2010) proposed methods for optimizing loss criteria in structured prediction that are different from the inference procedure of the prediction algorithm; the work has also been recently extended to the setting of deep networks (Song et al., 2016).”
Deep Value Networks Learn to Evaluate and Iteratively Refine Structured Outputs (2017, background): “There has been a surge of recent interest in using neural networks for structured prediction (Zheng et al., 2015; Chen et al., 2015; Song et al., 2016).”
Task-based End-to-end Model Learning in Stochastic Optimization (2017, methodology): “[21] propose methods for optimizing loss criteria in structured prediction that are different from the inference procedure of the prediction algorithm; this work has also recently been extended to deep networks [22].”
Imbalance Problems in Object Detection: A Review (2019, methodology): “To make this differentiable, the authors propose a novel error-driven update rule, which performs better than the previous works aiming to include average precision as a training objective [97], [98].”
Deep Structured Prediction with Nonlinear Output Transformations (2018, methodology): “[48] took advantage of accurate local classification for tasks such as semantic segmentation by combining deep nets with graphical models in a two step procedure.”
Learning With Average Precision: Training Image Retrieval With a Listwise Loss (2019, methodology): “[54] then expanded this framework to work with non-linear models.”
End-to-End Training of Object Class Detectors for Mean Average Precision (2016, methodology): “Moreover, we do not require a potentially expensive max-oracle to find the most-violating inputs with respect to the model and loss, as required by [18, 19, 2].”
Direct Optimization through arg max for Discrete Variational Auto-Encoder (2019, methodology): “The score function h φ ( x, y ) can be non-linear as a function of the parameters φ , as developed by Song et al. [32].”
Learning to Teach with Dynamic Loss Functions (2018, background): “Specially designed loss functions play important roles in boosting the performances of real-world tasks, either by approximating the non-smooth task-specific objective such as 0-1 accuracy in classification [40], NDCG in ranking [49], BLEU in machine translation [45, 3] and MAP in object detection [”
Deep Structured Energy-Based Image Inpainting (2018, methodology): “In addition, DNNs have been used for structured prediction due to its generative power [1], [2], [5], [15]–[18].”