Imposing Hard Constraints on Deep Networks: Promises and Limitations

By Pablo Márquez-Neila, Mathieu Salzmann, Pascal Fua. 2017

166 citations recorded by the source corpus.

Imposing constraints on the output of a Deep Neural Net is one way to improve the quality of its predictions while loosening the requirements for labeled training data. Such constraints are usually imposed as soft constraints by adding new terms to the loss function that is minimized during training. An alternative is to impose them as hard constraints, which has a number of theoretical benefits but has not been explored so far due to the perceived intractability of the problem. In this paper, we show that imposing hard constraints can in fact be done in a computationally feasible way and delivers reasonable results. However, the theoretical benefits do not materialize and the resulting technique is no better than existing ones relying on soft constraints. We analyze the reasons for this and hope to spur other researchers into proposing better solutions.

Subfields: Artificial Intelligence, Computer Networks and Communications, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Networks and Communications

In Computer Vision and Pattern Recognition

Citations in the corpus

Physics informed deep learning for computational elastodynamics without labeled data (2020, background): “This type of I/BC imposition is also called “soft” enforcement [17, 54] because the initial and boundary values may not be enforced accurately, due to the pathology issue of the gradient.”
A Deep Neural Network Surrogate for High-Dimensional Random Partial Differential Equations (2018, background): “Boundary or initial constraints in such formulation can in general be treated as soft or hard constraints [38].”
Deep Declarative Networks (2019, methodology): “ories. Promising resultsareachievedon standardbenchmarksand they report low training overhead in that solving the QP takes about the same time as image feature extraction. Last, Ma´rquez-Neila et al. [29] investigate the problem of imposing hard constraints on the output of a deep neural network usi”
Surrogate modeling for fluid flows based on physics-constrained deep learning without simulation data (2019, background): “7) which are often treated in a “soft” manner by modifying the original loss function with penalty terms [33, 64].”
A Semantic Loss Function for Deep Learning with Symbolic Knowledge (2017, background): “Various deep learning techniques have been proposed to enforce either arithmetic constraints (Pathak et al., 2015; Márquez-Neila et al., 2017) or logical constraints (Rocktäschel et al.”
Under review as a conference paper at ICLR 2020 3 L INEAR INEQUALITY CONSTRAINTS FOR DEEP LEARNING MODELS We consider a generic L layer neural network (2019, unknown): “Márquez-Neila et al. (2017) formulated generic differentiable equality constraints as soft constraints and employed a Lagrangian approach to train their model.”
Characterizing possible failure modes in physics-informed neural networks (2021, background): “Previous work has tried to impose hard constraints in ML (both within the context of SciML and otherwise) [6, 21, 22, 24, 40], although this can be computationally expensive, and does not guarantee better results or convergence.”
Enforcing Analytic Constraints in Neural Networks Emulating Physical Systems. (2019, background): “Despite the need for physically informed NNs for complex physical systems [2–5], enforcing hard constraints [6] has been limited to physical systems governed by specific equations, such as advection equations [7–9], Reynolds-averaged Navier-Stokes equations [10,11], boundary conditions of idealized ”
Physics-informed machine learning: case studies for weather and climate modelling (2021, background): “Custom-designed NN architectures are a powerful approach to incorporating physics because constraints can be strictly enforced, including in new scenarios [19].”