Label-Free Supervision of Neural Networks with Physics and Domain Knowledge

By Russell Stewart, Stefano Ermon. 2016

362 citations recorded by the source corpus.

In many machine learning applications, labeled data is scarce and obtaining more labels is expensive. We introduce a new approach to supervising neural networks by specifying constraints that should hold over the output space, rather than direct examples of input-output pairs. These constraints are derived from prior domain knowledge, e.g., from known laws of physics. We demonstrate the effectiveness of this approach on real world and simulated computer vision tasks. We are able to train a convolutional neural network to detect and track objects without any labeled examples. Our approach can significantly reduce the need for labeled training data, but introduces new challenges for encoding prior knowledge into appropriate loss functions.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning, AI for Physical Sciences

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

In AI for Physical Sciences

Citations in the corpus

Snorkel: Rapid Training Data Creation with Weak Supervision (2017, background): “Other forms include crowdsourced labels [37, 50], rules and heuristics for labeling data [39, 52], and others [29, 30, 30, 46, 51].”
Time-Contrastive Networks: Self-Supervised Learning from Video (2017, background): “[37] explore various types of physical priors, such as the trajectories of objects falling under gravity, to learn object tracking without explicit supervision.”
Visual Interaction Networks: Learning a Physics Simulator from Video (2017, methodology): “A third class of methods [5, 8, 9, 23], like our Visual Interaction Network, have been used to predict future state descriptions from pixels.”
Learning to Perform Physics Experiments via Deep Reinforcement Learning (2016, neutral): “The approach of encoding physical models as architecture constraints has also been proposed by Stewart & Ermon (2016).”
Optimization with Non-Differentiable Constraints with Applications to Fairness, Recall, Churn, and Other Goals (2018, background): “Another type of constrained machine learning aims to constrain the model parameters to obey known physical limits on the learned system (e.g. Long et al. (2018); Stewart and Ermon (2017)).”
Surrogate modeling for fluid flows based on physics-constrained deep learning without simulation data (2019, background): “, for solving a handful of computer vision problems [48], deterministic PDEs [33, 49, 50], high-dimensional stochastic partial differential equations (SPDE), and backward stochastic differential equations (BSDE) [51–55].”
Small Sample Learning in Big Data Era (2018, methodology): “For example, Stewart and Ermon (2017) introduced a new method for using physics and other domain constraints to supervise neural networks, detecting and tracking objects without any labeled examples.”
Generating Liquid Simulations with Deformation-aware Neural Networks (2017, background): “In addition, the unsupervised inference of physical functions poses interesting problems [37].”
Explainable Machine Learning for Scientific Insights and Discoveries (2019, background): “For example, [53] detects and tracks objects in videos in an unsupervised way.”
Knowledge-based Recurrent Attentive Neural Network for Traffic Sign Detection (2018, background): “[27] trained a convolutional neural network to detect and track objects without any labeled examples.”
A Semantic Loss Function for Deep Learning with Symbolic Knowledge (2017, background): “…arithmetic constraints (Pathak et al., 2015; M ´ arquez-Neila et al., 2017) or logical constraints (Rockt ¨ aschel et al., 2015; Hu et al., 2016; Demeester et al., 2016; Stewart & Ermon, 2017; Minervini et al., 2017; Diligenti et al., 2017; Donadello et al., 2017) on the output of a neural network.”
PVEs: Position-Velocity Encoders for Unsupervised Learning of Structured State Representations (2017, background): “A powerful but underexplored alternative is to express preference biases directly on the learned representation, an approach which can enable to learn symbol grounding [6] or perform label-free supervised learning [18].”
Snorkel: rapid training data creation with weak supervision (2019, background): “Other forms include crowdsourced labels [41,63], rules and heuristics for labeling data [47,65], and others [33,34,34,55,64].”
Time-Contrastive Networks: Self-Supervised Learning from Multi-view Observation (2017, background): “[25] explore various types of physical priors, such as the trajectories of objects falling under gravity, to learn object tracking without explicit supervision.”
Machine Learning for the Geosciences: Challenges and Opportunities (2017, background): “ a better chance against overﬁtting, especially when training data are scarce. For example, a recent work explored the use of physics-guided loss functions for tracking objects in sequences of images [112], where elementary knowledge of laws of motion was solely used for constraining outputs and lea”
Efficiently Guiding Imitation Learning Algorithms with Human Gaze (2020, neutral): “(Stewart and Ermon 2017).”
Deep Learning in Mobile and Wireless Networking: A Survey (2018, background): “Applications include clustering [91], data distributions approximation [90], un/semi-supervised learning [92], [93], and one/zero shot learning [94], [95], among others.”