Learning Graphical Model Parameters with Approximate Marginal Inference

By Justin Domke. 2013

188 citations recorded by the source corpus.

Likelihood-based learning of graphical models faces challenges of computational complexity and robustness to model misspecification. This paper studies methods that fit parameters directly to maximize a measure of the accuracy of predicted marginals, taking into account both model and inference approximations at training time. Experiments on imaging problems suggest marginalization-based learning performs better than likelihood-based approximations on difficult problems where the model being fit is approximate in nature.

Subfields: Artificial Intelligence, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Probabilistic & Bayesian Methods

Citations in the corpus

Barrier Frank-Wolfe for Marginal Inference (2015, background): “(3) Restricted Boltzmann Machines (RBMs): From the Probabilistic Inference Challenge 2011.3 (4) Horses: Large (N ≈ 12000) MRFs representing images from the Weizmann Horse Data (Borenstein and Ullman, 2002) with potentials learned by Domke (2013).”
Conditional Random Fields as Recurrent Neural Networks (2015, methodology): “ and joint training of classiers and graphical models. Barbu et al . [4] proposed a joint training of a MRF/CRF model together with an inference algorithm in their Active Random Field approach. Domke [15] advocated back-propagation based parameter optimization in graphical models when approximate in”
Higher Order Conditional Random Fields in Deep Neural Networks (2015, background): “Some early works that advocated gradient backpropagation through graphical model inference for parameter optimisation include [9,19,25] and [41].”
Structured Prediction Energy Networks (2015, methodology): “Alter-natively, non-iterative feed-forward predictors can be constructed using structured models as motivation (Stoyanov et al., 2011; Domke, 2013; Kunisch & Pock, 2013; Her-shey et al., 2014; Li & Zemel, 2014; Zheng et al., 2015).”
Deep Unfolding: Model-Based Inspiration of Novel Deep Architectures (2014, methodology): “In [12, 13], tree-reweighted belief propagation (TRW-BP) and mean ﬁeld inference were unrolled and trained via gradient descent.”
Learning Sparse High Dimensional Filters: Image Filtering, Dense CRFs and Bilateral Neural Networks (2015, background): “Several works [32, 17, 29, 49, 39] demonstrate how to learn free parameters of the dense CRF model.”
Efficient Likelihood Learning of a Generic CNN-CRF Model for Semantic Segmentation (2015, background): “s the approach of (Zheng et al., 2015) and (Schwing & Urtasun, 2015), there are many other works that consider the idea of backpropagation with a so-called unrolled CRF-inference scheme, such as (Domke, 2013; Kiefel & Gehler, 2014; Barbu, 2009; Ross et al., 2011; Stoyanov et al., 2011; Tomps”
The structure of optimal parameters for image restoration problems (2015, background): “In [41, 42, 27, 28, 18, 19], for instance the authors consider bilevel optimization for finite dimensional Markov random field (MRF) models.”
Human Pose Estimation with Fields of Parts (2014, methodology): “This is an application of the Back-Mean-Field idea of [8], a procedure advocated for learning with approximate inference when predicting with marginal inference.”
Learning Latent Permutations with Gumbel-Sinkhorn Networks (2018, methodology): “With this in mind, backpropagation through Sinkhorn is an end-to-end learning in an unrolled inference algorithm Stoyanov et al. (2011); Domke (2013).”
Mutual CRF-GNN for Few-shot Learning (2021, background): “Conditional Random Field (CRF) [27, 11, 12, 10, 6] is a popular probabilistic model to infer various dependencies within a image group or pixels in a image.”
Joint Training of Generic CNN-CRF Models with Stochastic Optimization (2015, background): “Besides the approaches [11] and [12], there are many other works that consider the idea of backpropagation with a so-called unrolled CRF-inference scheme, such as [14,15,16,17,18,19,20].”
Learning Binary Trees via Sparse Relaxation (2020, background): “This situation arises in as diverse scenarios as hyperparameter optimization (Pedregosa, 2016), meta-learning (Rajeswaran et al., 2019), or structured prediction (Stoyanov et al., 2011; Domke, 2013; Niculae et al., 2018).”
Deep Learning and Structured Prediction for the Segmentation of Mass in Mammograms (2015, methodology): “TRW solves these issues by first replacing the marginal polytope with a superset L ⊃ M that only accounts for the local constraints of the marginals, and then approximating the entropy calculation with an upper bound [7].”
Higher Order Potentials in End-to-End Trainable Conditional Random Fields (2015, background): “Some early works that advocated gradient backpropagation through graphical model inference for parameter optimization include [35, 8, 22] and [17].”
DeepFruits: A Fruit Detection System Using Deep Neural Networks (2016, methodology): “In prior work [4], we demonstrated that using a CRF [27] to model colour and visual texture features from multi-spectral images led to the impressive performance for sweet pepper segmentation.”
A deep learning approach for the analysis of masses in mammograms with minimal user intervention (2017, methodology): “Finally, the pairwise potential functions between abel nodes in (8) encode label and contrast dependent labelling omogeneity as ψ (2 , 1) ( ̂ y(v ) , ̂ y(q ) , ̂ x) and ψ (2 , 1+ n ) ( ̂ y(v ) , ̂ y(q ) , ̂ x) repectively ( Nowozin and Lampert, 2011; Domke, 2013; Dhungel t al., 2015d ).”