Minimax Entropy Principle and Its Application to Texture Modeling

By Song-Chun Zhu, Ying Nian Wu, David Mumford. 1997

This article proposes a general theory and methodology, called the minimax entropy principle, for building statistical models for images (or signals) in a variety of applications. This principle consists of two parts. The first is the maximum entropy principle for feature binding (or fusion): for a given set of observed feature statistics, a distribution can be built to bind these feature statistics together by maximizing the entropy over all distributions that reproduce them. The second part is the minimum entropy principle for feature selection: among all plausible sets of feature statistics, we choose the set whose maximum entropy distribution has the minimum entropy. Computational and inferential issues in both parts are addressed; in particular, a feature pursuit procedure is proposed for approximately selecting the optimal set of features. The minimax entropy principle is then corrected by considering the sample variation in the observed feature statistics, and an information criterion for feature pursuit is derived. The minimax entropy principle is applied to texture modeling, where a novel Markov random field (MRF) model, called FRAME (filter, random field, and minimax entropy), is derived, and encouraging results are obtained in experiments on a variety of texture images. The relationship between our theory and the mechanisms of neural computation is also discussed.

Subfields: Computer Vision and Pattern Recognition, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Probabilistic & Bayesian Methods

Citations in the corpus

Learning Energy-Based Spatial-Temporal Generative ConvNets for Dynamic Patterns (2019, methodology): “The FRAME (Filters, Random fields, And Maximum Entropy) model [31], [32], [33] is a Markov”
A Theory of Generative ConvNet (2016, methodology): “(Lu et al., 2016) proposed to learn the FRAME (Filters, Random field, And Maximum Entropy) models (Zhu et al., 1997) based on the pre-learned filters of existing ConvNets.”
Learning FRAME Models Using CNN Filters (2015, methodology): “A linear ﬁlter F k can be written as a two-dimensional function The original FRAME model (Zhu, Wu, and Mumford, 1997) for texture patterns is a stationary or spatially homogeneous Markov random ﬁeld or Gibbs distribution of the following form: where λ k () is a nonlinear function to be estimated…”
Wasserstein Introspective Neural Networks (2017, background): “Before the deep learning era [17], generative modeling had been an area with a steady pace of development [1, 5, 54, 36, 49, 53].”
Divergence Triangle for Joint Training of Generator Model, Energy-Based Model, and Inferential Model (2018, methodology): “An early version of the energy-based model is the FRAME (Filters, Random field, And Maximum Entropy) model [6], [7].”
Alternating Back-Propagation for Generator Network (2016, methodology): “ explicit latent factors Xie et al. (2016) recently proposed a generative ConvNet model that did not involve explicit latent factors. Their energy based model (LeCun et al., 2006; Ngiam et al., 2011; Zhu et al., 1997; Roth & Black, 2005), which can be derived from the commonly used discriminativ”
Cooperative Learning of Energy-Based Model and Latent Variable Model via MCMC Teaching (2018, methodology): “The first model is an energy-based model (LeCun et al. 2006) or undirected graphical model or Markov random field model (Zhu, Wu, and Mumford 1997) that plays the role of the teacher.”
From Information Scaling of Natural Images to Regimes of Statistical Models (2007, methodology): “This max-min divergence principle is a generalized version of the minimax entropy principle studied by [60].”
Dynamic Textures (2003, background): “One could also infer and choose the best filters as part of the learning process [26, 43].”
Object perception as Bayesian inference. (2004, methodology): “These hypotheses could predict incoming data through feedback and be tested by computing a di(cid:11)erence signal or residual at the earlier level (Mumford 1992, Rao & Ballard 1999).”
A Generative Method for Textured Motion: Analysis and Synthesis (2002, background): “Because textures are physically the status of systems with massive elements at thermodynamic equilibrium characterized by maximum entropy distributions[19].”
Equivalence of Julesz Ensembles and FRAME Models (2000, methodology): “The solid line (path 1) represents the research theme that pursues a Gibbs model based on a minimax entropy learning scheme (Zhu et al., 1997).”
Learning Active Basis Model for Object Detection and Recognition (2010, methodology): “For instance, Zhu et al. (1997) developed a Markov random field model for textures based on such histograms.”
Learning Sparse FRAME Models for Natural Image Patterns (2014, methodology): “In the original FRAME model for stochastic textures [66], λx,s,α() is assumed to be independent of x (but dependent of s and α, which index the scale and orientation of the filter).”
Bayesian models of object perception. (2003, background): “Recent studies of natural image data have shown how statistics can provide an efficient characterization of homogeneous textures [10,11 ,12], help to discount the illuminant [13,14 ], find object boundaries [15 ], contribute to scene recognition [16], and constrain the context for object recognition”
Entropy–Copula in Hydrology and Climatology (2014, background): “Similar concepts have been applied to derive a multivariate distribution from the entropy theory based on moments of the data (Zhu et al. 1997; Phillips et al. 2006; Hao and Singh 2011).”