Filters, Random Fields and Maximum Entropy (FRAME): Towards a Unified Theory for Texture Modeling

By Song-Chun Zhu, Ying Nian Wu, David Mumford. 1998

This article presents a statistical theory for texture modeling. This theory combines filtering theory and Markov random field modeling through the maximum entropy principle, and interprets and clarifies many previous concepts and methods for texture analysis and synthesis from a unified point of view. Our theory characterizes the ensemble of images I with the same texture appearance by a probability distribution f(I) on a random field, and the objective of texture modeling is to make inference about f(I), given a set of observed texture examples.In our theory, texture modeling consists of two steps. (1) A set of filters is selected from a general filter bank to capture features of the texture, these filters are applied to observed texture images, and the histograms of the filtered images are extracted. These histograms are estimates of the marginal distributions of f( I). This step is called feature extraction. (2) The maximum entropy principle is employed to derive a distribution p(I), which is restricted to have the same marginal distributions as those in (1). This p(I) is considered as an estimate of f( I). This step is called feature fusion. A stepwise algorithm is proposed to choose filters from a general filter bank. The resulting model, called FRAME (Filters, Random fields And Maximum Entropy), is a Markov random field (MRF) model, but with a much enriched vocabulary and hence much stronger descriptive ability than the previous MRF models used for texture modeling. Gibbs sampler is adopted to synthesize texture images by drawing typical samples from p(I), thus the model is verified by seeing whether the synthesized texture images have similar visual appearances to the texture images being modeled. Experiments on a variety of 1D and 2D textures are described to illustrate our theory and to show the performance of our algorithms. These experiments demonstrate that many textures which are previously considered as from different categories can be modeled and synthesized in a common framework.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Energy-Inspired Models: Learning with Sampler-Induced Distributions (2019, background): “Energy-based models (EBMs) have a long history in statistics and machine learning [16, 75, 44].”
Learning Non-Convergent Non-Persistent Short-Run MCMC Toward Energy-Based Model (2019, methodology): “An early version of EBM is the FRAME (Filters, Random field, And Maximum Entropy) model [52, 46, 51], which is an exponential family model, where the features are the responses from a bank of filters.”
On the Anatomy of MCMC-based Maximum Likelihood Learning of Energy-Based Models (2019, methodology): “The persistent chains can be initialized from noise as in (Zhu, Wu, and Mumford 1998; Lu, Zhu, and Wu 2016; Xie et al. 2016) or from data samples as in Persistent Con-trastive Divergence (PCD) (Tieleman 2008).”
Learning Descriptor Networks for 3D Shape Synthesis and Analysis (2018, methodology): “The FRAME (Filters, Random field, And Maximum Entropy) [35] model, which was developed for modeling stochastic textures.”
Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks (2015, methodology): “ture synthesis [3, 33, 22], with Portilla & Simoncelli [22] making use of a steerable pyramid wavelet representation [27], similar to our use of a Laplacian pyramid.”
Generative Image Modeling Using Style and Structure Adversarial Networks (2016, background): “Finally, the idea of factorizing image into two separate phenomena has been well studied in [46,47,48,49], which motivates us to decompose the generative process to structure and style.”
Visualizing Deep Convolutional Neural Networks Using Natural Pre-images (2015, methodology): “Julesz’ ideas were framed probabilistically by Zhu and Mumford [55] and their generation-by-sampling framework was later approximated by Portilla and Simoncelli [35] as a pre-image problem which can be seen as a special case of the inversion method discussed here.”
Texture Networks: Feed-forward Synthesis of Textures and Stylized Images (2016, methodology): “Where (Zhu et al., 1998; Portilla & Simoncelli, 2000) use linear filters, wavelets, and histograms to build their texture statistics, (Mahendran & Vedaldi, 2015; Gatys et al., 2015a;a) extract statistics from pre-trained deep neural networks.”
Improved Texture Networks: Maximizing Quality and Diversity in Feed-Forward Stylization and Texture Synthesis (2017, methodology): “Originally, researchers considered the obvious candidates: Gaussian derivative filters, Gabor filters, wavelets, histograms, and similar [20, 16, 21].”
Implicit Generation and Generalization in Energy-Based Models (2019, methodology): “, 1985] in machine learning, and the FRAME model [Zhu et al., 1998] in computer vision.”
A parametric texture model based on deep convolutional features closely matches texture appearance for humans (2017, methodology): “Parameteric models that aim to match the appearance of natural textures are typically assessed by examining artiﬁcial textures synthesised by the model (Heeger & Bergen, 1995; Portilla & Simoncelli, 2000; Safranek & Johnston, 1989; Safranek, Johnston, & Rosenholtz, 1990; Zhu, Wu, & Mumford, 1998).”
Face Hallucination: Theory and Practice (2007, background): “modeling the marginal of the band-pass filtering responses [45].”
Dark, Beyond Deep: A Paradigm Shift to Cognitive AI with Humanlike Common Sense (2020, background): “[187] Song-Chun Zhu, Yingnian Wu, David Mumford, Filters, random fields and maximum entropy (frame): Towards a unified theory for texture modeling, International Journal of Computer Vision (IJCV) 27 (2) (1998) 107–126.”
A Review of Statistical Approaches to Level Set Segmentation: Integrating Color, Texture, Motion and Shape (2007, unknown): “While texture analysis can rely on texture samples to learn accurate models (Hassner and Sklansky, 1980; Cross and Jain, 1993; Mallat, 1989; Simoncelli et al., 1992; Zhu et al., 1998), unsupervised image segmentation should learn these parameters on-line.”
Support Vector Machines for Texture Classification (2002, methodology): “Model-based methods characterize texture images based on probability distributions in random fields, such as Markov chains and Markov random fields (MRFs) [7], [9], [20], [30].”
Detecting Hidden Messages Using Higher-Order Statistics and Support Vector Machines (2002, background): “In contrast, the approach taken here relies on building higher-order statistical models for natural images [13, 19, 29, 14, 21] and looking for deviations from these models.”
Deep Probabilistic Modeling of Natural Images using a Pyramid Decomposition (2016, methodology): “However, because of the difficulty of the problem, until recently all existing models were restricted to small image patches, typically 9x9 pixels or less, and the modeling of only low-order statistics, such edge frequency and orientation (Zhu et al., 1998; Roth & Black, 2005; Zoran & Weiss, 2011).”
PixelCNN Models with Auxiliary Variables for Natural Image Modeling (2016, background): “3x3 and 9x9 pixels, and reflected only low-order statistics, such as edge frequency and orientation (Zhu et al., 1998; Roth & Black, 2005; Carlsson et al., 2008; Zoran & Weiss, 2011).”