A unified mixture framework for motion segmentation: incorporating spatial coherence and estimating the number of models

By Yair Weiss, Edward H. Adelson. 1996

Describing a video sequence in terms of a small number of coherently moving segments is useful for tasks ranging from video compression to event perception. A promising approach is to view the motion segmentation problem in a mixture estimation framework. However, existing formulations generally use only the motion, data and thus fail to make use of static cues when segmenting the sequence. Furthermore, the number of models is either specified in advance or estimated outside the mixture model framework. In this work we address both of these issues. We show how to add spatial constraints to the mixture formulations and present a variant of the EM algorithm that males use of both the form and the motion constraints. Moreover this algorithm estimates the number of segments given knowledge about the level of model failure expected in the sequence. The algorithm's performance is illustrated on synthetic and real image sequences.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

On the Binding Problem in Artificial Neural Networks (2020, background): “Mixture models can also be biased towards spatial coherence to explicitly account for the spatial structure of visual objects (Weiss and Adelson, 1996; Blekas et al., 2005).”
Object Tracking with Bayesian Estimation of Dynamic Layer Representations (2002, methodology): “Researchers have employed Markov random fields to address the first problem [6], [8] by imposing smoothness priors on the segmentations.”
Optical Flow Estimation (2006, methodology): “Layered motion estimation can be formulated using probabilistic mixture models, with the Expectation-Maximization (EM) algorithm for parameter estimation [4, 31, 53, 54].”
Fast Feature Pyramids for Object Detection (2014, background): “Such representations have proven effective for visual processing tasks such as denoising [11], image enhancement [12], texture analysis [13], stereoscopic correspondence [14], motion flow [15], [16], attention [17], boundary detection [18] and recognition [19], [20], [21].”
Fast approximate energy minimization via graph cuts (2001, unknown): “Some exceptions include [5], [33], [45].”
Layered segmentation and optical flow estimation over time (2012, background): “Weiss and Adelson [33] incorporate static image cues into the layered segmentation and estimate the number of layers under fairly weak assumptions.”
Spectral grouping using the Nystrom method (2004, background): “Unified treatment of the spatial and temporal domains is also appealing as it could solve some of the well known problems in grouping schemes based on motion alone (e.g., layered motion models [33], [31]).”
Extraction of 2D Motion Trajectories and Its Application to Hand Gesture Recognition (2002, unknown): “Such methodsincludealgorithmswhichutilizeconstraintsbased onlocalspatialandtemporalderivations[19],[39],[17],[43], andtheblock-basedcorrelationalgorithms[2].”
A Fully-Connected Layered Model of Foreground and Background Flow (2013, methodology): “Previous methods [27, 31] allow the flow to vary smoothly or discontinuously within layers.”
Layered image motion with explicit occlusions, temporal consistency, and depth ordering (2010, methodology): “A full review of more recent methods is beyond our scope [1, 3, 2, 13, 16, 17, 20, 24, 27, 29].”
Blobworld: Image Segmentation Using Expectation-Maximization and Its Application to Image Querying (2002, methodology): “This approach is related to earlier work using EM and/or the Minimum Description Length (MDL) principle to perform segmentation based on motion [3], [44] or scaled intensities [45].”