Image Segmentation in Video Sequences: A Probabilistic Approach

By Nir Friedman, Stuart J. Russell. 1997

"Background subtraction" is an old technique for finding moving objects in a video sequence--for example, cars driving on a freeway. The idea is that subtracting the current image from a time-averaged background image will leave only nonstationary objects. It is, however, a crude approximation to the task of classifying each pixel of the current image; it fails with slow-moving objects and does not distinguish shadows from moving objects. The basic idea of this paper is that we can classify each pixel using a model of how that pixel looks when it is part of different classes. We learn a mixture-of-Gaussians classification model for each pixel using an unsupervised technique--an efficient, incremental version of EM. Unlike the standard image-averaging approach, this automatically updates the mixture component for each class according to likelihood of membership; hence slow-moving objects are handled perfectly. Our approach also identifies and eliminates shadows much more effectively than other techniques such as thresholding. Application of this method as part of the Roadwatch traffic surveillance project is expected to result in significant improvements in vehicle identification and tracking.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

On the Binding Problem in Artificial Neural Networks (2020, methodology): “Early approaches model images with a mixture model that treats the color values of individual pixels as independent data points that are identically distributed (Samadani, 1995; Friedman and Russell, 1997).”
Statistical modeling of complex backgrounds for foreground object detection (2004, methodology): “A mixture of Gaussian distributions can represent various background appearances, e.g., road surfaces under the sun or in the shadows [23].”
Background modeling and subtraction of dynamic scenes (2003, background): “Promising results demonstrate the potentials of the proposed framework.”
Unsupervised Video Object Segmentation for Deep Reinforcement Learning (2018, methodology): “We propose a new approach that leverages existing work in computer vision to estimate the segmentation and motion of objects in video sequences [6].”
Learning Patterns of Activity Using Real-Time Tracking (2000, methodology): “Friedman and Russell [ 5 ] have recently implemented a pixelwise EM framework for detection of vehicles that bears the most similarity to our work.”
Moving Object Detection in Spatial Domain using Background Removal Techniques - State-of-Art (2008, methodology): “retical detail of online learning algorithm can be found in [98, 108, 109] as incremental version of the EM algorithm.”
An Improved Adaptive Background Mixture Model for Real-time Tracking with Shadow Detection (2001, methodology): “Friedman and Russell modelled each pixel in a camera scene by an adaptive parametric mixture model of three Gaussian distributions [4].”
Background and foreground modeling using nonparametric kernel density estimation for visual surveillance (2002, methodology): “Instead, a generalization based on a mixture of Gaussians has been used in [9]–[11] to model such variations.”
Video Processing From Electro-Optical Sensors for Object Detection and Tracking in a Maritime Environment: A Survey (2016, methodology): “Expectation maximization approaches for training HMM have been discussed in [143], [174], [175].”
A survey on visual surveillance of object motion and behaviors (2004, methodology): “There are many algorithms for resolving these problems including temporal average of an image sequence [15], [82], adaptive Gaussian estimation [70], and parameter estimation based on pixel processes [79], [80], etc. Ridder et al. [81] model each pixel value with a Kalman Filter to compensate for…”
An adaptive color-based particle filter (2003, methodology): “A similar approach is often used for model updates in figurebackground segmentation algorithms [7,10].”
Improved adaptive Gaussian mixture model for background subtraction (2004, methodology): “Gaussian mixture model (GMM) was proposed for background subtraction in [2].”
Effective Gaussian mixture learning for video background subtraction (2005, background): “Index Terms—Adaptive Gaussian mixture, online EM, background subtraction.”
Recent Advanced Statistical Background Modeling for Foreground Detection - A Systematic Survey (2011, methodology): “In the context of a traffic surveillance system, Fr iedman and Russel [45] proposed to model each background pixel using a mixture of three Gaussians corresponding to road, vehicle and shadows.”
Background Subtraction for Freely Moving Cameras (2009, background): “This has been achieved by the near ubiquitous assumption in modern surveillance systems of stationary (or nominally stationary) cameras, [11, 34, 4, 5, 27, 2, 12, 26].”
Bayesian modeling of dynamic scenes for object detection (2005, methodology): “To address some of these issues, Friedman and Russell, and independently Stauffer and Grimson, [9], [33] proposed modeling each pixel intensity as a mixture of Gaussians, instead, to account for the multimodality of the " underlying " likelihood function of the background color.”
A texture-based method for modeling the background and detecting moving objects (2006, background): “Even in a static scene frame-to-frame changes can occur due to noise and camera jitter.”