Video matting of complex scenes

By Yung-Yu Chuang, Aseem Agarwala, Brian Curless, David Salesin, Richard Szeliski. International Conference on Computer Graphics and Interactive Techniques, 2002

388 citations recorded by the source corpus.

This paper describes a new framework for video matting, the process of pulling a high-quality alpha matte and foreground from a video sequence. The framework builds upon techniques in natural image matting, optical flow computation, and background estimation. User interaction is comprised of garbage matte specification if background estimation is needed, and hand-drawn keyframe segmentations into "foreground," "background" and "unknown". The segmentations, called trimaps, are interpolated across the video volume using forward and backward optical flow. Competing flow estimates are combined based on information about where flow is likely to be accurate. A Bayesian matting technique uses the flowed trimaps to yield high-quality mattes of moving foreground elements with complex boundaries filmed by a moving camera. A novel technique for smoke matte extraction is also demonstrated.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/8fd18a0f65134b1abfbb1adf8653ebba63bb2c0e

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Video object cut and paste (2005, background): “, [Chuang et al. 2002; Apostoloff and Fitzgibbon 2004]) have relaxed the solid color background requirement to allow smooth color changes.”
Geodesic Matting: A Framework for Fast Interactive Image and Video Segmentation and Matting (2009, methodology): “This technique is applied to videos in [10], where the trimap is temporally propagated using optical flow and the matte is pulled out individually in each frame by the Bayesian matting algorithm.”
Interactive video cutout (2005, methodology): “[Chuang et al. 2001; Chuang et al. 2002] extend this approach by using Bayesian statistics to accurately estimate alpha for partially transparent foreground objects in both images and video.”
A Closed-Form Solution to Natural Image Matting (2006, methodology): “Most existing methods for natural image matting require the input image to be accompanied by a trimap [1], [2], [5], [6], [14], [17], labeling each pixel as foreground, background , or unknown.”
A Geodesic Framework for Fast Interactive Image and Video Segmentation and Matting (2007, methodology): “This technique is applied to videos in [6], where the trimap is temporally propagated using optical flow and the matte is pulled out individually in each frame by the Bayesian matting algorithm.”
Beyond pixels: exploring new representations and applications for motion analysis (2009, neutral): “Interactive object segmentation in video sequences has recently been explored in computer graphics, including contour-based video rotoscoping [3], soft alpha matting [27], oversegmentation [122] and 3D graph cut [59].”
High-quality video view interpolation using a layered representation (2004, methodology): “(Chuang et al. [2002] later extended their technique to videos using optic ﬂ ow.)”
Bilayer Segmentation of Live Video (2006, neutral): “frames Figure 1: An example”
Fast, automatic and fine-grained tampered JPEG image detection via DCT coefficient analysis (2009, methodology): “2(c)) refers to a block in the tampered region or along the synthesis edge and an unchanged block is a block in the unchanged region.”
iLamps: geometrically aware and self-configuring projectors (2003, methodology): “This method has been extended to video [Chuang et al. 2002]; however, trimaps still need to be specified manually for keyframes.”
Content-preserving warps for 3D video stabilization (2009, methodology): “An accurate solution would require solving several challenging computer vision problems, such as video layer separation [Chuang et al. 2002], non-rigid 3D tracking [Torresani et al.”
Video SnapCut: robust video object cutout using localized classifiers (2009, methodology): “…[Armstrong et al. 2007; Li et al. 2005; Wang et al. 2005], using optical flow to propagate the segmentation across time [Blake and Isard 1998; Chuang et al. 2002], tracking and moving spatially-weighted color models [Yu et al. 2007], or interpolating partial boundary splines using…”