Material recognition in the wild with the Materials in Context Database

By Sean Bell, Paul Upchurch, Noah Snavely, Kavita Bala. 2014

578 citations recorded by the source corpus.

Recognizing materials in real-world images is a challenging task. Real-world materials have rich surface texture, geometry, lighting conditions, and clutter, which combine to make the problem particularly difficult. In this paper, we introduce a new, large-scale, open dataset of materials in the wild, the Materials in Context Database (MINC), and combine this dataset with deep learning to achieve material recognition and segmentation of images in the wild. MINC is an order of magnitude larger than previous material databases, while being more diverse and well-sampled across its 23 categories. Using MINC, we train convolutional neural networks (CNNs) for two tasks: classifying materials from patches, and simultaneous material recognition and segmentation in full images. For patch-based classification on MINC we found that the best performing CNN architectures can achieve 85.2% mean class accuracy. We convert these trained CNN classifiers into an efficient fully convolutional framework combined with a fully connected conditional random field (CRF) to predict the material at every pixel in an image, achieving 73.1% mean class accuracy. Our experiments demonstrate that having a large, well-sampled dataset such as MINC is crucial for real-world material recognition and segmentation.

Subfields: Computer Vision and Pattern Recognition, Media Technology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Media Technology

Citations in the corpus

Semantic Understanding of Scenes Through the ADE20K Dataset (2016, methodology): “forts include Pascal-Context [22], NYU Depth V2 [23], SUN database [34], SUN RGB-D dataset [31], CityScapes Semantic Understanding of Scenes through the ADE20K Dataset 3 dataset [7], and OpenSurfaces [2, 3]. Recently COCO stuff dataset [4] provides additional stuff segmentation complementary to the ”
COCO-Stuff: Thing and Stuff Classes in Context (2016, background): “Although the recent Materials in Context dataset [5] features realistic and difficult images, they are mostly restricted to indoor scenes with man-made materials.”
DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs (2016, methodology): “In independent work, [57] use a very similar densely connected CRF model to refine the results of DCNN for the problem of material classification.”
Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs (2014, methodology): “Traditionally, conditional random ﬁelds (CRFs) have been employed to smooth noisy segmentation maps (Rother et al., 2004; Kohli et al., 2009).”
Scene Parsing through ADE20K Dataset (2017, methodology): “Recently, two datasets were released that go beyond the typical labeling setup by also providing pixel-level annotation for the object parts, i.e. Pascal-Part dataset [5], or material classes, i.e. OpenSurfaces [2, 3].”
Conditional Random Fields as Recurrent Neural Networks (2015, methodology): “ecognitionwhereaslowlayers keep information about the structure of the image, such as edges. In their work, connections from early layers to later layers were used to combine these cues. Bell et al . [5] and Chen et al . [10, 41] used a CRF to rene segmentation resultsobtainedfromaCNN.Bell etal .foc”
Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation (2018, background): “, image pyramid) [18, 16, 58, 44, 11, 9] or those that adopt probabilistic graphical models (such as DenseCRF [37] with efficient inference algorithm [2]) [8, 4, 82, 44, 48, 55, 63, 34, 72, 6, 7, 9].”
UberNet: Training a Universal Convolutional Neural Network for Low-, Mid-, and High-Level Vision Using Diverse Datasets and Limited Memory (2016, background): “…103], depth estimation [25, 63, 64], intrinsic image decomposition [73] shadow detection [108], texture classiﬁcation [19], material classiﬁcation [4], saliency estimation [58, 112], semantic segmentation [14, 27,67], region proposal generation [30,82,85], instance segmentation [21, 37, 82], pose…”
Visually Indicated Sounds (2015, background): “A central goal of other methods has been to use a proxy signal ( e.g ., temporal proximity) to learn a generically useful representation of the world.”
Learning Visual Clothing Style with Heterogeneous Dyadic Co-Occurrences (2015, background): “This means that with high probability a shirt labeled as shoe will be closer to the queried shirt than real shoes.”
Searching for Efficient Multi-Scale Architectures for Dense Image Prediction (2018, background): “…models that contain encoder-decoder structures [2, 72, 49, 28, 64, 93, 96], or (3) designing models that employ a multi-scale context module, e.g ., DenseCRF module [42, 4, 10, 98, 50, 76], global context [56, 95], or atrous convolutions deployed in cascade [57, 94, 12] or in parallel [11, 12].”
Weakly-and Semi-Supervised Learning of a Deep Convolutional Network for Semantic Image Segmentation (2015, methodology): “Various approaches have been tried over the years, but according to the results on the challenging Pascal VOC 2012 segmentation benchmark, the best performing methods all use some kind of Deep Convolutional Neural Network (DCNN) [2, 5, 8, 14, 25, 27, 41].”
A Wholistic View of Continual Learning with Deep Neural Networks: Forgotten Lessons and the Bridge to Active and Open World Learning (2020, methodology): “Because there is an abundance of material types, it has become the de-facto standard to collapse any available imagery that is connected to less important materials or where meager amounts of data are available into a single “other” material (Cimpoi et al., 2015; Bell et al., 2015).”
ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks (2018, methodology): “erceptual loss that adopts a VGG network trained for image classication, we develop a more suitable perceptual loss for SR { MINC loss. It is based on a ne-tuned VGG network for material recognition [38], which focuses on textures rather than object. Although the gain of perceptual index brought by ”
What's the Point: Semantic Segmentation with Point Supervision (2015, neutral): “tion cost through effective interfaces [30, 21, 22, 41, 3], e.”
Physics 101: Learning Physical Object Properties from Unlabeled Videos (2016, methodology): “Our incorporation of physical laws into our model distinguish our framework from previous methods introduced in the computer vision and robotics community for predicting physical interactions or properties of objects for various purposes [4, 5, 9, 12, 16, 21, 22, 26].”
Real-time user-guided image colorization with learned deep priors (2017, background): “Deep neural networks (Krizhevsky et al. 2012) excel at extracting rich semantics from images, from middle-level concepts like material (Bell et al. 2015; Wang et al. 2016) and segmentation (Xie and Tu 2015), to high-level knowledge such as objects (Girshick et al. 2014) and scene categories (Zhou…”
Learning Non-Lambertian Object Intrinsics Across ShapeNet Categories (2016, background): “While benchmark scores for many CNN intrinsics models [21, 34, 35, 20, 22] are improving, the visual quality of results remains poor, compared with those from traditional approaches based on hand-crafted features and multitudes of priors [6].”
MSeg: A Composite Dataset for Multi-Domain Semantic Segmentation (2020, background): “MINC OPENSURFACES [65] Materials Annotation categories are material-centric, instead of object- or stuffcentric.”