Learning Visual Clothing Style with Heterogeneous Dyadic Co-Occurrences

By Andreas Veit, Balazs Kovacs, Sean Bell, Julian McAuley, Kavita Bala, Serge J. Belongie. 2015

333 citations recorded by the source corpus.

With the rapid proliferation of smart mobile devices, users now take millions of photos every day. These include large numbers of clothing and accessory images. We would like to answer questions like 'What outfit goes well with this pair of shoes?' To answer these types of questions, one has to go beyond learning visual similarity and learn a visual notion of compatibility across categories. In this paper, we propose a novel learning framework to help answer these types of questions. The main idea of this framework is to learn a feature transformation from images of items into a latent space that expresses compatibility. For the feature transformation, we use a Siamese Convolutional Neural Network (CNN) architecture, where training examples are pairs of items that are either compatible or incompatible. We model compatibility based on co-occurrence in large-scale user behavior data, in particular co-purchase data from Amazon.com. To learn cross-category fit, we introduce a strategic method to sample training data, where pairs of items are heterogeneous dyads, i.e., the two elements of a pair belong to different high-level categories. While this approach is applicable to a wide variety of settings, we focus on the representative problem of learning compatible clothing style. Our results indicate that the proposed framework is capable of learning semantic information about visual style and is able to generate outfits of clothes, with items from different categories, that go well together.

Subfields: Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Ups and Downs: Modeling the Visual Evolution of Fashion Trends with One-Class Collaborative Filtering (2016, background): “[36] fine-tunes a Siamese Convolutional Neural Network (CNN) to learn a feature transformation from the image space to a latent space of metric distances.”
Deep Interest Network for Click-Through Rate Prediction (2017, methodology): “sets and experiment codes are made available1. 6.1 Datasets and Experimental Setup Amazon Dataset2. Amazon Dataset contains product reviews and metadata from Amazon, which is used as benchmark dataset[13, 18, 23]. We conduct experiments on a subset named Electronics, which contains 192,403 users, 63”
Learning Fashion Compatibility with Bidirectional LSTMs (2017, methodology): “To demonstrate the effectiveness of our approach for modeling the compatibility of fashion outfits, we compare with the following alternative methods: SiameseNet [25].”
Answering Visual-Relational Queries in Web-Extracted Knowledge Graphs (2017, background): “For instance, in the fashion industry visual attributes are crucial for product recommendations [Liu et al., 2016, Veit et al., 2015].”
Visually-Aware Fashion Recommendation and Design with Generative Image Models (2017, methodology): “This builds upon recent work on comparative image models [7], [29], though differs in terms of the formulation and training objective used.”
Meta-Prod2Vec: Product Embeddings Using Side-Information for Recommendation (2016, methodology): “Therefore we do not compare against methods where the metadata is used directly at prediction time, such as the supervised content-based embedding models proposed in [14] where both the user and item are represented as linear combinations of the item content embeddings and [32], where the products a”
VITON: An Image-Based Virtual Try-on Network (2017, methodology): “Most existing methods focus on clothing parsing [44, 28], clothing recognition by attributes [31], matching clothing seen on the street to online products [30, 14], fashion recommendation [19], visual compatibility learning [43, 16], and fashion trend prediction [2].”
Representation Learning for Visual-Relational Knowledge Graphs (2017, background): “[17] propose a siamese CNN to learn a metric representation on pairs of textile products so as to learn which products have similar styles.”
DeepFashion: Powering Robust Clothes Recognition and Retrieval with Rich Annotations (2016, background): “Recently, extensive research efforts have been devoted to clothes classification [11, 1, 29], attribute prediction [3, 13, 4, 24], and clothing item retrieval [17, 6, 10, 27, 15], because of their potential values to the industry.”
Fashion IQ: A New Dataset Towards Retrieving Images by Natural Language Feedback (2021, background): “Many fashion datasets have been proposed over the past few years, covering different applications such as fashionability and style prediction [50, 28, 22, 51], fashion image generation [46], product search and recommendation [25, 72, 19, 41, 63], fashion apparel pixelwise segmentation [27, 74, 69], ”
Mining Latent Structures for Multimedia Recommendation (2021, background): “Multimodal recommendation systems extends the recommendation approach by considering massive multimedia content information of items, which have been successfully applied to many applications, such as e-commerce, instant video platforms and social media platforms [11, 25, 33].”
Fine-Grained Categorization and Dataset Bootstrapping Using Deep Metric Learning with Humans in the Loop (2015, background): “Recently, it has been successfully applied to variety of problems including face recognition and verification [39, 36], image retrieval [44], semantic hashing [29], product design [4], geo-localization [31] and style matching [41].”