Personalized Fashion Recommendation with Visual Explanations based on Multimodal Attention Network: Towards Visually Explainable Recommendation

By Xu Chen, Hanxiong Chen, Hongteng Xu, Yongfeng Zhang, Yixin Cao, Zheng Qin, Hongyuan Zha. 2019

358 citations recorded by the source corpus.

Fashion recommendation has attracted increasing attention from both industry and academic communities. This paper proposes a novel neural architecture for fashion recommendation based on both image region-level features and user review information. Our basic intuition is that: for a fashion image, not all the regions are equally important for the users, i.e., people usually care about a few parts of the fashion image. To model such human sense, we learn an attention model over many pre-segmented image regions, based on which we can understand where a user is really interested in on the image, and correspondingly, represent the image in a more accurate manner. In addition, by discovering such fine-grained visual preference, we can visually explain a recommendation by highlighting some regions of its image. For better learning the attention model, we also introduce user review information as a weak supervision signal to collect more comprehensive user preference. In our final framework, the visual and textual features are seamlessly coupled by a multimodal attention network. Based on this architecture, we can not only provide accurate recommendation, but also can accompany each recommended item with novel visual explanations. We conduct extensive experiments to demonstrate the superiority of our proposed model in terms of Top-N recommendation, and also we build a collectively labeled dataset for evaluating our provided visual explanations in a quantitative manner.

Subfields: Cognitive Neuroscience, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

1. Liang Wang
2. Qiang Liu
3. Shuhui Wang
4. Xiangnan He

In Computer Vision and Pattern Recognition

1. Xiangnan He
2. Xiang Wang
3. Meng Wang
4. Yongfeng Zhang
5. Xu Chen
6. Le Wu
7. Kun Zhang
8. Zuohui Fu

Citations in the corpus

• Explainable Recommendation: A Survey and New Perspectives (2018, background): “, 2017) or images (Chen et al., 2019b), recurrent neural networks over user behavior sequence (Hidasi et al.”
• Mining Latent Structures for Multimedia Recommendation (2021, background): “VECF [4] models users’ various attentions on different image regions and reviews.”
• A Survey on Accuracy-Oriented Neural Recommendation: From Collaborative Filtering to Information-Rich Recommendation (2021, background): “For example, user and item attributes [113], [117] are multimodal item data [118], which are considered in the encoder.”
• Recommendation as Language Processing (RLP): A Unified Pretrain, Personalized Prompt & Predict Paradigm (P5) (2022, background): “The performance comparison between P5-S and P5-SN (N ∈ [1, 2, 3, 4, 5]) is illustrated in Figure 5.”
• Chat-REC: Towards Interactive and Explainable LLMs-Augmented Recommender System (2023, background): “The field of recommender systems has had a long-standing relationship with natural language processing (NLP) techniques, especially when pre-trained language models (PLMs) comes out, which improve the performance of recommender systems and explainability [3,10,11].”