Is a picture worth a thousand words? A Deep Multi-Modal Fusion Architecture for Product Classification in e-commerce
By Tom Zahavy, Alessandro Magnani, Abhinandan Krishnan, Shie Mannor. 2016
68 citations recorded by the source corpus.
Classifying products precisely and efficiently is a major challenge in modern e-commerce. The high traffic of new products uploaded daily and the dynamic nature of the categories raise the need for machine learning models that can reduce the cost and time of human editors. In this paper, we propose a decision level fusion approach for multi-modal product classification based on text and image neural network classifiers. We train input specific state-of-the-art deep neural networks for each input source, show the potential of forging them together into a multi-modal architecture and train a novel policy network that learns to choose between them. Finally, we demonstrate that our multi-modal network improves classification accuracy over both networks on a real-world large-scale product classification dataset that we collected from Walmart.com. While we focus on image-text fusion that characterizes e-commerce businesses, our algorithms can be easily applied to other modalities such as audio, video, physical sensors, etc.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Daniel Jaymin Mankowitz
- Douwe Kiela
- Tom Zahavy
- Shie Mannor
- Vedanuj Goswami
- Amanpreet Singh
- Mohit Bansal
- Samuel R. Bowman
In Computer Vision and Pattern Recognition
- Douwe Kiela
- Vedanuj Goswami
- Amanpreet Singh
- Davide Testuggine
- Pratik Ringshia
- Shie Mannor
- Aravind Mohan
- Hamed Firooz
In Multimodal Learning
- Davide Testuggine
- Pratik Ringshia
- Douwe Kiela
- Aravind Mohan
- Hamed Firooz
- Vedanuj Goswami
- Amanpreet Singh
- Ronghang Hu
Citations in the corpus
- The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes (2020, methodology): “Other datasets using internet data include Food101 [80], where the goal is to predict the dish of recipes and images; various versions of Yelp reviews [52]; Walmart and Ferramenta product classification [90, 21]; social media name tagging (Twitter and Snapchat) [49]; social media target-oriented sen”
- Learn What Not to Learn: Action Elimination with Deep Reinforcement Learning (2018, background): “For classification tasks with millions of labeled data, random embeddings were shown to outperform state-of-the-art techniques (Zahavy et al., 2018).”