E-commerce in Your Inbox: Product Recommendations at Scale

By Mihajlo Grbovic, Vladan Radosavljevic, Nemanja Djuric, Narayan L. Bhamidipati, Jaikit Savla, Varun Bhagwan, Doug Sharp. 2015

323 citations recorded by the source corpus.

In recent years online advertising has become increasingly ubiquitous and effective. Advertisements shown to visitors fund sites and apps that publish digital content, manage social networks, and operate e-mail services. Given such large variety of internet resources, determining an appropriate type of advertising for a given platform has become critical to financial success. Native advertisements, namely ads that are similar in look and feel to content, have had great success in news and social feeds. However, to date there has not been a winning formula for ads in e-mail clients. In this paper we describe a system that leverages user purchase history determined from e-mail receipts to deliver highly personalized product ads to Yahoo Mail users. We propose to use a novel neural language-based algorithm specifically tailored for delivering effective product recommendations, which was evaluated against baselines that included showing popular products and products predicted based on co-occurrence. We conducted rigorous offline testing using a large-scale product purchase data set, covering purchases of more than 29 million users from 172 e-commerce websites. Ads in the form of product recommendations were successfully tested on online traffic, where we observed a steady 9% lift in click-through rates over other ad formats in mail, as well as comparable lift in conversion rates. Following successful tests, the system was launched into production during the holiday season of 2014.

Subfields: Information Systems, Marketing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Information Systems

In Marketing

Citations in the corpus

Contextual Sequence Modeling for Recommendation with Recurrent Neural Networks (2017, methodology): “The first type of methods includes factor models that learn a low-rank decomposition of sparse user-item interactions matrix [3, 12, 21].”
Portrait of an Online Shopper: Understanding and Predicting Consumer Behavior (2015, background): “eet merchants’ needs by delivering targeted advertisements. In a recent study, Grbovic et al. addressed the problem of predicting the next item a user is going to purchase using a variety of features [50]. In this work, we consider the complementary problems of predicting i) the time of the next pur”
Causal embeddings for recommendation (2017, methodology): “Mail (Grbovic et al. 2015), for video recommendation at YouTube (Covington, Adams, and Sargin 2016) and for restaurant recommendations by OpenTable (Das 2015) However, even with the recent model developments, the current state-of-the-art machine learning models still frame the recommendation task…”
Context Mover's Distance & Barycenters: Optimal transport of contexts for building representations (2018, background): “Further, this co-occurrence information that is the crucial building block of our approach is inherent to a wide variety of problems, for instance, recommending products such as movies or web-advertisements (Grbovic et al., 2015), nodes in a graph (Grover & Leskovec, 2016), sequence data, or other e”
Evaluation of session-based recommendation algorithms (2018, background): “…representations , e.g., in the form of latent Markov embeddings [Chen et al. 2012, 2013; Feng et al. 2015; Wu et al. 2013] or distributional embeddings [Baeza-Yates et al. 2015; Djuric et al. 2014; Grbovic et al. 2015; Reddy et al. 2016; Tagami et al. 2015; Vasile et al. 2016; Zheleva et al. 2010].”
Neural Embeddings of Graphs in Hyperbolic Space (2017, background): “These include word analogies [18, 20], machine translation [28], document comparison [16], missing edge prediction [12], vertex attribution [24], product recommendations [2, 10], customer value prediction [6, 14] and item categorisation [3].”
Real-time Personalization using Embeddings for Search Ranking at Airbnb (2018, methodology): “• Real-time Personalization - Most of the previous work on personalization and item recommendations using embeddings [8, 11] is deployed to production by forming tables of user-item and item-item recommendations offline, and then reading from them at the time of recommendation.”
Meta-Prod2Vec: Product Embeddings Using Side-Information for Recommendation (2016, methodology): “In this paper, we present an extension of the Prod2Vec algorithm initially proposed in [9].”
Sequence-Aware Recommender Systems (2018, methodology): “In sequence-aware recommendation problems, we can recommend the next item(s) given the user’s most recent actions by traversing the embedding space in a stochastic fashion [19] or by searching the nearest neighbors to the last item(s) that were explored by the user [37].”