Ad click prediction: a view from the trenches

By H. Brendan McMahan, Gary Holt, David Sculley, Michael Young, D. Ebner, Julian Grady, Lan Nie, Todd Phillips, Eugene Davydov, D. Golovin and 6 more. 2013

1,043 citations recorded by the source corpus.

Predicting ad click-through rates (CTR) is a massive-scale learning problem that is central to the multi-billion dollar online advertising industry. We present a selection of case studies and topics drawn from recent experiments in the setting of a deployed CTR prediction system. These include improvements in the context of traditional supervised learning based on an FTRL-Proximal online learning algorithm (which has excellent sparsity and convergence properties) and the use of per-coordinate learning rates. We also explore some of the challenges that arise in a real-world system that may appear at first to be outside the domain of traditional machine learning research. These include useful tricks for memory savings, methods for assessing and visualizing performance, practical methods for providing confidence estimates for predicted probabilities, calibration methods, and methods for automated management of features. Finally, we also detail several directions that did not turn out to be beneficial for us, despite promising results elsewhere in the literature. The goal of this paper is to highlight the close relationship between theoretical advances and practical engineering in this industrial setting, and to show the depth of challenges that appear when applying traditional machine learning methods in a complex dynamic system.

Subfields: Artificial Intelligence, Information Systems, Management Science and Operations Research, ML Systems & Hardware

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Management Science and Operations Research

In ML Systems & Hardware

Citations in the corpus

Deep Interest Network for Click-Through Rate Prediction (2017, methodology): “t 0.001 and decay rate is set to 0.9. ThestatisticsofalltheabovedatasetsisshowninTable2.Volume ofAlibabaDatasetismuchlargerthanbothAmazonandMovieLens, which brings more challenges. 6.2 Competitors •LR[19]. Logistic regression (LR) is a widely used shallow model before deep networks for CTR predictio”
Practical and Private (Deep) Learning without Sampling or Shuffling (2021, methodology): “…r t ( θ ) = (cid:107) θ (cid:107) 1 , is perhaps the most important practical example, playing a critical role in high-dimensional statistics (e.g., in the LASSO method) [9], as well as for applications like click-through-rate (CTR) prediction where very sparse models are needed for efﬁciency [50].”
Explaining Classification Models Built on High-Dimensional Sparse Data (2016, background): “The latter beneﬁt is cited as the prime motivating beneﬁt for their use in large-scale production systems (McMahan et al., 2013).”
Petuum: A New Platform for Distributed Machine Learning on Big Data (2013, methodology): “dimensional problems, such as gene-disease association studies, or in online advertising via ‘1-penalized regression [27].”
DeepFM: A Factorization-Machine based Neural Network for CTR Prediction (2017, methodology): “Despite their simplicity, generalized linear models, such as FTRL [McMahan et al. , 2013], have shown decent performance in practice.”
Stochastic, Distributed and Federated Optimization for Machine Learning (2017, background): “This is considered to be a huge number, yet nowhere near the scale of convex problems in click-through rate prediction systems [116].”
Clipper: A Low-Latency Online Prediction Serving System (2016, background): “Since feedback is likely to return soon after predictions are rendered [39], even infrequent or unique queries can benefit from caching.”
AdaBatch: Efficient Gradient Aggregation Rules for Sequential and Parallel Stochastic Gradient Methods (2017, methodology): “For such problems, stochastic gradient (SG) methods have been used successfully [1, 2, 3].”
xDeepFM: Combining Explicit and Implicit Feature Interactions for Recommender Systems (2018, background): “ization Models. For web-scale recommender systems (RSs), the input features are usually sparse, categoricalcontinuous-mixed, and high-dimensional. Linear models, such as logistic regression with FTRL [27], are widely adopted as they are easy to manage, maintain, and deploy. Because linear models lac”
DiFacto: Distributed Factorization Machines (2016, background): “A popular choice is `1 regularization [20], which has been widely used in linear models for high dimensional data such as computational advertising [12].”
FiBiNET: combining feature importance and bilinear feature interaction for click-through rate prediction (2019, methodology): “The shallow baseline models include LR(logistic regression)[17], FM[19, 20], FFM[9, 10], AFM[23], and the deep baseline models include FNN[25], DCN[22], DeepFM[4], XDeepFM[15].”
Evaluating Prediction-Time Batch Normalization for Robustness under Covariate Shift (2020, methodology): “When deployed models are replicated globally, as in the setting described by McMahan et al. [2013], each local model may see different distributions that reﬂect local data characteristics.”
When Do Neural Nets Outperform Boosted Trees on Tabular Data? (2023, background): “They are the oldest and among the most ubiquitous dataset types in machine learning in practice [7, 55], due to their numerous applications across medicine [36, 58], finance [5, 13], online advertising [28, 45, 50], and many other areas [9, 10, 59].”
A Modern Introduction to Online Learning (2019, methodology): “a very influential paper that they were using FTRL-Proximal to train the classifier for click prediction [McMahan et al., 2013].”
The ML test score: A rubric for ML production readiness and technical debt reduction (2017, methodology): “So, what should be tested and how much is enough? In this paper, we try to answer this question with a test rubric, which is based on engineering decades of productionlevel ML systems at Google, in systems such as ad click prediction [2] and the Sibyl ML platform [3].”
Real-Time Bidding by Reinforcement Learning in Display Advertising (2017, background): “In [18, 32], researchers have proposed to seek the optimal bidding function that directly maximizes the campaign’s key performance indicator (KPI), e.g., total clicks or revenue, based on the static distribution of input data and market competition models.”
AUC Maximization in the Era of Big Data and AI: A Survey (2022, background): “Such datasets are abundant in online spam iltering [145], ad click prediction [112], and identifying malicious URLs [108].”