Real-Time Bidding by Reinforcement Learning in Display Advertising

By Han Cai, Kan Ren, Weinan Zhang, Kleanthis Malialis, Jun Wang, Yong Yu, Defeng Guo. 2017

310 citations recorded by the source corpus.

The majority of online display ads are served through real-time bidding (RTB) --- each ad display impression is auctioned off in real-time when it is just being generated from a user visit. To place an ad automatically and optimally, it is critical for advertisers to devise a learning algorithm to cleverly bid an ad impression in real-time. Most previous works consider the bid decision as a static optimization problem of either treating the value of each impression independently or setting a bid price to each segment of ad volume. However, the bidding for a given ad campaign would repeatedly happen during its life span before the budget runs out. As such, each bid is strategically correlated by the constrained budget and the overall effectiveness of the campaign (e.g., the rewards from generated clicks), which is only observed after the campaign has completed. Thus, it is of great interest to devise an optimal bidding strategy sequentially so that the campaign budget can be dynamically allocated across all the available impressions on the basis of both the immediate and future rewards. In this paper, we formulate the bid decision process as a reinforcement learning problem, where the state space is represented by the auction information and the campaign's real-time parameters, while an action is the bid price to set. By modeling the state transition via auction competition, we build a Markov Decision Process framework for learning the optimal bidding policy to optimize the advertising performance in the dynamic real-time bidding environment. Furthermore, the scalability problem from the large real-world auction volume and campaign budget is well handled by state value approximation using neural networks. The empirical study on two large-scale real-world datasets and the live A/B testing on a commercial platform have demonstrated the superior performance and high efficiency compared to state-of-the-art methods.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Efficient Architecture Search by Network Transformation (2017, methodology): “Reinforcement Learning Background Our metacontroller in this work is based on RL (Sutton and Barto 1998), techniques for training the agent to maximize the cumulative reward when interacting with an environment (Cai et al. 2017).”
Near-Optimal Model-Free Reinforcement Learning in Non-Stationary Episodic MDPs (2021, background): “…models cannot capture the time-varying environments in a wide range of sequential decision-making problems, such as online advertisement auctions (Cai et al., 2017; Lu et al., 2019), dynamic pricing (Chawla et al., 2016; Mao et al., 2018), trafﬁc management (Chen et al., 2020), health-care…”
Conversational Recommender System (2018, methodology): “Deep RL has been applied for better sequential decision making in various domains, including End-to-End dialogue systems using deep RL for information access [5] [32], information extraction [16], query reformulation [17], real time ads bidding [2].”
Challenges of real-world reinforcement learning: definitions, benchmarks and analysis (2021, background): “Another approach is to design algorithms with the explicit goal of running in real-time (Cai et al. 2017; Wang and Yuan 2015).”
An empirical investigation of the challenges of real-world reinforcement learning (2020, background): “Another approach is to design algorithms with the explicit goal of running in real-time [Cai et al. 2017; Wang and Yuan 2015].”
Reinforcement Learning for Non-Stationary Markov Decision Processes: The Blessing of (More) Optimism (2020, background): “It turns out that many applications, such as real-time bidding in advertisement (ad) auctions, can be captured by this framework (Cai et al., 2017; Flajolet & Jaillet, 2017; Bal-seiro & Gur, 2019; Guo et al., 2019; Han et al., 2020).”