Chapter 12. Deep Reinforcement Learning in Ad Tech

So far in this unit of discussing reinforcement learning application research domains, we saw how reinforcement learning is disrupting the field of robotics, autonomous driving, financial portfolio management, and solving games of extremely high complexity, such as Go. Another important domain which is likely to be disrupted by reinforcement learning is advertisement technology.

Before getting into the details of the problem statement and it's solution based on reinforcement learning, let's understand the challenges, business models, and bidding strategies involved, which will work as a basic prerequisite in understanding the problem that we will try to solve using a reinforcement learning framework. The topics that we will be covering in this chapter are as follows:

• Computational advertising challenges and bidding strategies

• Real-time bidding by reinforcement learning in display advertising

Computational advertising challenges and bidding strategies

Advertising is a mode of conveying information. The core task of computational advertising is to find the best match between a given user in a given context and an advertisement, where the following factors apply:

• Context/Auctioneer: A platform visited by a user and that is deemed fit for advertisements, for example:
  • A user using a search engine. Therefore, sponsored advertisements in such a scenario form a good plan.
  • A user reading a web page. Therefore, display advertisements fit such cases.
  • A user watching any video (movie, clips, short videos, and so on). Therefore, short video advertisements are good.
• Constraints: The biggest constraint of all for the advertiser is limited budget and limited time period.

The core challenges to meet regarding the preceding goals are as follows:

• Designing markets and exchanges that can facilitate the task and maximize value for all the participating stake holders, which are users, advertisers, and...

Real-time bidding by reinforcement learning in display advertising

Online displays are majorly served through real-time bidding where each impression of the display advertisement is auctioned in real time simultaneously when generated from a user visit. Placing a bid automatically, and in real time, is highly critical for advertisers to maximize their profits. Thus, a learning algorithm needs to be devised that can devise an optimal learning strategy in real time based on historical data, so that dynamic allocation of the budget takes place across different impressions according to immediate and future returns. Here, we will discuss formulating a bid-decision process in terms of a reinforcement learning framework published in Real-Time Bidding by Reinforcement Learning in Display Advertising by Cai et. al. 2017.

In this research by Cai et. al., the machine bidding in the context of display advertising is considered, where real-time bidding is a highly challenging task because, in the case...

Summary

In this chapter, we understood the basic concepts and challenges in the domain of advertising technology. We also learned about the relevant business models, such as CPC, CPM, and CPA, and real-time strategy bidding and why there's a need for an autonomous agent to automate the process. Moreover, we discussed a basic approach to converting the problem state of real-time bidding in online advertising into a reinforcement-learning framework. This is a totally new domain for reinforcement learning to disrupt. Many more exploratory works utilizing reinforcement learning for advertising technology, and their results, are yet to be published.

In the next chapter, we will study how reinforcement learning is being used in the field of computer vision, especially for object detection.