Reader small image

You're reading from  Practical Machine Learning Cookbook

Product typeBook
Published inApr 2017
Reading LevelIntermediate
PublisherPackt
ISBN-139781785280511
Edition1st Edition
Languages
Right arrow
Author (1)
Atul Tripathi
Atul Tripathi
author image
Atul Tripathi

Atul Tripathi has spent more than 11 years in the fields of machine learning and quantitative finance. He has a total of 14 years of experience in software development and research. He has worked on advanced machine learning techniques, such as neural networks and Markov models. While working on these techniques, he has solved problems related to image processing, telecommunications, human speech recognition, and natural language processing. He has also developed tools for text mining using neural networks. In the field of quantitative finance, he has developed models for Value at Risk, Extreme Value Theorem, Option Pricing, and Energy Derivatives using Monte Carlo simulation techniques.
Read more about Atul Tripathi

Right arrow

Chapter 8. Reinforcement Learning

In this chapter, we will cover the following recipes:

  • Markov chains - the stocks regime switching model
  • Markov chains - the multi-channel attribution model
  • Markov chains - the car rental agency service
  • Continuous Markov chains - vehicle service at the gas station
  • Monte Carlo simulations - calibrated hull and white short-rates

Introduction


The Markov chain: A sequence  of trials of an experiment is a Markov chain if the outcome of each experiment is one of the set of discrete states, and the outcome of the experiment is dependent only on the present state and not of any of the past states. The probability of changing from one state to another state is represented as. It is called a transition probability. The transition probability matrix is an n × n matrix such that each element of the matrix is non-negative and each row of the matrix sums to one.

Continuous time Markov chains: Continuous-time Markov chains can be labeled as transition systems augmented with rates that have discrete states. The states have continuous time-steps and the delays are exponentially distributed. Continuous-time Markov chains are suited to model reliability models, control systems, biological pathways, chemical reactions, and so on.

Monte Carlo simulations: Monte Carlo simulation  is a stochastic simulation of system behavior. The...

Markov chains - the stocks regime switching model


In the last few decades, a lot of studies have been conducted on the analysis and forecasting of volatility. Volatility is the degree of variation of a trading price series over time as measured by the standard deviation of returns. Models of stock returns assume that the returns follow a geometric Brownian motion. This implies that over any discrete time interval, the return on stocks is log normally distributed and that returns in non-overlapping intervals are independent. Studies have found that this model fails to capture extreme price movements and stochastic variability in the volatility parameter. Stochastic volatility takes discrete values, switching between these values randomly. This is the basis of the regime-switching lognormal process (RSLN).

Getting ready

In order to perform the Markov chains regime switching model we shall be using data collected from the Stock's dataset.

Step 1 - collecting and describing the data

The dataset...

Markov chains - the multi-channel attribution model


A customer's journey undergoes a path of different channels before a purchase on an e-commerce website. Multi-channel attribution assigns a value to each step in the journey. The question is how one can identify the value in the actions people take on your website that lead to a conversion. Commonly, businesses use "last click" attribution, which means assigning all the conversion value to the last step in the journey or "first click" attribution. The first step in developing multi-channel attribution analytics is to understand the customer's journey - from awareness through purchase to after-purchase support. The ultimate goal is to develop loyal customers who spend a significant money on purchases, recommend the brand to others, and it potentially becomes a brand.

Getting ready

In order to perform a Markov chains multi-channel attribution model we shall be simulating customer journeys containing three unique channels.

How to do it...

Let...

Markov chains - the car rental agency service


Suppose a car rental agency has three locations in Ottawa: A downtown location (labeled A), A East End location (labeled B), and a West End location (labeled C). The agency has a group of delivery drivers to serve all three locations. The agency's statistician has determined the following:

  • Of the calls to the Downtown location, 30% are delivered in the Downtown area, 30% are delivered in the East end, and 40% are delivered in the West end
  • Of the calls to the East end location, 40% are delivered in the downtown area, 40% are delivered in the East end, and 20% are delivered in the West end
  • Of the calls to the West end location, 50% are delivered in the Downtown area, 30% are delivered in the East end, and 20% are delivered in the West end

After making a delivery, a driver goes to the nearest location to make the next delivery. This way, the location of a specific driver is determined only by their previous location.

How to do it...

Let's get into the...

Continuous Markov chains - vehicle service at a gas station


A gas station has a single pump. There is no space for vehicles to wait. If a vehicle arrives at the pump and there is no place the vehicle leaves without filling at the pump. Vehicles arrive at the gas station following a Poisson process with a rate of 3/20 vehicles per minute. Of the vehicles arriving at the pump, 75% are cars and 25% are motorcycles. The refueling time can be modeled with an exponential random variable with a mean of eight minutes for cars and three minutes for motorcycles.

Getting ready

In order to perform continuous Markov chains for vehicle service at a gas station we shall be simulating data.

How to do it...

Let's get into the details.

Step 1 - preparing the dataset

Load the following packages:

    > install.packages("simmer")
    > install.packages("ggplot2")
    > library(simmer)
    > library(ggplot2)

Note

Version info: Code for this page was tested in R version 3.2.2 (2015-08-14)

Initializing the...

Monte Carlo simulations - calibrated Hull and White short-rates


Monte Carlo simulation is a stochastic simulation of system behavior. The simulation uses sampling experiments to be performed on the model and it then conducts numerical experiments using the computer to obtain a statistical understanding of the system behavior.

Getting ready

In order to perform Monte Carlo simulations for calibrated hull and white short-rates, data is taken from sample code that has been shipped with QuantLib 0.3.10, market data used to construct the term structure of interest rates and swaption volatility matrix with corresponding maturities and tenors.

Step 1 - installing the packages and libraries

Load the following packages:

    >install.packages("RQuantLib", type="binary")
    >install.packages("ESGtoolkit")
    >library(RQuantLib)
    >library(ESGtoolkit)

Note

Version info: Code for this page was tested in R version 3.2.2 (2015-08-14)

In order to make part of the Quantlib package accessible to...

lock icon
The rest of the chapter is locked
You have been reading a chapter from
Practical Machine Learning Cookbook
Published in: Apr 2017Publisher: PacktISBN-13: 9781785280511
Register for a free Packt account to unlock a world of extra content!
A free Packt account unlocks extra newsletters, articles, discounted offers, and much more. Start advancing your knowledge today.
undefined
Unlock this book and the full library FREE for 7 days
Get unlimited access to 7000+ expert-authored eBooks and videos courses covering every tech area you can think of
Renews at $15.99/month. Cancel anytime

Author (1)

author image
Atul Tripathi

Atul Tripathi has spent more than 11 years in the fields of machine learning and quantitative finance. He has a total of 14 years of experience in software development and research. He has worked on advanced machine learning techniques, such as neural networks and Markov models. While working on these techniques, he has solved problems related to image processing, telecommunications, human speech recognition, and natural language processing. He has also developed tools for text mining using neural networks. In the field of quantitative finance, he has developed models for Value at Risk, Extreme Value Theorem, Option Pricing, and Energy Derivatives using Monte Carlo simulation techniques.
Read more about Atul Tripathi