Chapter 2. Statistical Modeling

In this chapter, we are going to discuss statistical modeling, which will be the first step in learning quantitative finance in R as the concepts of statistical modeling are the driving force for quantitative finance. Before starting this chapter, the assumption is that learners are familiar with basic programming in R and have a sound knowledge of statistical concepts. We will not be discussing statistical concepts in this chapter. We will be discussing how to do the statistical modeling in R.

This chapter covers the following topics:

• Probability distributions

• Sampling

• Statistics

• Correlation

• Hypothesis testing

• Parameter estimation

• Outlier detection

• Standardization

• Normalization

Probability distributions

Probability distributions determine how the values of random variables are spread. For example, the set of all the possible outcomes of the tossing of a sequence of coins gives rise to binomial distribution. The means of large samples of the data population follow normal distribution, which is the most common and useful distribution.

The features of these distributions are very well known and can be used to extract inferences about the population. We are going to discuss in this chapter some of the most common probability distributions and how to compute them.

Normal distribution

Normal distribution is the most widely used probability distribution in the financial industry. It is a bell-shaped curve and mean, median mode is the same for normal distribution. It is denoted by  where  is the mean and  is the variance of the sample. If the mean is 0 and variance is 1 then the normal distribution is known as standard normal distribution N(1, 0).

Now let us discuss the main...

Sampling

When building any model in finance, we may have very large datasets on which model building will be very time-consuming. Once the model is built, if we need to tweak the model again, it is going to be a time-consuming process because of the volume of data. So it is better to get the random or proportionate sample of the population data on which model building will be easier and less time-consuming. So in this section, we are going to discuss how to select a random sample and a stratified sample from the data. This will play a critical role in building the model on sample data drawn from the population data.

Random sampling

Select the sample where all the observation in the population has an equal chance. It can be done in two ways, one without replacement and the other with replacement.

A random sample without replacement can be done by executing the following code:

> RandomSample <- Sampledata[sample(1:nrow(Sampledata), 10,  
>+ replace=FALSE),] 

This generates the...

Statistics

In a given dataset, we try to summarize the data by the central position of the data, which is known as measure of central tendency or summary statistics. There are several ways to measure the central tendency, such as mean, median, and mode. Mean is the widely used measure of central tendency. Under different scenarios, we use different measures of central tendency. Now we are going to give an example of how to compute the different measures of central tendency in R.

Mean

mean is the equal weightage average of the sample. For example, we can compute the mean of Volume in the dataset Sampledata by executing the following code, which gives the arithmetic mean of the volume:

mean(Sampledata$Volume) 

Median

Median is the mid value of the matrix when it is arranged in a sorted way, which can be computed by executing the following code:

median(Sampledata$Volume) 

Mode

Mode is the value present in the attribute which has maximum frequency. For mode, there does not exist an inbuilt...

Correlation

Correlation plays a very important role in quant finance. It not only determines the relation between the financial attributes but also plays a crucial role in predicting the future of financial instruments. Correlation is the measure of linear relationship between the two financial attributes. Now let us try to compute the different types of correlation in R using Sampledata, which is used in identifying the orders of components of predictive financial models.

Correlation can be computed by the following code. Let's first subset the data and then run the function for getting correlation:

x<-Sampledata[,2:5] 
rcorr(x, type="pearson") 

This generates the following correlation matrix, which shows the measure of linear relationship between the various daily level prices of a stock:

Hypothesis testing

Hypothesis testing is used to reject or retain a hypothesis based upon the measurement of an observed sample. We will not be going into theoretical aspects but will be discussing how to implement the various scenarios of hypothesis testing in R.

Lower tail test of population mean with known variance

The null hypothesis is given by  where is the hypothesized lower bound of the population mean.

Let us assume a scenario where an investor assumes that the mean of daily returns of a stock since inception is greater than $10. The average of 30 days' daily return sample is $9.9. Assume the population standard deviation is 0.011. Can we reject the null hypothesis at .05 significance level?

Now let us calculate the test statistics z which can be computed by the following code in R:

> xbar= 9.9           
> mu0 = 10            
> sig = 1.1            
> n = 30                  
> z = (xbar-mu0)/(sig/sqrt(n))  
> z  

Here:

• xbar: Sample...

Parameter estimates

In this section, we are going to discuss some of the algorithms used for parameter estimation.

Maximum likelihood estimation

Maximum likelihood estimation (MLE) is a method for estimating model parameters on a given dataset.

Now let us try to find the parameter estimates of a probability density function of normal distribution.

Let us first generate a series of random variables, which can be done by executing the following code:

> set.seed(100) 
> NO_values <- 100 
> Y <- rnorm(NO_values, mean = 5, sd = 1) 
> mean(Y) 

This gives 5.002913.

> sd(Y) 

This gives 1.02071.

Now let us make a function for log likelihood:

LogL <- function(mu, sigma) { 
+      A = dnorm(Y, mu, sigma) 
+      -sum(log(A)) 
+  } 

Now let us apply the function mle to estimate the parameters for estimating mean and standard deviation:

  > library(stats4) 
> mle(LogL, start = list(mu = 2, sigma=2)) 

mu and sigma have been...

Outlier detection

Outliers are very important to be taken into consideration for any analysis as they can make analysis biased. There are various ways to detect outliers in R and the most common one will be discussed in this section.

Boxplot

Let us construct a boxplot for the variable volume of the Sampledata, which can be done by executing the following code:

> boxplot(Sampledata$Volume, main="Volume", boxwex=0.1) 

The graph is as follows:

Figure 2.16: Boxplot for outlier detection

An outlier is an observation which is distant from the rest of the data. When reviewing the preceding boxplot, we can clearly see the outliers which are located outside the fences (whiskers) of the boxplot.

LOF algorithm

The local outlier factor (LOF) is used for identifying density-based local outliers. In LOF, the local density of a point is compared with that of its neighbors. If the point is in a sparser region than its neighbors then it is treated as an outlier. Let us consider some of the variables from...

Standardization

In statistics, standardization plays a crucial role as we have various attributes for modeling and all of them have different scales. So for comparison purposes, we need to standardize the variables to bring them on the same scale. Centering the values and creating the z scores is done in R by the scale() function. It takes the following arguments:

• x: A numeric object

• center: If TRUE, the object's column means are subtracted from the values in those columns (ignoring NAs); if FALSE, centering is not performed

• scale: If TRUE, the centered column values are divided by the column's standard deviation (when center is also TRUE; otherwise, the root mean square is used); if FALSE, scaling is not performed

If we want to center the data of Volume in our dataset, we just need to execute the following code:

>scale(Sampledata$Volume, center=TRUE, scale=FALSE) 

If we want to standardize the data of volume in our dataset, we just need to execute the following code:

>scale(Sampledata...

Normalization

Normalization is done using the minmax concept to bring the various attributes on the same scale. It is calculated by the formula given here:

normalized = (x-min(x))/(max(x)-min(x))

So if we want to normalize the volume variable, we can do it by executing the following code:

> normalized = (Sampledata$Volume-+min(Sampledata$Volume))/(max(Sampledata$Volume)-+min(Sampledata$Volume)) 
> normalized 

Questions

1. Construct examples of normal, Poisson, and uniform distribution in R.

2. How do you do random and stratified sampling in R?

3. What are the different measures of central tendency and how do you find them in R?

4. How do you compute kurtosis and skewness in R?

5. How do you do hypothesis testing in R with known/unknown variance of population in R?

6. How do you detect outliers in R?

7. How do you do parameter estimates for a linear model and MLE in R?

8. What is standardization and normalization in R and how do you perform it in R?

Summary

In this chapter, we have discussed the most commonly used distributions in the finance domain and associated metrics computations in R; sampling (random and stratified); measures of central tendencies; correlations and types of correlation used for model selections in time series; hypothesis testing (one-tailed/two-tailed) with known and unknown variance; detection of outliers; parameter estimation; and standardization/normalization of attributes in R to bring attributes on comparable scales.

In the next chapter, analysis done in R associated with simple linear regression, multivariate linear regression, ANOVA, feature selection, ranking of variables, wavelet analysis, fast Fourier transformation, and Hilbert transformation will be covered.

The rest of the chapter is locked

Unlock this book and the full library FREE for 7 days

Get unlimited access to 7000+ expert-authored eBooks and videos courses covering every tech area you can think of

Renews at $15.99/month. Cancel anytime

Start free trial

Previous Chapter

Chapter 9 of 16

Next Chapter

You have been reading a chapter from

Learning Quantitative Finance with R

Published in: Mar 2017Publisher: PacktISBN-13: 9781786462411

© 2017 Packt Publishing Limited All Rights Reserved

Register for a free Packt account to unlock a world of extra content!

A free Packt account unlocks extra newsletters, articles, discounted offers, and much more. Start advancing your knowledge today.

Sign up now

undefined

Unlock this book and the full library FREE for 7 days

Get unlimited access to 7000+ expert-authored eBooks and videos courses covering every tech area you can think of

Start free trial

Renews at $15.99/month. Cancel anytime

Authors (2)

Dr. Param Jeet

Dr. Param Jeet is a Ph.D. in mathematics from one of India's leading technological institute in Madras (IITM), India. Dr. Param Jeet has a couple of mathematical research papers published in various international journals. Dr. Param Jeet has been into the analytics industry for the last few years and has worked with various leading multinational companies as well as consulted few of companies as a data scientist.
Read more about Dr. Param Jeet

See other products by Dr. Param Jeet

PRASHANT VATS

Prashant Vats is a masters in mathematics from one of India's leading technological institute, IIT Mumbai. Prashant has been into analytics industry for more than 10 years and has worked with various leading multinational companies as well as consulted few of companies as data scientist across several domain.
Read more about PRASHANT VATS

See other products by PRASHANT VATS

Other recommended products

Related to this chapter

Python for Finance

Python is a free and powerful tool used for quantitative finance and is a popular choice amongst many financial analysts. This book will teach you the basics of quantitative finance, and how they can be implemented by making use of the various Python libraries and modules.This book introduces you to the basic concepts and operations related to Python and teaches you how to work with the various Python libraries like NumPy, Scipy, Matplotlib, and Pandas for quantitative analysis. You will work with time-series data, and implement concepts like stochastics for Monte-Carlo simulation, hedging, derivatives, portfolio optimization and more.This book is a hands-on guide with easy-to-follow examples to help you learn about option theory, quantitative finance, financial modeling, and time series using Python.

BookJun 2017586 pages

Practical Time Series Analysis

Practical Time Series Analysis will introduce you to the basic concepts of time series analysis and describe powerful yet simple techniques in Python which data scientists and data engineers would find useful in dealing with real life datasets in industrial settings. This book focuses on explaining important concepts and practical techniques to process, summarize and model time series data. Real life case studies with code snippets in Python are used to demonstrate the concepts and techniques.

BookSep 2017244 pages

Mastering Python for Finance

This book enables you to develop financial applications by harnessing Python’s strengths in data visualization, interactive analytics, and scientific computing. You will be using popular libraries such as TensorFlow, Keras, scikit-learn, and so on to extend the functionalities of your financial applications by using smart machine learning techniques.

BookApr 2019426 pages

Hands-On Python for Finance

With this book, you will learn and implement various Quantitative Finance concepts using popular Python libraries like Numpy, pandas, Keras and more. We provide techniques to apply statistical methods used for data preprocessing and predict some of the best real-world case scenarios like stock prediction, sales prediction and many examples as such.

BookMar 2019378 pages

Learn Algorithmic Trading

This book will provide knowledge and hands-on practical experience required to build a good understanding of how modern electronic trading markets and market participants operate. You will learn how to design, build and operate all the components required to build a practical and profitable algorithmic trading business using Python.

BookNov 2019394 pages

Python for Finance Cookbook

Python is becoming the number one language for data science and also quantitative finance. This book provides you with solutions to common tasks from the intersection of quantitative finance and data science, using modern Python libraries.

BookJan 2020432 pages

Hands-On Time Series Analysis with R

This book introduces you to time series analysis and forecasting with R; this is one of the key fields in statistical programming and includes techniques for analyzing data to extract meaningful insights. You will explore methods, such as prediction with time series analysis, and identify the relationship between each data point in the series.

BookMay 2019448 pages

Hands-On Data Science with Anaconda

Hands-On Data Science with Anaconda gets you started with Anaconda and demonstrates how you can use it to perform data science operations in the real world. You will learn different ways to retrieve data from various sources and different visualization tools packages available in Python, R, and Julia.

BookMay 2018364 pages

SAS for Finance

SAS is the ground-breaking tool for advanced, predictive, and statistical analytics. Right from refining your data using power of SAS analytics, you will be able to exploit the capabilities of high-powered package to create accurate financial models. You can easily assess the pros and cons of models to suit unique business needs.

BookMay 2018306 pages

Statistical Application Development with R and Python

Statistical Analysis involves collecting and examining data to describe the nature of data that needs to be analyzed. It helps you explore the relation of data and build models to make better decisions. You will begin with a brief understanding of the nature of data and end with modern and advanced statistical models like CART. Every step is taken with DATA and R code, and further enhanced by Python. By the end of this book you will be able to apply your statistical learning in major domains at work or in your projects.

BookAug 2017432 pages

Hands-On Financial Trading with Python

This book focuses on key Python analytics and algorithmic trading libraries used for backtesting. With the help of practical examples, you will learn the principle aspects of trading strategy development. The 14 profitable strategies included in the book will also help you build intuitions that will enable you to create your own strategy.

BookApr 2021360 pages

Machine Learning for Algorithmic Trading

This thoroughly revised and expanded second edition demonstrates on over 800 pages how machine learning can add value to algorithmic trading in a practical yet comprehensive way. It has four parts that cover how to work with a diverse set of market, fundamental, and alternative data sources, design ML solutions for real-world trading challenges, and manage the strategy development process from idea to backtesting and evaluation.

BookJul 2020822 pages

Personalised recommendations for you

Based on your interests and search pattern

Et al.

Ever wonder why speech recognition systems don't understand the Scottish accent, or what would happen if an astronaut only ate mac 'n' cheese, or other spurious reflections you'd have at a bar? We did, then collated those deliberations into absurd research articles with fake figures and methodologies inspired by even more fictionally absurd studies.

BookAug 2023230 pages5

Generative AI with LangChain

This book is a comprehensive introduction to LLMs and LangChain, demystifying the basic mechanics of LangChain, its functionalities, and the myriad of applications it can be integrated into.

BookDec 2023360 pages4

Generative AI with LangChain

This book is a comprehensive introduction to LLMs and LangChain, demystifying the basic mechanics of LangChain, its functionalities, and the myriad of applications it can be integrated into.

BookDec 2023360 pages5

Generative AI with LangChain

This book is a comprehensive introduction to LLMs and LangChain, demystifying the basic mechanics of LangChain, its functionalities, and the myriad of applications it can be integrated into.

BookDec 2023360 pages1

Generative AI with LangChain

This book is a comprehensive introduction to LLMs and LangChain, demystifying the basic mechanics of LangChain, its functionalities, and the myriad of applications it can be integrated into.

BookDec 2023360 pages5

Mastering Tableau 2023

This book is a comprehensive resource to mastering your Tableau skills and becoming a BI expert. As you progress, you will learn how to build advanced dashboards and improve your storytelling to derive key business insight, as well as make you well-versed with advanced functionalities of Tableau in the business intelligence domain.

BookAug 2023684 pages

Building AI Applications with ChatGPT APIs

This guide covers all ChatGPT API features for effortless creation of robust AI powered apps. With its help, you’ll be able to leverage ChatGPT’s cutting-edge NLP models to take your app development skills to the next level. You’ll also work on ten exciting projects that will give you the practical know-how that you can apply to your existing applications.

BookSep 2023258 pages5

Building AI Applications with ChatGPT APIs

This guide covers all ChatGPT API features for effortless creation of robust AI powered apps. With its help, you’ll be able to leverage ChatGPT’s cutting-edge NLP models to take your app development skills to the next level. You’ll also work on ten exciting projects that will give you the practical know-how that you can apply to your existing applications.

BookSep 2023258 pages2

Data Engineering with AWS

Embark on a journey to master data engineering pipelines on AWS! Our book offers a hands-on experience of AWS services for ingesting, transforming, and consuming data. Whether you're an absolute beginner or someone with basic data engineering experience, this guide is an indispensable resource.

BookOct 2023636 pages5

Modern Data Architecture on AWS

Every organization wants an agile, performant, and cost-effective data platform that meets all their current and future business needs. Purpose-built AWS analytics services and their features play a big part in building such a modern data platform. This book brings to you all the design and architectural patterns that’ll help you achieve this goal.

BookAug 2023420 pages5

Practical Guide to Applied Conformal Prediction in Python

Discover the power of Conformal Prediction with the "Practical Guide to Applied Conformal Prediction in Python." Master the latest techniques to quantify uncertainty in machine learning and computer vision models, and seamlessly apply them to your industry applications.

BookDec 2023240 pages

TinyML Cookbook

With over 70 project-based recipes, the TinyML Cookbook is a practical guide that will help you to get the most out of your microcontrollers. It provides a comprehensive understanding of the theoretical foundations while giving you hands-on experience training ML models for deployment on Arduino Nano 33 BLE Sense, Raspberry Pi Pico, and SparkFun RedBoard Artemis Nano microcontrollers.

BookNov 2023664 pages

GO TO TOP

Machine LearningKubernetesReactPythonCybersecurityUnity Game DevelopmentPower BIAzureWeb Development

• Machine Learning
• Kubernetes
• React
• Python
• Cybersecurity
• Unity Game Development
• Power BI
• Azure
• Web Development

Legal

Terms and ConditionsPrivacy PolicyCookie PolicyShipping PolicyReturn Policy

Support

HelpContact Us

Business

Partnerships

Sponsored eBooks

Custom eBooks

CareersBecome an author

Packt+ Membership

SubscriptionDataProSecPro

Packt Subscription Packt NewsLetter

Legal

Terms and Conditions

Privacy Policy

Cookie Policy

Shipping Policy

Return Policy

Support

Help

Contact Us

Business

Packt Subscription

Packt NewsLetter

---

Company Address: Packt Publishing Ltd, Grosvenor House, 11 St Paul's Square, Birmingham, B3 1RB© 2024 Packt Publishing Limited All Rights Reserved

	Open	High	Low	Close
Open	1	0.962062	0.934174	0.878553
High	0.962062	1	0.952676	0.945434
Low	0.934174	0.952676	1	0.960428
Close	0.878553	0.945434	0.960428...

Hypothesis testing

Hypothesis testing is used to reject or retain a hypothesis based upon the measurement of an observed sample. We will not be going into theoretical aspects but will be discussing how to implement the various scenarios of hypothesis testing in R.

Lower tail test of population mean with known variance

The null hypothesis is given by  where is the hypothesized lower bound of the population mean.

Let us assume a scenario where an investor assumes that the mean of daily returns of a stock since inception is greater than $10. The average of 30 days' daily return sample is $9.9. Assume the population standard deviation is 0.011. Can we reject the null hypothesis at .05 significance level?

Now let us calculate the test statistics z which can be computed by the following code in R:

> xbar= 9.9           
> mu0 = 10            
> sig = 1.1            
> n = 30                  
> z = (xbar-mu0)/(sig/sqrt(n))  
> z  

Here:

• xbar: Sample...

Parameter estimates

In this section, we are going to discuss some of the algorithms used for parameter estimation.

Maximum likelihood estimation

Maximum likelihood estimation (MLE) is a method for estimating model parameters on a given dataset.

Now let us try to find the parameter estimates of a probability density function of normal distribution.

Let us first generate a series of random variables, which can be done by executing the following code:

> set.seed(100) 
> NO_values <- 100 
> Y <- rnorm(NO_values, mean = 5, sd = 1) 
> mean(Y) 

This gives 5.002913.

> sd(Y) 

This gives 1.02071.

Now let us make a function for log likelihood:

LogL <- function(mu, sigma) { 
+      A = dnorm(Y, mu, sigma) 
+      -sum(log(A)) 
+  } 

Now let us apply the function mle to estimate the parameters for estimating mean and standard deviation:

  > library(stats4) 
> mle(LogL, start = list(mu = 2, sigma=2)) 

mu and sigma have been...

Outlier detection

Outliers are very important to be taken into consideration for any analysis as they can make analysis biased. There are various ways to detect outliers in R and the most common one will be discussed in this section.

Boxplot

Let us construct a boxplot for the variable volume of the Sampledata, which can be done by executing the following code:

> boxplot(Sampledata$Volume, main="Volume", boxwex=0.1) 

The graph is as follows:

Figure 2.16: Boxplot for outlier detection

An outlier is an observation which is distant from the rest of the data. When reviewing the preceding boxplot, we can clearly see the outliers which are located outside the fences (whiskers) of the boxplot.

LOF algorithm

The local outlier factor (LOF) is used for identifying density-based local outliers. In LOF, the local density of a point is compared with that of its neighbors. If the point is in a sparser region than its neighbors then it is treated as an outlier. Let us consider some of the variables from...

Standardization

In statistics, standardization plays a crucial role as we have various attributes for modeling and all of them have different scales. So for comparison purposes, we need to standardize the variables to bring them on the same scale. Centering the values and creating the z scores is done in R by the scale() function. It takes the following arguments:

• x: A numeric object

• center: If TRUE, the object's column means are subtracted from the values in those columns (ignoring NAs); if FALSE, centering is not performed

• scale: If TRUE, the centered column values are divided by the column's standard deviation (when center is also TRUE; otherwise, the root mean square is used); if FALSE, scaling is not performed

If we want to center the data of Volume in our dataset, we just need to execute the following code:

>scale(Sampledata$Volume, center=TRUE, scale=FALSE) 

If we want to standardize the data of volume in our dataset, we just need to execute the following code:

>scale(Sampledata...

Normalization

Normalization is done using the minmax concept to bring the various attributes on the same scale. It is calculated by the formula given here:

normalized = (x-min(x))/(max(x)-min(x))

So if we want to normalize the volume variable, we can do it by executing the following code:

> normalized = (Sampledata$Volume-+min(Sampledata$Volume))/(max(Sampledata$Volume)-+min(Sampledata$Volume)) 
> normalized 

Questions

1. Construct examples of normal, Poisson, and uniform distribution in R.

2. How do you do random and stratified sampling in R?

3. What are the different measures of central tendency and how do you find them in R?

4. How do you compute kurtosis and skewness in R?

5. How do you do hypothesis testing in R with known/unknown variance of population in R?

6. How do you detect outliers in R?

7. How do you do parameter estimates for a linear model and MLE in R?

8. What is standardization and normalization in R and how do you perform it in R?

Summary

In this chapter, we have discussed the most commonly used distributions in the finance domain and associated metrics computations in R; sampling (random and stratified); measures of central tendencies; correlations and types of correlation used for model selections in time series; hypothesis testing (one-tailed/two-tailed) with known and unknown variance; detection of outliers; parameter estimation; and standardization/normalization of attributes in R to bring attributes on comparable scales.

In the next chapter, analysis done in R associated with simple linear regression, multivariate linear regression, ANOVA, feature selection, ranking of variables, wavelet analysis, fast Fourier transformation, and Hilbert transformation will be covered.