Practical Predictive Analytics: Analyse current and historical data to predict future trends using R, Spark, and more

One industry that has embraced PA for quite a long time is marketing. Marketing has always been concerned with customer acquisition and retention, and has developed predictive models involving various promotional offers and customer touch points, all geared to keeping customers and acquiring new ones. This is very pronounced in certain segments of marking, such as wireless and online shopping cards, in which customers are always searching for the best deal.

Specifically, advanced analytics can help answer questions such as, If I offer a customer 10% off with free shipping, will that yield more revenue than 15% off with no free shipping? The 360-degree view of the customer has expanded the number of ways one can engage with the customer, therefore enabling marketing mix and attribution modeling to become increasingly important. Location-based devices have enabled marketing predictive applications to incorporate real-time data to issue recommendation to the customer while in the store.

Predictive analytics in healthcare has its roots in clinical trials, which use carefully selected samples to test the efficacy of drugs and treatments. However, healthcare has been going beyond this. With the advent of sensors, data can be incorporated into predictive analytics to monitor patients with critical illness, and to send alerts to the patient when he is at risk. Healthcare companies can now predict which individual patients will comply with courses of treatment advocated by health providers. This will send early warning signs to all parties, which will prevent future complications, as well as lower the total cost of treatment.

Other examples can be found in just about every other industry. Here are just a few:

Although these industries can be quite different, the goals of predictive analytics are typically implemented to increase revenue, decrease costs, or alter outcomes for the better.

Along with the term predictive analytics, here are some terms that are very much related:

Open source emphasizes agile development and community sharing. Of course, open source software is free, but free must also be balanced in the context of Total Cost Of Ownership (TCO). TCO costs include everything that is factored into a softwares cost over a period of time: that not only includes the cost of the software itself, but includes training, infrastructure setup, maintenance, people costs, as well as other expenses associated with the quick upgrade and development cycles which exist in some products.

Closed source (or proprietary) software such as SAS and SPSS was at the forefront of predictive analytics, and has continued to this day to extend its reach beyond the traditional realm of statistics and machine learning. Closed source software emphasizes stability, better support, and security, with better memory management, which are important factors for some companies.

There is much debate nowadays regarding which one is better. My prediction is that they both will coexist peacefully, with one not replacing the other. Data sharing and common APIs will become more common. Each has its place within the data architecture and ecosystem that are deemed correct for a company. Each company will emphasize certain factors, and both open and closed software systems are constantly improving themselves. So, in terms of learning one or the other, it is not an either/or decision. Predictive analytics, per second does not care what software you use. Please be open to the advantages offered by both open and closed software. If you do, that will certainly open up possibilities for working for different kinds of companies and technologies

Given that the predictive analytics space is so huge, once you are past the basics, ask yourself what area of predictive analytics really interests you, and what you would like to specialize in. Learning all you can about everything concerning predictive analytics is good at the beginning, but ultimately you will be called upon because you are an expert in certain industries or techniques. This could be research, algorithmic development, or even managing analytics teams.

But, as general guidance, if you are involved in, or are oriented toward, data, the analytics or research portion of data science, I would suggest that you concentrate on data mining methodologies and specific data modeling techniques that are heavily prevalent in the specific industries that interest you.

For example, logistic regression is heavily used in the insurance industry, but social network analysis is not. Economic research is geared toward time series analysis, but not so much cluster analysis. Recommender engines are prevalent in online retail.

If you are involved more on the data engineering side, concentrate more on data cleaning, being able to integrate various data sources, and the tools needed to accomplish this.

If you are a manager, concentrate on model development, testing and control, metadata, and presenting results to upper management in order to demonstrate value or return on investment.

Of course, predictive analytics is becoming more of a team sport, rather than a solo endeavor, and the data science team is very much alive. There is a lot that has been written about the components of a data science team, much of which can be reduced to the three basic skills that I outlined earlier.

Various industries interpret the goals of predictive analytics differently. For example, social science and marketing like to understand the factors which go into a model, and can sacrifice a bit of accuracy if a model can be explained well enough. On the other hand, a black box stock trading model is more interested in minimizing the number of bad trades, and at the end of the day tallies up the gains and losses, not really caring which parts of the trading algorithm worked. Accuracy is more important in the end.

Depending upon how you intend to approach a particular problem, look at how two different analytical mindsets can affect the predictive analytics process:

Of course, the previous examples illustrate two disparate approaches. Combination models, which use the best of both worlds, should be the ones we should strive for. Therefore, one goal for a final model is one which:

You will learn later that is this related to Bias/Variance tradeoff.

The Comprehensive R Archive Network (CRAN) is a go-to site which aggregates R distributions, binaries, documentation, and packages. To get a sense of the kind of packages that could be valuable to you, check out the Task Views section maintained by CRAN here:

https://cran.r-project.org/web/views/

R installation is typically done by downloading the software directly from the Comprehensive R Archive Network (CRAN) site:

Although installing R directly from the CRAN site is the way most people will proceed, I wanted to mention some alternative R installation methods. These methods are often good in instances when you are not always at your computer:

We will start by creating folders for our environment. Often projects start with three subfolders which roughly correspond to:

There may be more in certain cases, but lets keep it simple:

Here is an example of how the structure will look after you have created the folders:

To rearrange the layout, see the following steps:

After the changes are applied the layout should more closely match the layout previously shown . However, it may not match exactly. A lot will depend upon the version of RStudio that you are using as well as the packages you may have already installed.

Once you are set with your layout, proceed to create a new project by following these steps:

Create a new project by following these steps:

All screen panes will then appear as blank (except for the log), and the title bar at the top left of the screen will show the path to the project.

To verify that the R, outputs, and data directories are contained within the project, select File, and then File Open from the top menu bar. The three folders should appear, as indicated as follows:

Once you have verified this, cancel the Open File dialogue, and return to RStudio main screen.

To create a new script select File | New File | R Script from the top navigation bar. A new blank script window will appear with the name Untitled1.

Once it appears you can start entering code!

Here are some more details on what the code has accomplished:

Here are some notes and explanations for the script code that you have just run:

        lm_output <- lm(women$height ~ women$weight) 

                      attach(women)
                      lm_output <- lm(height ~ weight)

        summary(lm_output) 

        Coefficients:
                    Estimate Std. Error t value Pr(>|t|)   
      (Intercept)  25.723456   1.043746   24.64 2.68e-12 ***
      women$weight  0.287249   0.007588   37.85 1.09e-14 ***

To predict the value for all of the values we will use a function called predict(). This function reads each input (independent) variable and then predicts a target (dependent) variable based on the linear regression equation. In the code we have assigned the output of this function to a new object named prediction.

Switch over to the console area, and type prediction, then Enter, to see the predicted values for the 15 women. The following should appear in the console.

> prediction
       1        2        3        4        5        6        7  
58.75712 59.33162 60.19336 61.05511 61.91686 62.77861 63.64035  
       8        9       10       11       12       13       14  
64.50210 65.65110 66.51285 67.66184 68.81084 69.95984 71.39608  
      15  
72.83233  

Notice that the value of the first prediction is very close to what you just calculated by hand. The difference is due to rounding error.

Another R object produced by our linear regression is the error object. The error object is a vector that was computed by taking the difference between the predicted value of height and the actual height. These values are also known as the residual errors, or just residuals.

error <- women$height-prediction

Since the error object is a vector, you cannot use the nrow() function to get its size. But you can use the length() function:

>length(error)
[1] 15

In all of the previous cases, the counts all total 15, so all is good. If we want to see the raw data, predictions, and the prediction errors for all of the data, we can use the cbind() function (Column bind) to concatenate all three of those values, and display as a simple table.

At the console enter the follow cbind command:

> cbind(height=women$height,PredictedHeight=prediction,ErrorInPrediction=error)
   height PredictedHeight ErrorInPrediction
1      58        58.75712       -0.75711680
2      59        59.33162       -0.33161526
3      60        60.19336       -0.19336294
4      61        61.05511       -0.05511062
5      62        61.91686        0.08314170
6      63        62.77861        0.22139402
7      64        63.64035        0.35964634
8      65        64.50210        0.49789866
9      66        65.65110        0.34890175
10     67        66.51285        0.48715407
11     68        67.66184        0.33815716
12     69        68.81084        0.18916026
13     70        69.95984        0.04016335
14     71        71.39608       -0.39608278
15     72        72.83233       -0.83232892

From the preceding output, we can see that there are a total 15 predictions. If you compare the ErrorInPrediction with the error plot shown previously, you can see that for this very simple model, the prediction errors are much larger for extreme values in height (shaded values).

Just to verify that we have one for each of our original observations we will use the nrow() function to count the number of rows.

At the command prompt in the console area, enter the command:

nrow(women)

The following should appear:

>nrow(women)
[1] 15

Refer back to the seventh line of code in the original script: plot(women$height,error) plots the predicted height versus the errors. It shows how much the prediction was off from the original value. You can see that the errors show a non-random pattern.

After you are done, save the file using File | File Save, navigate to the PracticalPredictiveAnalytics/R folder that was created, and name it Chapter1_LinearRegression.

For an example of a package which can just make life easier, first lets consider the output produced by running a summary function on the regression results, as we did previously. You can run it again if you wish.

summary(lm_output)

The amount of statistical information output by the summary() function can be overwhelming to the initiated. This is not only related to the amount of output, but the formatting. That is why I did not show the entire output in the previous example.

One way to make output easier to look at is to first reduce the amount of output that is presented, and then reformat it so it is easier on the eyes.

To accomplish this, we can utilize a package called stargazer, which will reformat the large volume of output produced by summary() function and simplify the presentations. Stargazer excels at reformatting the output of many regression models, and displaying the results as HTML, PDF, Latex, or as simple formatted text. By default, it will show you the most important statistical output for various models, and you can always specify the types of statistical output that you want to see.

To obtain more information on the stargazer package you can first go to CRAN, and search for documentation about stargazer package, and/or you can use the R help system:

IF you already have installed stargazer you can use the following command:

packageDescription("stargazer")

If you havent installed the package, information about stargazer, (or other packages) can also be found using R specific internet searches:

RSiteSearch("stargazer")

If you like searching for documentation within R, you can obtain more information about the R help system at:

https://www.r-project.org/help.html

Now, on to installing stargazer:

        install.packages("stargazer") 
        library(stargazer) 
        stargazer(lm_output, , type="text")

After the script has been run, the following should appear in the Console:

After you are done, select File | File Save from the menu bar.

Then navigate to the PracticalPredictiveAnalytics/Outputs folder that was created, and name it Chapter1_LinearRegressionOutput. Press Save.