Chapter 2. The Modeling Process

Remember that all models are wrong; the practical question is how wrong do they have to be to not be useful.

-George Edward Pelham Box

Today, we are at a juncture in which many different types of skill sets are needed to participate in predictive analytics projects. Once, this was the pure domain of statisticians, programmers, and business analysts. Now, the roles have expanded to include visualization experts, data storage experts, and other types of specialists. Yet, so many are unfamiliar with an understanding of how predictive analytics projects can be structured. This lack of structure can be inhibited by several factors. Often there is a lack of understanding of the critical parts of a business problem, and a model is developed much too early. Alternatively, a formal methodology may be put off to the future, in favor of a quick solution.

In this chapter, we will start by discussing the advantages of using structured analytics methodologies. Methodologies...

Advantages of a structured approach

Analytic projects have many components. That is where a structured methodology can help. Many benefits can be gained if there is a structure which is placed upon discovery and analysis, rather than only on pure model building. The discovery and insight gained will certainly be utilized past the original intent of the problem.

We assume that the quick-thinking "hare brain" will beat out the slower Intuition of the "tortoise mind." However, now research in cognitive science is changing this understanding of the human mind. It suggests that patience and confusion--rather than rigor and certainty--are the essential precursors of wisdom.

-Guy Claxton

Ways in which structured methodologies can help

Here are several points to bear in mind concerning the advantages of structured methodologies:

• Data is coming at us fast and furious. We need to keep track of the many data sources, evaluate which ones are the best ones to use at any given time and continually monitor...

Analytic process methodologies

There are several analytic process methodologies which are currently practiced; however, I will be discussing only two longstanding methodologies that have been in existence for a while, CRISP-DM and SEMMA, which can help you organize your journey from problem definition to insight.

CRISP-DM and SEMMA

Cross-Industry Standard process for Data Mining (CRISP-DM) and Sample, Explore, Modify, Model, and Assess (SEMMA) are two standard data mining methodologies that have been utilized for many years and describe a general methodology for implementing analytical projects. There is a good deal of overlap between the methodologies, even though the names for each step are different. All of the listed steps are important to the success of a predictive analytics project. However, it is not necessary that these steps be followed exactly in order. The concepts outlined are more or less an outline of best practices. It helps to be aware of the importance of each of these steps...

An analytics methodology outline specific steps

This section will look at each of the analytics methodology steps individually. I will use CRISP-DM as the template, because it covers model deployment, and we have already mentioned the benefits of sampling (which is the first step in SEMMA).

Step 1 business understanding

Many predictive modelers assume that the actual modeling phase is where the most insightful model development takes place. However, much of the groundwork and insight can be discovered early on, and a good understanding of business objectives can avoid pitfalls later on.

Communicating business goals the feedback loop

I must admit, business people and technical people can be better at communicating with each other. How business goals are communicated can run the gamut. It can be anything from a business partner stating, "Tell me how sales need to be increased" or "Tell me something I don't know."

So, it really starts with understanding what the specific business objectives are...

Step 2 data understanding

Once an objective is established and data sources have been identified, you can begin looking at the data in order to understand how each data element behaves individually, as well as how it interacts in combination with other variables. But even before you start looking at the values of variables, it is important to understand the different types of data levels of measurement and the kind of analyses you can perform with them.

Levels of measurement

Levels of measurement is a classification system for classifying data into 4 different categories which is discussed as follows (ratio, ordinal, interval, and nominal). It is an important aspect of the project or studies metadata.

Levels of measurement is important in the world of predictive analytics since the specific measurements will often dictate which algorithm or techniques can be applied. For example k-means clustering does work if you want to incorporate nominal data, and logistic regression can not use ratio data...

Step 3 data preparation

As was mentioned in Chapter 1, Getting Started with Predictive Analysis, one purpose of data preparation is preparing an input data modeling file, which can go directly into an algorithm. In theory, the input file will encompass all of the knowledge gained in steps 1 and 2. Ideally, this file will consist of a target variable, all meaningful predictor variables and other identification variables to aid in the modeling process, and any additional variables which would have been created based on the raw data sources. Data preparation, such as the previous steps outlined is an iterative process. Here are some typical steps you might follow when preparing the data:

• Identifying the data sources: These are the critical data inputs that you will need to read in and manipulate. They can be sourced from various data formats such as CSV files, databases, or XML or JSON files. They can be in structured format or unstructured format.
• Identify the expected input: Read in some test...

Step 4 modeling

In the modeling stage, you will pick an appropriate predictive modeling technique that fits your problem and apply it to your data. There are several factors which influence the selection of a model:

1. Who will use the model?
2. How will the model be used?
3. What are the assumptions of the model?
4. How much data do I have?
5. How many variables do I need to use?
6. What is the accuracy level needed by the model?
7. Am I willing to trade some accuracy for interpretability?

Particularly related to the last point is the concept of bias and variance.

Bias is related to the ability of a model to approximate the data. Low bias algorithms are able to fit the data with little error. While this may seem to an advantage all of the time, it can result in a complex model which is unstable, and difficult to explain. On the other hand, a high bias model is relatively simple to explain (like linear regression), but may sacrifice some accuracy for explanability, and stability. You will usually start by looking at...

Step 5 evaluation

Model evaluation deals with how accurate or useful the model you have just developed is or will be in the future. Model evaluation can take different forms. Some are more subjective and are domain oriented, such as placing it under the scrutiny of experts in your field, and some are more technically oriented. There are many metrics and procedures available to assess a model. At the basic level, you have many statistics (some of them with acronyms known as AIC, BIC, and AUC) which purport to convey the goodness of a model in a single metric. However, these metrics by themselves are unable to convey the purpose and application of a predictive model to a larger audience and often these metrics are in conflict. Some context is needed. Some would argue that one could also develop a perfectly good predictive model and then be unable to convey its purpose and application to a larger audience. In my opinion, that is a bad model, regardless of how well an evaluation metric fits...

Step 6 deployment

Deployment of a model is the process by which you put your models into a real-world production setting. This can depend on many factors, such as the environment in which it was developed, the algorithm that was chosen, assumptions concerning the data that was made when the model was developed, and of course, the level of the developer. Often a model is unable to scale up to the demands of a production environment and knowing your possible production environment in advance will dictate what problems or techniques are feasible.

Model scoring

Model scoring makes the model actionable. If you develop a model and you are unable to apply the results to new data, then you will be unable to do any prediction on an ongoing basis. New model scoring often involves outputing the development model outputs to a real-time scoring engine. That engine is often Java or C++. How that is performed varies vastly depending upon the modeling technique. Sometimes the scoring is performed separately...

References

You can refer to the following articles:

• Determine the Root Cause: 5 Whys. Retrieved from https://www.isixsigma.com/tools-templates/cause-effect/determine-root-cause-5-whys/
• Gettysburg Address, (1863, November 19). Retrieved from Abraham Lincoln Online: http://www.abrahamlincolnonline.org/lincoln/speeches/gettysburg.html
• Stevens, S. (1946). On the Theory of Scales of Measurement. Science
• Wikipedia. Bias-variance tradeoff. Retrieved from Wikipedia: https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff

Notes

Random Forests (tm) is a trademark of Leo Breiman and Adele Cutler and is licensed exclusively to Salford Systems for the commercial release of the software.

Summary

In this chapter, we learned about the various structured approaches to predictive analytics and how implementing an analytics project in a methodical way can enhance the success of an analytics project through collaboration and communication. We went through the various steps of the CRISP-DM methodology and demonstrated tools that you could use to help you progress along these steps.

We discussed the benefits of sampling and how it could speed up your project. SQL was demonstrated to illustrate basic charts and plots, so that you can begin to develop insight even before you create a first model. We showed that data simulation could also be used at the data understanding phase as a preliminary modeling tool to do "what ifing", even before actual company data is obtained.

We learned about the various types of data that you will encounter, and showed some examples of independent and dependent variables and the importance of doing preliminary 1-way and 2-way variable analysis as a precursor...