Mastering Predictive Analytics with Python: Exploit the power of data in your business by building advanced predictive modeling applications with Python

The beginning of any analytic pipeline is the data itself, which serves as the basis for predictive modeling. This input can vary both in the rate at which updates are available and the amount of transformation that needs to be applied to form the final set of features used in the predictive model. The data layer serves as the repository for this information.

Traditionally, data used for analytics might simply be stored on disk in flat files, such as a spreadsheet or document. As the diversity and scale of data have increased, so have the scale and complexity of resources needed to house and process them. Indeed, a modern view of the data layer encompasses both real-time (stream) data and batch data in the context of many potential downstream uses. This combined system, known as Lambda Architecture (Marz, Nathan, and James Warren. Big Data: Principles and best practices of scalable realtime data systems. Manning Publications Co., 2015.), is diagrammed in the following figure:

Figure 2: Data layer as a Lambda Architecture

The components of this data layer are:

How might streaming and batch data be processed differently in the data layer? In batch pipelines, the allowed delay between receiving and processing the data allows for potentially complex transformations of the source data: elements may be aggregated (such as calculating a user or product's average properties over a period of time), joined to other sources (for example, indexing additional website metadata on search logs), and filtered (for example, many web logging systems need to remove bot activity that would otherwise skew the results of predictive models). The source data could be obtained, for example, from simple text files posted to a server, a relational database system, or a mixture of different storage formats (see as follows).

Conversely, due to the speed at which incoming data must often be consumed, streaming processes typically involve less complex processing of inputs than batch jobs, and instead use simple filters or transformations. The sources for such applications are typically continuously updated streams from web services (such as social media or news feeds), events (such as geo-locations of vehicles and mobile phones), or customer activities (such as searches or clicks).

The choice between batch and stream processing at this stage is largely determined by the data source, which is either available as a continuously updated series of events (streaming) or larger, periodically available chunks (batch). In some cases, the nature of the data will also determine the form of the subsequent pipeline and an emphasis on real-time or higher latency processing. In others, the use of the application will take precedent in downstream choices. The normalized view surfaced in the data layer is used downstream in the next stage of the analytic pipeline, the modeling layer.

The modeling layer involves a number of interconnected tasks, diagrammed in the following figure (Figure 3). As the data layer accommodates both real-time and batch data, we can imagine two main kinds of modeling systems:

If both types of pipeline are viable for a given problem (for example, if the streams are stock prices, a dataset whose volume and simple format – a set of numbers – should allow it to be readily stored offline and processed in its entirety at a later date), the choice between the two frameworks may be dictated by technical or business concerns. For example, sometimes the method used in a predictive model allows only for batch updates, meaning that continuously processing a stream as it is received does not add additional value. In other cases, the importance of the business decisions informed by the predictive model necessitates real-time updates and so would benefit from stream processing.

Figure 3: Overview of the modeling layer

The details of the generic components of each type of pipeline as shown in Figure 3 are as follows:

In the Model Input step the source data is loaded and potentially transformed by the pipeline into the inputs required for a predictive model. This can be as simple as exposing a subset of columns in a database table, or transforming an unstructured source such as text into a form that may be input to a predictive model. If we are fortunate, the kinds of features we wish to use in a model are already the form in which they are present in the raw data. In this case, the model fitting proceeds directly on the inputs. More often, the input data just contains the base information we might want to use as inputs to our model, but needs to be processed into a form that can be utilized in prediction.

In the case of numerical data, this might take the form of discretization or transformation. Discretization involves taking a continuous number (such as consumer tenure on a subscription service) and dividing it into bins (such as users with <30 or >=30 days of subscription) that either reduce the variation in the dataset (by thresholding an outlier on a continuous scale to a reasonable bin number) or turn a numerical range into a set of values that have more direct business implications. Another example of discretization is turning a continuous value into a rank, in cases where we don't care as much about the actual number as its relative value compared to others. Similarly, values that vary over exponential scales might be transformed using a natural logarithm to reduce the influence of large values on the modeling process.

In addition to these sorts of transformations, numerical features might be combined in ratios, sums, products, or other combinations, yielding a potential combinatorial explosion of features from even a few basic inputs. In some models, these sorts of interactions need to be explicitly represented by generating such combined features between inputs (such as the regression models we discuss in Chapter 4, Connecting the Dots with Models – Regression Methods). Other models have some ability to decipher these interactions in datasets without our direct creation of the feature (such as random forest algorithms in Chapter 5, Putting Data in its Place – Classification Methods and Analysis or gradient boosted decision trees in Chapter 6, Words and Pixels – Working with Unstructured Data).

In the case of categorical data, such as country codes or days of the week, we may need to transform the category into a numerical descriptor. This could be a number (if the data is ordinal, meaning for example that a value of 2 has an interpretation of being larger than another record with value 1 for that feature) or a vector with one or more non-zero entries indicating the class to which a categorical feature belongs (for example, a document could be represented by a vector the same length as the English vocabulary, with a number indicating how many times each word represented by a particular vector position appears in the document).

Finally, we might find cases where we wish to discover the hidden features represented by a particular set of inputs. For example, income, occupation, and age might all be correlated with the zip code in which a customer lives. If geographic variables aren't part of our dataset, we could still discover these common underlying patterns using dimensionality reduction, as we will discuss in Chapter 6, Words and Pixels – Working with Unstructured Data.

Sanity checking may also be performed at this stage, as it is crucial to spot data anomalies when they appear, such as outliers that might degrade the performance of the model. In the first phase of quality checks, the input data is evaluated to prevent outliers or incorrect data from impacting the quality of models in the following stages. These sanity checks could take many forms: for categorical data (for example, a state or country), there are only a fixed number of allowable values, making it easy to rule out incorrect inputs. In other cases, this quality check is based on an empirical distribution, such as variation from an average value, or a sensible minimum or maximum range. More complex scenarios usually arise from business rules (such as a product being unavailable in a given territory, or a particular combination of IP addresses in web sessions being illogical).

Such quality checks serve as more than safeguards for the modeling process: they can also serve as warnings of events such as bot traffic on websites that may indicate malicious activity. Consequently, these audit rules may also be incorporated as part of the visualization and reporting layer at the conclusion of the pipeline.

In the second round of quality checks following model development, we want to evaluate whether the parameters of the model make sense and whether the performance on the test data is in an acceptable range for deployment. The former might involve plotting the important parameters of a model if the technique permits, visualizations that can then also be utilized by the reporting step downstream. Similarly, the second class of checks can involve looking at accuracy statistics such as precision, recall, or squared error, or the similarity of the test set to data used in model generation in order to determine if the reported performance is reasonable.

As with the first round of sanity checks, not only can these quality control measures serve to monitor the health of the model development process, but also potentially highlight changes in the actual modeling code itself (especially if this code is expected to be regularly updated).

There isn't inherently much difference between streaming and batch-oriented processing in the sanity checking process, just the latency at which the application can uncover anomalies in the source data or modelling process and deliver them to the reporting layer. The complexity of the sanity checks may guide this decision: simple checks that can be done in real-time are well suited for stream processing, while evaluation of the properties of a predictive model could potentially take longer than the training of the algorithm itself, and is thus more suited for a batch process.

In the model development or update step, once the input data has undergone any necessary processing or transformation steps and passed the quality checks described above, it is ready to be used in developing a predictive model. This phase of the analytic pipeline can have several steps, with the exact form depending upon the application:

The output of our predictive modeling can be made broadly available to both individual users and other software services through a deployment layer, which encapsulates the modeling, scoring, and evaluation functions in the previous layer inside of web applications, as shown in the following Figure 4:

Figure 4: Deployment layer components

This application layer receives network calls over the web, transmitted either through a web browser or from a programmatic request generated by another software system. As we will describe in Chapter 8, Sharing Models with Prediction Services, these applications usually provide a standard set of commands to initiate an action, get a result, save new information, or delete unwanted information. They also typically interact with the data layer to both store results and, in the case of long-running tasks, to store information about the progress of modeling computations.

The network calls received by these applications are brokered by the Server Layer, which serves to route traffic between applications (usually based on url patterns). As we will cover in Chapter 8, Sharing Models with Prediction Services, this separation between the server and application allows us to scale our application by adding more machines, and independently add more servers to balance incoming requests.

The client layer, which initiates the requests received by the server, could be both interactive systems, such as a dashboard, or an independent system such as an e-mail server, that uses the output of a model to schedule outgoing messages.

The output of the analytical pipeline may be surfaced by the reporting layer, which involves a number of distinct tasks, as shown in the following Figure 5:

Figure 5: Reporting applications for prediction services

The input data to this application are social media posts. This data is available in real time, but a number of steps need to be applied to make it usable by the sentiment-scoring model. Common words (such as and and the) need to be filtered, messages to be selected which actually refer to the company, and misspellings and word capitalization need to be normalized. Once this cleaning is done, further transformations may turn the message into a vector, with a count of each word in the model's allowed vocabulary, or hashed to populate a fixed-length vector.

The outputs of the preceding transformations need to be sanity checked – are there any users who account for an unusually large number of messages (which might indicate bot spam)? Are there unexpected words in the input (which could be due to character encoding issues)? Are any of the input messages longer than the allowed message size for the service (which could indicate incorrect separation of messages in the input stream)?

Once the model is developed, sanity checking involves some human guidance. Do the sentiments predicted by the model correlate with the judgment of human readers? Do the words that correspond to high probability for a given sentiment in the model make intuitive sense?

These and other sanity checks can be visualized as a webpage or document summary that can be utilized by both the modeler, to evaluate model health, and the rest of the marketing staff to understand new topics that may correspond to positive or negative brand sentiment.

The model used in this pipeline is a multinomial logistic regression (Chapter 5, Putting Data in its Place – Classification Methods and Analysis) that takes as input counts of the words in each social media message and outputs a predicted probability that the message belongs to a given sentiment category: VERY POSITIVE, POSITIVE, NEUTRAL, NEGATIVE, and VERY NEGATIVE. While in theory (because the multinomial logistic regression can be trained using stochastic gradient updates), we could perform model training online, in practice this is not possible because the labels (sentiments) need to be assigned by a human expert. Therefore, our model is developed in an offline batch-process each week as a sufficient set of social media messages labelled by an expert becomes available. The hyperparameters of this model (the regularization weight and learning weight) have been estimated previously, so the batch retraining calculates the regression coefficient weights for a set of training messages and evaluates the performance on a separate batch of test messages.

Incoming messages processed by this pipeline can be scored by the existing model and assigned to one of the five sentiment classes, and the volume of each category is updated in real time to allow monitoring of brand sentiment and immediate action if there is an extremely negative response to one of the marketing department's campaigns.

As the model scores new social media messages, it updates a real-time dashboard with the volume of messages in each category compared to yesterday, the preceding week, and the preceding month, along with which words are given most weight in this week's model for the different classes. It also monitors the presence of new words, which may not have been present in the model's vocabulary, and which could indicate new features that the model cannot appropriately score, and suggest the need for inter-week retraining. In addition to this real-time dashboard, which the marketing department uses to monitor response to its campaigns, the analyst develops a more detailed report concerning model parameters and performance along with input dataset summary statistics, which they use to determine if the model training process each week is performing as expected, or if the quality of the model is degrading over time.

During the initial data ingestion step, customer records stored in a company's data warehouse (a relational database system) are aggregated to generate features such as the average amount spent per week, frequency with which a customer visits the company's website, and the number of items purchased in a number of categories, such as furniture, electronics, clothing, and media. This is combined with a set of features for the set of items that are potentially promoted in the e-mail campaign, such as price, brand, and the average rating of similar items on the site. These features are constructed through a batch process that runs once per week, before e-mails are sent, on Mondays, to customers.

The inputs to the model are checked for reasonable values: are the average purchase behaviors or transactions volume of a customer far outside the expected range? These could indicate errors in the data warehouse processing, or bot traffic on the website. Because the transformation logic involved in constructing features for the model is complex and may change over time as the model evolves, its outputs are also checked. For example, the purchase numbers and average prices should never be less than zero, and no category of merchandise should have zero records.

Following scoring of potential items prior to e-mail messaging, the top-scoring items per customer are sanity checked by comparing them to either the customer's historical transactions (to determine if they are sensible), or if no history is available, to the purchases of customers most similar in demographics.

In this example, the model is a random forest regression Chapter 4, Connecting the Dots with Models – Regression Methods that divides historical items – customer pairs into purchases (labeled 1) and non-purchases (labeled 0) and produces a scored probability that customer A purchases item X. One complexity in this model is that items which haven't been purchased might simply not have been seen by the customer yet, so a restriction is imposed in which the negative examples must be drawn from items already available for a month or more on the website. The hyperparameters of this model (the number and size of each tree) are calibrated during weekly retraining, along with the influence of individual variables on the resulting predictions.

After the model is retrained each week using historical data, the set of new items on the website are scored using this model for each customer, and the top three are sent in the e-mail campaign.

Either class of sanity checking (of either input data or model performance) can be part of a regular diagnostics report on the model. Because the random forest model is more complex than other approaches, it is particularly important to monitor changes in feature importance and model accuracy as problems may require more time to debug and resolve.

Because the predictions are used in a production system rather than delivering insights themselves, this reporting is primarily used by the analyst who developed the pipeline rather than the other members of the marketing department.

The success of these promotional e-mails will typically be monitored over the next month, and updates on the accuracy (for example, how many e-mails led to purchases above expected levels) can form the basis of a longer-term report that can help guide both the structure of the campaign itself (for example, varying the number of items in the messages) and the model (perhaps training should be performed more frequently if the predictions seem to become significantly worse between weeks).