Descriptive data exploration is, without a doubt, one of the most important steps in an ML project. If you want to clean data and build derived features or select an ML algorithm to predict a target variable in your dataset, then you need to understand your data first. Your data will define many of the necessary cleaning and preprocessing steps; it will define which algorithms you can choose and it will ultimately define the performance of your predictive model.

Hence, data exploration should be considered an important analytical step to understanding whether your data is informative to build an ML model in the first place. By analytical step, we mean that the exploration should be done as a structured analytical process rather than a set of experimental tasks. Therefore, we will go through a checklist of data exploration tasks that you can perform as an initial step in every ML project—before starting any data cleaning, preprocessing, feature engineering, or model selection.

Once the data is provided, we will work through the following data exploration checklist and try to get as many insights as possible about the data and its relation to the target variable:

1. Analyze the data distribution and check for the following:
   • Data types (continuous, ordinal, nominal, or text)
   • Mean, median, and percentiles
   • Data skew
   • Outliers and minimum and maximum values
   • Null and missing values
   • Most common values
   • The number of unique values (in categorical features)
   • Correlations (in continuous features)
2. Analyze how the target variable is influenced by the features and check for the following:
   • The regression coefficient (in regression)
   • Feature importance (in classification)
   • Categorical values with high error rates (in binary classification)
3. Analyze the difficulty of your prediction task.

By applying these steps, you will be able to understand the data and gain knowledge about the required preprocessing tasks for your data—features and target variables. Along with that, it will give you a good estimate of what difficulties you can expect in your prediction task, which is essential for judging required algorithms and validation strategies. You will also gain an insight into what possible feature engineering methods could apply to your dataset and have a better understanding of how to select a good error metric.

Note

You can use a representative subset of the data and extrapolate your hypothesis and insights to the whole dataset

Moving data to the cloud

Before we can start exploring the data, we need to make it available in our cloud environment. While this seems like a trivial task, efficiently accessing data from a new environment inside a corporate environment is not always easy. Also, uploading, copying, and distributing the same data to many Virtual Machines (VMs) and data science environments is not sustainable and doesn't scale well. For data exploration, we only need a significant subset of the data that can easily be connected to all other environments—rather than live access to a production database or data warehouse.

There is no wrong practice of uploading Comma-Separated Values (CSV) or Tab-Separated Values (TSV) files to your experimentation environment or accessing data via Java Database Connectivity (JDBC) from the source system. However, there are a few easy tricks to optimize your workflow.

First, we will choose a data format optimized for data exploration. In the exploration phase, we need to glance at the source data multiple times and explore the values, feature dimensions, and target variables. Hence, using a human-readable text format is usually very practical. In order to parse it efficiently, a delimiter-separated file, such as CSV, is strongly recommended. CSV can be parsed efficiently and you can open and browse it using any text editor.

Another small tweak that will bring you a significant performance improvement is compressing the file using Gzip before uploading it to the cloud. This will make uploads, loading, and downloads of this file much faster, while the compute resources spent on decompression are minimal. Thanks to the nature of the tabular data, the compression ratio will be very high. Most analytical frameworks for data processing, such as pandas and Spark, can read and parse Gzipped files natively, which requires minimal-to-no code changes. In addition, this only adds a small extra step for reading and analyzing the file manually with an editor.

Once your training data is compressed, it's recommended to upload the Gzipped CSV file to an Azure Storage container; a good choice would be Azure Blob storage. When the data is stored in Blob storage, it can be conveniently accessed from any other services within Azure, as well as from your local machine. This means if you scale your experimentation environment from an Azure notebook to a compute cluster, your code for accessing and reading the data will stay the same.

A fantastic cross-platform GUI tool to interact with many different Azure Storage services is Azure Storage Explorer. Using this tool, it is very easy to efficiently upload small and large files to Blob storage. It also allows you to generate direct links to your files with an embedded access key. This technique is simple yet also super effective when uploading hundreds of terabytes (TBs) from your local machine to the cloud. We will discuss this in much more detail in Chapter 4, ETL, data preparation, and feature extraction.

Understanding missing values

Once the data is uploaded to the cloud—for example, using Azure Storage Explorer and Azure Blob storage for your files—we can bring up a Notebook environment and start exploring the data. The goal is to thoroughly explore your data in an analytical process to understand the distribution of each dimension of your data. This is essential for choosing any appropriate data preprocessing feature engineering and ML algorithms for your use case.

Note

Please keep in mind that not only the feature dimensions but also the target variable needs to be preprocessed and thoroughly analyzed.

Analyzing each dimension of a dataset with more than 100 feature dimensions is an extremely time-consuming task. However, instead of randomly exploring feature dimensions, you can analyze the dimensions ordered by feature importance and hence significantly reduce your time working through the data. Like many other areas of computer science, it is good to use an 80/20 principle for the initial data exploration and so only use 20% of the features to achieve 80% of the performance. This sets you up for a great start and you can always come back later to add more dimensions if needed.

The first thing to look for in a new dataset is missing values for each feature dimension. This will help you to gain a deeper understanding of the dataset and what actions could be taken to resolve those. It's not uncommon to remove missing values or impute them with zeros at the beginning of a project—however, this approach bears the risk of not properly analyzing missing values in the first place.

Note

Missing values can be disguised as valid numeric or categorical values. Typical examples are minimum or maximum values, -1, 0, or NaN. Hence, if you find the values 32,767 (= 215-1) or 65,535 (= 216-1) appearing multiple times in an integer data column, they might well be missing values disguised as the maximum signed or unsigned 16-bit integer representation. Always assume that your data contains missing values and outliers in different shapes and representations. Your task is to uncover, find, and clean them.

Any prior knowledge about the data or domain will give you a competitive advantage when working with the data. The reason for this is that you will be able to understand missing values, outliers, and extremes in relation to the data and domain—which will help you to perform better imputation, cleansing, or transformation. As the next step, you should look for these outliers in your data, specifically for the following values:

• The absolute number (or percentage) of the null values (look for Null, "Null", "", NaN, and so on)
• The absolute number (or percentage) of minimum and maximum values The absolute number (or percentage) of the most common value (MODE) The absolute number (or percentage) of value 0
• The absolute number (or percentage) of unique values

Once you have identified these values, we can use different preprocessing techniques to impute missing values and normalize or exclude dimensions with outliers. You will find many of these techniques, such as group mean imputation, in action in Chapter 4, ETL, data preparation, and feature extraction.

Visualizing data distributions

Knowing the outliers, you can finally approach exploring the value distribution of your dataset. This will help you understand which transformation and normalization techniques should be applied during data preparation. Common distribution statistics to look for in a continuous variable are the following:

• The mean or median value
• The minimum and maximum value
• The 25th, 50th (median), and 75th percentiles
• The data skew

Common techniques for visualizing these distributions are boxplots, density plots, or histograms. Figure 1.1 shows these different visualization techniques plotted per target class for a multi-class recognition dataset. Each of those methods has advantages and disadvantages—boxplots show all relevant metrics, while being a bit harder to read; density plots show very smooth shapes, while hiding some of the outliers; and histograms don't let you spot the median and percentiles easily, while giving you a good estimate for the data skew:

Figure 1.1: Common techniques for visualizing data distributions—namely, boxplots, density plots, and histograms

From the preceding visualization techniques, only histograms work well for categorical data (both nominal and ordinal)—however, you could look at the number of values per category. Another nice way to display the value distribution versus the target rate is in a binary classification task. Figure 1.2 shows the version number of Windows Defender against the malware detection rate (for non-touch devices) from the Microsoft malware detection dataset:

Figure 1.2: The version number of Windows Defender against the malware detection rate (for non-touch devices)

Many statistical ML algorithms require that the data is normally distributed and hence needs to be normalized or standardized. Knowing the data distribution helps you to choose which transformations need to be applied during data preparation. In practice, it is often the case that data needs to be transformed, scaled, or normalized.

Finding correlated dimensions

Another common task in data exploration is looking for correlations in the dataset. This will help you dismiss feature dimensions that are highly correlated and therefore might influence your ML model. In linear regression models, for example, two highly correlated independent variables will lead to large coefficients with opposite signs that ultimately cancel each other out. A much more stable regression model can be found by removing one of the correlated dimensions.

The Pearson correlation coefficient, for example, is a popular technique used to measure the linear relationship between two variables on a scale from -1 (strongly negatively correlated) to 1 (strongly positively correlated). 0 indicates no linear relation between the two variables in the Pearson correlation coefficient.

Figure 1.3 shows an example of a correlation matrix of the Boston housing price dataset, consisting of only continuous variables. The correlations range from -1 to 1 and are colored accordingly. The last row shows us the linear correlation between each feature dimension and the target variable. We can immediately tell that the median value (MEDV) of owner-occupied homes and the lower status (LSTAT) percentage of the population are negatively correlated:

Figure 1.3: An example of a correlation matrix of the Boston housing price dataset, consisting continuous variables

It is worth mentioning that many correlation coefficients can only be between numerical values. Ordinal variables can be encoded, for example, using integer encoding and can also compute a meaningful correlation coefficient. For nominal data, you need to fall back on different methods, such as Cramér's V to compute correlation. It is worth noting that the input data doesn't need to be normalized (linearly scaled) before computing the correlation coefficient.

Measuring feature and target dependencies for regression

Once we have analyzed missing values, data distribution, and correlations, we can start analyzing the relationship between the features and the target variable. This will give us a good indication of the difficulty of the prediction problem and hence, the expected baseline performance— which is essential for prioritizing feature engineering efforts and choosing an appropriate ML model. Another great benefit of measuring this dependency is ranking the feature dimensions by the impact on the target variable, which you can use as a priority list for data exploration and preprocessing.

In a regression task, the target variable is numerical or ordinal. Therefore, we can compute the correlation coefficient between the individual features and the target variable to compute the linear dependency between the feature and the target. High correlation, and so a high absolute correlation coefficient, indicates a strong linear relationship exists. This gives us a great place to start for further exploration. However, in many practical problems, it is rare to see a high (linear) correlation between the feature and target variables.

One can also visualize this dependency between the feature and the target variable using a scatter or regression plot. Figure 1.4 shows a regression plot between the feature average number of rooms per dwelling (RM) and the target variable median value of owner-occupied homes (MEDV) from the UCI Boston housing dataset. If the regression line is at 45 degrees, then we have a perfect linear correlation:

Figure 1.4: A regression plot between the feature, RM, and the target variable, MEDV

Another great approach to determining this dependency is to fit a linear or logistic regression model to the training data. The resulting model coefficients now give a good explanation of the relationship—the higher the coefficient, the larger the linear (for linear regression) or marginal (for logistic regression) dependency on the target variable. Hence, sorting by coefficients results in a list of features ordered by importance. Depending on the regression type, the input data should be normalized or standardized.

Figure 1.5 shows an example of the correlation coefficients (the first column) of a fitted Ordinary Least Squares (OLS) regression model:

Figure 1.5: The correlation coefficients of the OLS regression model

While the resulting R-squared metric (not shown) might not be good enough for a baseline model, the ordering of the coefficients can help us to prioritize further data exploration, preprocessing, and feature engineering.

Visualizing feature and label dependency for classification

In a classification task with a multi-class nominal target variable, we can't use the regression coefficients without further preprocessing the data. Another popular method that works well out of the box is fitting a simple tree-based classifier to the training data. Depending on the size of the training data, we could use a decision tree or a tree-based ensemble classifier, such as random forest or gradient-boosted trees. Doing so results in a feature importance ranking of the feature dimensions according to the chosen split criterion. In the case of splitting by entropy, the features would be sorted by information gain and hence, indicate which variables carry the most information about the target.

Figure 1.6 shows the feature importance fitted by a tree-based ensemble classifier using the entropy criterion from the UCI wine recognition dataset:

Figure 1.6: A tree-based ensemble classifier using the entropy criterion

The lines represent variations in the information gain of features between individual trees. This output is a great first step to further data analysis and exploration in order of feature importance.

Here is another popular approach to discovering the separability of your dataset. Figure 1.7 shows two graphs—one that is linearly separable (left) and one that is not separable (right)—show a dataset with three classes:

Figure 1.7: The graphs showing the separability of the dataset

You can see this when looking at the three clusters and the overlaps between these clusters. Having clearly separated clusters means that a trained classification model will perform very well on this dataset. On the other hand, when we know that the data is not linearly separable, we know that this task will require advanced feature engineering and modeling to produce good results.

The preceding figure showed two datasets in two dimensions; we actually used the first two feature dimensions for visualization. However, high-dimensional most data cannot be easily and accurately visualized in two dimensions. To achieve this, we need a projection or embedding technique to embed the feature space in two dimensions. Many linear and non- linear embedding techniques to produce two-dimensional projections of data exist; here are the most common ones:

• Principal Component Analysis (PCA)
• Linear Discriminant Analysis (LDA)
• t-Distributed Stochastic Neighbor Embedding (t-SNE)
• Uniform Manifold Approximation and Projection (UMAP)

Figure 1.8 shows, the LDA (left) and t-SNE (right) embeddings for the 13-dimensional UCI wine recognition dataset (https:/). In the LDA embedding, we can see that all the classes should be linearly separable. That's a lot we have learned from using two lines of code to plot the embedding before we even start with model selection or training:

Figure 1.8: LDA (left) and t-SNE (right) embeddings for the 13-dimensional UCI wine recognition dataset

Both the LDA and t-SNE embeddings are extremely helpful for judging the separability of the individual classes and hence the difficulty of your classification task. It's always good to assess how well a particular model will perform on your data before you start selecting and training a specific algorithm. You will learn more about these techniques in Chapter 3, Data experimentation and visualization using Azure.

After the data experimentation phase, you should have gathered enough knowledge to start preprocessing the data. This process is also often referred to as feature engineering. When coming from multiple sources, such as applications, databases, or warehouses, as well as external sources, your data cannot be analyzed or interpreted immediately.

It is, therefore, of imminent importance to preprocess data before you choose a model to interpret your problem. In addition to this, there are different steps involved in data preparation, which depend on the data that is available to you, such as the problem you want to solve, and with that, the ML algorithms that could be used for it.

You might ask yourself why data preparation is so important. The answer is that the preparation of your data might lead to improvements in model accuracy when done properly. This could be due to the relationships within your data that have been simplified due to the preparation. By experimenting with data preparation, you would also be able to boost the model's accuracy later on. Usually, data scientists spend a significant amount of their time on data preparation, feature engineering, and understanding their data. In addition to this, data preparation is important for generating insights.

Data preparation means collecting data, cleaning it up, transforming the data, and consolidating it. By doing this, you can enrich your data, transform it, and as mentioned previously, improve the accuracy of your model. In fact, in many cases, an ML model's performance can be improved significantly through better feature engineering.

The challenges that come along with data preparation are, for example, the different file formats, the data types, inconsistency in data, limited or too much access to data, and sometimes even insufficient infrastructure around data integration. Another difficult problem is converting text, such as nominal or ordinal categories or free text, into a numeric value.

The way people currently view data preparation and perform this step of the process is through the extract, transform, and load tools. It is of utmost importance that data within organizations is aligned and transformed using various data standards. Effective integration of various data sources should be done by aligning the data, transforming it, and then promoting the development and adoption of data standards. All this effectively helps in managing the volume, variety, veracity, and velocity of the data.

In the following subparagraphs, some of the key techniques in data preparation, such as labeling, storing, encoding, and normalizing data, as well as feature extraction, will be shown in more depth.

Labeling the training data

Let's start with a bummer; the first step in the data preparation journey is labeling, also called annotation. It is a bummer because it is the least exciting part of an ML project, yet one of the most important tasks in the whole process. Garbage in, garbage out—it's that simple. The ultimate goal is to feed high-quality training data into the ML algorithms, which is why labeling training data is so important.

While proper labels greatly help to improve prediction performance, the labeling process will also help you to study the dataset in greater detail. However, let me clarify that labeling data requires deep insight and understanding of the context of the dataset and the prediction process. If we were aiming to predict breast cancer using CT scans, we would also need to understand how breast cancer can be detected in CT images in order to label the data.

Mislabeling the training data has a couple of consequences, such as label noise, which you want to avoid as it will the performance of every downstream process in the ML pipeline, such as feature selection, feature ranking and ultimately model performance. Learning relies crucially on the accuracy of labels in the training dataset. However, we should always take label noise into account when aiming for a specific target metric because it's highly unlikely that all the provided labels will be absolutely precise and accurate.

In some cases, your labeling methodology is dependent on the chosen ML approach for a prediction problem. A good example is the difference between object detection and segmentation, both of which require completely differently labeled data. As labeling for segmentation is much more time-consuming than labeling for object detection or even classification, it is also an important trade-off to make before starting an ML project.

There are some techniques you can use to speed up the labeling process, which are hopefully provided by your labeling system:

• Supervised learning: Through supervised learning, an ML model could recommend the correct labels for your data at labeling time. You can then decide whether you use the predicted label or choose a different or modified label. This works very well with object detection and image data.
• Active learning: Another technique to accelerate the labeling process is to allow a semi-supervised learning process to learn and predict based on a few manually labeled samples. Using those labeled samples, the model automatically proposes new labels that can either be accepted or changed and modified. Each label will fine-tune the model to predict better labels.
• Unsupervised learning: Through clustering similar data samples together, the labeling environment can prioritize which data points should be labeled next. Using these insights, the labeling environment can always try to propose loads of greatly varying samples in the training data for manual labeling.

Labeling is a necessary, long, and cost-intensive step in an ML process. There are techniques to facilitate labeling; however, they always require the domain knowledge to be carried out properly. If there is any chance that you can collect labeled data through your application directly, you are very lucky and should start collecting this data. A good example is collecting training data for a click-through rate of search results based on the actual results and clicks of real users.

Normalization and transformation in machine learning

Normalization is a common data preprocessing technique where the data is scaled to a different value range through a (linear) transformation. For many statistical ML models, the training data needs to follow a certain distribution and so it needs to first be normalized along all its dimensions. The following are some of the most commonly used methods to normalize data:

• Scaling to unit length, or standard scaling
• Minimum/maximum scaling
• Mean normalization
• Quantile normalization

In addition to these, you can also monitor normalization by ensuring the values fall between 0 and 1 in the case of probability density functions, which are used in fields such as chemistry. For exponential distribution and Poisson distribution, you could use the coefficient of variation because it deals well with positive distributions.

Note

In ML algorithms such as Support Vector Machines (SVM), logistic regression, and neural networks, a very common normalization technique is standardization, which standardizes features by giving them a 0 mean and unit variance. This is often referred to as a standard scaler.

Besides linear transformations, it's also quite common to apply non-linear transformations to your data for the same reason as for normalization, which is to pass the assumption for a specifically required distribution. If your data is skewed, you can use power or log transformations to normalize the distributions. This is very important, for example, for linear models where the normality assumption is a required conditional to the predictor's vector. For highly skewed data, you can also apply these transformations multiple times. For data ranges containing 0, it's also common to apply log plus 1 transformations to avoid numerical instability.

Encoding categorical variables

With a real-world dataset, you will quickly reach the limits of normalization and transformation as the data for these transformations needs to be continuous. The same is true for many statistical ML algorithms, such as linear regression, SVM, or neural networks; the input data needs to be numerical. Hence, in order to work with categorical data, we need to look at different numerical encoding techniques.

We differentiate between three different types of categorical data: ordinal, nominal, and textual (for example, free text). We make this distinction between nominal and textual data as textual data is often used to extract semantic meaning whereas nominal categorical data is often just encoded.

There are various types of numerical encoding techniques you could use. They are listed here:

• Label encoding: This is where each category is mapped to a number or label. The labels for the categories are not related to each other; therefore, categories that are related will lose this information after encoding.
• One-hot encoding: Another popular approach is dummy coding, also called one- hot encoding. Each category is replaced with an orthogonal feature vector, where the dimension of the feature vector is dependent on the number of distinct values. This approach is not efficient with high cardinality features.
• Bin encoding: Even though bin encoding is quite similar to one-hot encoding, it differs from storing categories as binary bitstrings. The goal of bin encoding is to hash the cardinalities into binary values and each binary digit gets one column. This will result in some information loss; however, you can deal with fewer dimensions. It also creates fewer columns and so the speed of learning is higher and more memory efficient.
• Count encoding: In count encoding, we replace the categorical values with the relative or absolute count of the value over the whole training set. This is a common technique for encoding large amounts of unique labels.
• Target encoding: In this encoding methodology, we replace the categorical value with the mean value of the target variable of this category. This is also effective with high-cardinality features.
• Hashing encoding: This is used when there are a lot of large-scale categorical features. The hash function maps a lot of values into a small, finite set of values. Different values could create the same hash, which is called a collision.

We will take a closer look at some of these encoding techniques in Chapter 6, Advanced feature extraction with NLP.

A feature engineering example using time-series data

Feature engineering is strongly dependent on the domain of your dataset. When dealing with demographics or geographics, you can model personas and demographic metrics or join geographic attributes, such as proximity to a large city, or to the border, GDP, and others. Let's look at an example of time-series data, which is extremely common in real- world examples.

Many real-world datasets have a temporal dependency and so they store the date and time in at least one of the dimensions of the training data. This date-time field can be treated either as an encoded or an ordinal categorical variable, depending on the distribution of the date-time variable.

Depending on the distribution and patterns in the date-time data, you want to transform the date-time field into different values that encode a specific property of the current date or time. The following are a few features that can be extracted from date-time variables:

• The absolute time
• The hour of the day
• The day of the week
• The day of the month
• The day of the year
• The month of the year

If you see a periodic relationship between a dimension over time, you can also encode the cycle features of the time dimension. This can be achieved by computing the absolute hour of the day to compute the sine and cosine components of the normalized hour of the day. Using this technique, the resulting values will contain a cyclic dependency on the encoded date-time dimension.

Another great way of improving your model's performance is to include additional data in your training data. This works really well on the date-time dimension as you can, for example, join public holidays, public events, or other types of events by date. This lets you create features such as the following:

• The number of days until or since the next or last campaign
• The number of days until or since the next or last holiday
• Mark a date as a public holiday
• Mark a date as a major sporting event

As you can see, there are many ways to transform and encode date-time variables. It is encouraged to dive into the raw data and look for visual patterns in the data that should be interpreted by the ML model. Whenever you deal with a date-time dimension, there is room for creative feature engineering.

Using NLP to extract complex features from text

Using NLP to extract features from text is very useful as an input for ML models. NLP is used to apply ML algorithms to text and speech and is often used to preprocess raw text data and categorical embeddings. We often distinguish between occurrence-based embeddings, such as bag-of-words, and semantic embeddings, such as Word2Vec. NLP is extremely useful for any time that you are dealing with textual data.

Similar to categorical embeddings, NLP techniques transform text into numerical values. These values are often high-dimensional and need to be simplified—commonly done through Singular Value Decomposition (SVD)—or aggregated. Some popular techniques that are used in NLP to extract features from text are as follows:

• Lemmatization, stemming, and stop-word removal n-grams
• tf-idf
• Embeddings, such as Word2vec
• Fully trained models, such as sequence-to-sequence models

If we aim to convert text to numerical values, we can practically implement encodings using bag-of-words predictors, Word2Vec embeddings, or sequence-to-sequence models. The same idea can be extended to documents where instead of learning feature representations for words, you learn them for documents.

We will take a closer look at feature extraction through NLP and all the previously mentioned techniques in Chapter 6, Advanced feature extraction with NLP.

Similar to data experimentation and preprocessing, training ML model is an analytical, step-by-step process. Each step involves a thought process that evaluates the pros and cons of each algorithm according to the results of the experimentation phase. Like in every other scientific process, it is recommended that you come up with a hypothesis first and verify whether this hypothesis is true afterward.

Let's look at the steps that define the process of training an ML model:

• Define your ML task: First, we need to define the ML task we are facing, which most of the time is defined by the business decision behind your use case. Depending on the amount of labeled data, you can choose between non-supervised, semi-supervised, and supervised learning, as well as many other subcategories.
• Pick a suitable model to perform this task: Pick a suitable model for the chosen ML task. This includes logistic regression, a gradient-boosted tree ensemble, and a deep neural network, just to name a few popular ML model choices. The choice is mainly dependent on the training (or production) infrastructure (such as Python, R, Julia, C, and so on) and on the shape of the data. It is recommended that you favor simple traditional ensemble techniques, such as gradient-boosted tree ensembles, when training data is limited. These models perform well on a broad set of input values (ordinal, nominal, and numeric) as well as training efficiently and they are understandable. When strong generalization and expressiveness is required, and given a reasonable amount of training data, you should go with deep learning models. When limited data is available to build a highly complicated model (for example, for object detection), it is recommended you use pre-trained models as feature extractors and only train the classifiers on top. However, whenever possible, it is recommended you build on top of pre- trained models when deep learning techniques are used.
• Pick a train-validation split: Splitting your data into a training and validation set gives you additional insights in the performance of your training and optimization process. This includes a group shuffle split, temporal split, stratified split, and so on.
• Pick or implement an error metric: During the data experimentation phase, you should have already come up with a strategy on how to test your model's performance. Hence, you should have picked a validation split and error metric already. If you have not done so, I recommend you evaluate what you want to measure and optimize (such as absolute errors, relative errors, percentages, true positive rates, true negative rates, and so on). This includes F1-score, MSE, ROC, weighted Cohen's kappa, and so on.
• Train a simple model using cross-validation: Finally, when all the preceding choices are made, you can go ahead and train your ML model. Optimally, this is done as cross-validation on a training and validation set, without leaking training data into validation. After training a baseline model, it's time to interpret the error metric of the validation runs. Does it make sense? Is it as high or low as expected? Is it (hopefully) better than random and better than always predicting the most popular target? What's the influence of the random seed on this result?

Once the answers to these questions are gathered, you can go back to the fun part: improving the model performance—by data analysis, feature engineering and data preprocessing.

Choosing an error metric

After looking at the relationship between the feature and target dimensions, as well as the separability of the data, you should continue to evaluate which error metric will be used to train the ML model later on. There are many metric choices that measure absolute, squared, and percentage errors for regression, as well as the accuracy, true positive rate, true negative rate for classification, and weighted distance metrics for ordinal categories—just to name a few.

Note

Defining an appropriate error metric for an optimization problem is not straightforward as it depends on multiple circumstances. In a classification problem, for example, we are confronted with the precision- recall dilemma, you can either optimize for maximal precision (and hence minimize false positives) or for maximal recall (and hence maximize true positives). Either decision will result in a different model with opposite strengths and weaknesses.

Many machine learning practitioners don't value the importance of a proper error metric highly enough but instead go with the default metric for their use case (for example, accuracy, mean squared error, and so on). If you find yourself in this trap, remember the importance of the right error metric. The choice of error metric is absolutely critical and might even result in your ML use case succeeding or failing.

Before diving into model training and optimization—which includes tasks such as model selection and parameter tuning—it is useful to understand the baseline performance and the model's robustness to noise. The first can be achieved by computing the error metric using only the target variable with the highest occurrence as a prediction—this will be your baseline performance. The second can be done by modifying the random seed of your ML model (for example, the tree-based model used for feature importance) and observing the changes to the error metric. This will show you what decimal place you can trust the error metric to.

The training and testing split

Once you have selected an ML approach and an error metric, you need to think about splitting your dataset for training. Optimally, the data should be split into three disjointed sets: a training, a validation, and a testing set. We use multiple sets to ensure that the model generalizes well on unseen data and that the reported error metric can be trusted. Hence, you can see that dividing the data into representative sets is a task that should be performed as an analytical process.

You need to avoid training data leaking into the validation or testing set, hence overfitting the training data and skewing the validation and testing results, at all costs. To ensure this, you need to always create disjointed datasets and use the validation set for cross-validation and parameter tuning and the testing set only for reporting your final model performance.

There are many different techniques available, such as stratified splitting (sampling based on class distributions), temporal splitting, and group-based splitting. We will take a look at these in Chapter 7, Building ML models using Azure Machine Learning.

Achieving great performance using tree-based ensemble models

Many amazing traditional ML approaches exist, such as naive Bayes SVM, and linear regression. However, there is one technique that, due to its flexibility, gets you started quickly while delivering great prediction performance without a ton of tuning and data preparation. While most decision tree-based ensemble estimators fit this description, we want to look at one in particular: gradient-boosted trees.

In the previous section, we mentioned that building a baseline model for estimating the baseline performance is a good start to every ML project. Indeed, we will see in many chapters that building a baseline model helps you focus on all the important aspects of your project, such as the data, infrastructure, and automation.

Decision trees are extremely versatile. They can be used with numerical and categorical data as input and can predict both continuous and categorical values. Tree-based ensemble models combine many weak learners into a single predictor based on decision trees. This greatly reduces the problem of the overfitting and instability of single decision trees. When boosting, we use an iterative approach to optimize the model performance by weighting misclassified training samples after each training iteration. The output after a few iterations using the default parameter usually delivers great baseline results for many different applications.

In Chapter 7, Building ML models using Azure Machine Learning, we have dedicated a complete section to training a gradient-boosted tree-ensemble classifier using LightGBM, a popular tree-ensemble library from Microsoft.

Modeling large and complex data using deep learning techniques

To capture the meaning of large amounts of complex training data, we need large parametric models. However, training parametric models with many hundreds of millions of parameters is no easy task, due to exploding and vanishing gradient, loss propagation through such a complex model, numerical instability, and normalization. In recent years, a branch of such high parametric models achieved extremely good results through many complex tasks—namely, deep learning.

Deep neural networks work extremely well on complex prediction tasks with large amounts of complex input data. Most models combine both the feature extractor and classification/regression parts and are trained in a fully end-to-end approach. Fully connected neural networks—also called Artificial Neural Networks (ANNs)—work very similar to logistic regression models, with a different loss function and the stacking of multiple layers. Convolutional Neural Networks (CNNs) use local constraint connections with shared weights to remove the number of required parameters while taking advantage of data locality. They work extremely well with image data where the convolution and pooling layers correspond to classical computer vision filters and operators.

Recurrent Neural Networks (RNNs) and Long Short-Term Memory (LSTM) layers help to model time dependency by keeping a state over time. Most model architectures can take advantage of parallel computing through General Programming Graphical Processing Units (GPGPU), or even virtualized or dedicated deep learning hardware.

Chapter 8, Training deep neural networks on Azure, and Chapter 10, Distributed machine learning on Azure, are dedicated to training large, complex deep learning models on a single machine and distributed GPU clusters.

If we have trained a simple ensemble model that performs reasonably better than the baseline model and achieves acceptable performance according to the expected performance estimated during data preparation, we can progress with optimization. This is a point we really want to emphasize. It's strongly discouraged to begin model optimization and stacking when a simple ensemble technique fails to deliver useful results. If this is the case, it would be much better to take a step back and dive deeper into data analysis and feature engineering.

Common ML optimization techniques, such as hyperparameter optimization, model stacking, and even automated machine learning, help you get the last 10% of performance boost out of your model while the remaining 90% is achieved by a single ensemble model. If you decide to use any of those optimization techniques, it is advised to perform them in parallel and fully automated on a distributed cluster.

After seeing too many ML practitioners manually parametrizing, tuning, and stacking models together, we want to raise the important message that training and optimizing ML models is boring. It should rarely be done manually as it is much faster to perform it automatically as an end-to-end optimization process. Most of your time and effort should go into experimentation, data preparation, and feature engineering—that is, everything that cannot be easily automated and optimized using raw compute power. Prior knowledge about the data and an understanding of the ML use case and the business insights are the best places to dig deeper into when investing time in improving the model performance.

Hyperparameter optimization

Once you have achieved reasonable performance using a simple single model with default parameterization, you can move on to optimizing the hyperparameters of the model. Due to the combination and complexity of multiple parameters, it doesn't make a lot of sense to waste time on tuning the parameters by hand. Instead, this tuning should always be performed in an optimal automated way, which will always lead to a better cross- validation performance.

First, you need to define the parameter search space and sampling distribution for each trainable hyperparameter. This definition is either a continuous or categorical region with different sampling distributions; for example, uniform, logarithmic, or normal distributed sampling. This can usually be done by generating a parameter configuration using a hyperparameter optimization library.

The next thing you need to decide is the sampling technique of parameters in the search space. The three most common sampling and optimization techniques are the following:

• Grid sampling
• Random sampling
• Bayesian optimization

While the first two algorithms sample either in a grid or at random in the search space, the third algorithm performs a more intelligent search through Bayesian optimization. In practice, random and Bayesian sampling are used most often.

Note

To avoid any unnecessary compute time spent on wrong parameter configurations, it is recommended to define early stopping criteria when using hyperparameter optimization.

Training many combinations of different parameter sets is a computationally complex task. Hence, it is strongly recommended to parallelize this task on multiple machines and track all parameter combinations and model cross-validation performance at a central location. This is a particularly beneficial task for a highly scalable cloud computing environment where these tasks are performed automatically. In Azure Machine Learning, you can use the HyperDrive functionality to do exactly this. We will look at this in great detail in Chapter 9, Hyperparameter tuning and Automated Machine Learning.

Model stacking

Model stacking is a very common technique used to improve prediction performance by putting a combination of multiple models into a single stacked model. Hence, the output of each model is fed into a meta-model, which itself is trained through cross-validation and hyperparameter tuning. By combining significantly different models into a single stacked model, you can always outperform a single model.

Figure 1.9 shows a stacked model consisting of different supervised models in level 1 that feed their output into another meta-model. This is a common architecture that further boosts prediction performance once all the feature engineering and model tuning options are fully exploited:

Figure 1.9: Model stacking

Model stacking adds a lot of complexity to your ML process while almost always leading to better performance. This technique will get out the last 1% performance gain of your algorithm. To efficiently stack models into a meta ensemble, it is recommended that you do it fully automated; for example, through techniques such as Azure Automated Machine Learning. One thing to be aware of, however, is that you can easily overfit the training data or create stacked models that are magnitudes larger in size than single models.

Azure Automated Machine Learning

As we have shown, constructing ML models is a complex step-by-step process that requires a lot of different skills, such as domain knowledge (prior knowledge that allows you to get insight into data), mathematical expertise, and computer science skills. During this process, there is still human error and bias involved, which might not only affect the model's performance and accuracy, but also the insights that you want to gain out of it.

Azure Automated Machine Learning could be used to combine and automated all of this by reducing the time to value. For several industries, automated machine learning can leverage ML and AI technology by automating manual modeling tasks, such that the data scientists can focus on more complex issues. Particularly when using repetitive ML tasks, such as data preprocessing, feature engineering, model selection, parameter tuning and model stacking, it could be useful to use Azure Automated Machine Learning.

We will go into much more detail and see real-world examples in Chapter 9, Hyperparameter tuning and Automated Machine Learning.

Once you have trained and optimized an ML model, it is ready for deployment. Many data science teams, in practice, stop here and move the model to production as a Docker image, often embedded in a REST API using Flask or similar frameworks. However, as you can imagine, this is not always the best solution depending on your use case requirements. An ML or data engineer's responsibility doesn't stop here.

The deployment and operation of an ML pipeline can be best seen when testing the model on live data in production. A test is done to collect insights and data to continuously improve the model. Hence, collecting model performance over time is an essential step to guaranteeing and improving the performance of the model.

In general, we differentiate two architectures for ML-scoring pipelines, which we will briefly discuss in this section:

• Batch scoring using pipelines
• Real-time scoring using a container-based web service

These architectures are discussed in increasing order of operational complexity, with offline scoring being the least complex and asynchronous scoring being the more complex system. The complexity arises from the number of components involved in operating such a pipeline at scale.

Finally, we will investigate an efficient way of collecting runtimes, latency, and other operational metrics, as well as model performance. It's also good practice to log all scored requests in order to analyze and improve an ML model over time.

Both architectures, as well as the monitoring solutions, will be discussed in more detail and implemented in Chapter 12, Deploying and operating machine learning models.

Batch scoring using pipelines

With offline scoring, or batch scoring, we are talking about an offline process where you evaluate an ML model against a batch of data. The result of this scoring technique is usually not time-critical and the data to be scored is usually larger than the model. This process is usually used when an ML model is scored within another batch process, such as a daily, hourly, or weekly task.

Here is what we expect as input and output data:

• Input: A location to find the input data
• Output: A response with all the scores

While the input and output format is quite intuitive, we still want to give a list of examples of when such an architecture is used. The reason for this is that you can decide the proper architecture for your use case when dealing with a similar ML task. Here are some practical examples:

• A recommendation engine of a streaming service generates new recommendations every week for the upcoming week.
• A classification algorithm of a mobile telecommunication operator computes a churn score for every customer once a month.

If the model was trained on a distributed system, it is very common to perform batch scoring on the same system that was used for training as the scoring task is identical to computing the score for the test set.

Real-time scoring using a container-based web service

The term online synchronous scoring, or real-time scoring, refers to a technique where we score an ML model and instantly need the resulting score. This is very common in stream processing, where single events are scored in real time. It's obvious that this task is highly time-critical and the execution is blocked until the resulting score is computed.

Here is what we expect as input and output data:

• Input: One or multiple observations
• Output: A response with a single score

The input and output configuration is also quite intuitive. Here are some practical examples of typical real-time scoring use cases:

• An object detection algorithm in a self-driving vehicle detects obstacles so it can control and steer the vehicle safely around the objects.
• An object detection algorithm detects faces in a camera image and focuses the camera.
• A classification algorithm decides whether the current product on the conveyor meets the quality standards.

Models for online synchronous scoring services are often deployed to the cloud as parallelized services in a distributed cluster with a load balancer in front of them. This way, the scoring cluster can be easily scaled up or down when higher throughput is required. If the latency requirements are restricted, these models are also deployed to edge devices, such as mobile phones or industrial computers, in order to avoid a round trip of the data to the nearest cloud region.

Tracking model performance, telemetry, and data skew

Tracking the proper metrics of a deployed ML model is essential. While popular metrics about include consumed CPU time, RAM, GPU memory, as well as latency, we also want to focus on the model's scoring performance. As we have already seen, most real- world data has a dependency on time and so many habits change over time. Operating an ML model in production means also continuously guaranteeing the quality and performance of the model.

In order to track the model's performance, you can use a standard application monitoring tool, such as Azure Application Insights or any other highly scalable key-value database. This is important to understand how your users are using your model and what your model is predicting in production.

Another important insight is tracking the data used for scoring the model. If we keep this data, we can compare it to the training data used for the deployed model and compute the data skew between the training data and the scoring data. By defining a threshold for maximum model skew, we can use this as a trigger to re-train the model once the skew is too big. We will see this in action in Chapter 12, Deploying and operating machine learning models.

In this chapter, we saw an overview of all the steps involved in making a custom ML pipeline. You might have seen familiar concepts for data preprocessing or analytics and learned an important lesson. Data experimentation is a step-by-step approach rather than an experimental process. Look for missing values, data distribution, and relationships between features and targets. This analysis will greatly help you to understand which preprocessing steps to perform and what model performance to expect.

You now know that data preprocessing, or feature engineering, is the most important part of the whole ML process. The more prior knowledge you have about the data, the better you can encode categorical and temporal variables or transform text to numerical space using NLP techniques. You learned that choosing the proper ML task, model, error metric, and train-test split is mostly defined by business decisions (for example, object detection against segmentation) or a performance trade-off (for example, stacking).

Using your newly acquired skills, you should now be able to draft an end-to-end ML process and understand each step from experimentation to deployment. In the next chapter, we will look at an overview of which specific Azure services can be used to efficiently train ML models in the cloud.