Deep Learning for Predictive Maintenance

Predictive maintenance is one of the most sought after machine learning solutions for IoT. It is also one of the most elusive machine learning solutions for IoT. Other areas of machine learning can easily be solved, implementing Computer Vision, for example, can be done in hours using tools such as OpenCV or Keras. To be successful with predictive maintenance you first need the right sensors. The Data collection design recipe in Chapter 2, Handling Data, can be used to help determine proper sensor placement. The Exploratory factor analysts recipe in Chapter 2, Handling Data can help determine the cadence with which the data needs to be stored. One of the biggest hurdles to implementing predictive maintenance is that there needs to be a sufficient amount of device failures. For rugged industrial devices, this can take...

Enhancing data using feature engineering

One of the best use of time in improving models is feature engineering. The ecosystem of IoT has many tools that can make it easier. Devices can be geographically connected or hierarchically connected with digital twins, graph frames, and GraphX. This can add features such as showing the degree of contentedness to other failing devices. Windowing can show how the current reading differs over a period of time. Streaming tools such as Kafka can combine different data streams allowing you to combine data from other sources. Machines that are outdoor may be negatively affected by high temperatures or moisture as opposed to machines that are in a climate-controlled building.

In this recipe, we are going to look at enhancing our data by looking at time-series data such as deltas, seasonality, and windowing. One of the most valuable uses of time for a data scientist is feature engineering. Being able to slice the data into meaningful features can...

Getting ready

In the Predictive maintenance with XGBoost recipe in the previous chapter, we used XGBoost to predict whether or not a machine needed maintenance. We have imported the NASA Turbofan engine degradation simulation dataset which can be found at https://data.nasa.gov/dataset/Turbofan-engine-degradation-simulation-data-set/vrks-gjie. In the rest of this chapter, we will continue to use that dataset. To get ready you will need the dataset.

Then if you have not already imported numpy, pandas, matplotlib, and seaborn into Databricks do so now.

How to do it...

The following steps need to be observed to follow this recipe:

1. Firstly, import the required libraries. We will be using pyspark.sql, numpy, and pandas for data manipulation and matplotlib and seaborn for visualization:

from pyspark.sql import functions as F
from pyspark.sql.window import Window

import pandas as pd
import numpy as np
np.random.seed(1385)

import matplotlib as mpl
import matplotlib.pyplot as plt
import seaborn as sns

2. Next, we're going to import the data and apply a schema to it so that the data types can be correctly used. To do this we import the data file through the wizard and then apply our schema to it:

file_location = "/FileStore/tables/train_FD001.txt"
file_type = "csv"

from pyspark.sql.types import *
schema = StructType([
  StructField("engine_id", IntegerType()),
  StructField("cycle", IntegerType()),
  StructField("setting1", DoubleType()),
  StructField("setting2",...

How it works...

In this recipe, we have performed feature engineering so that we could make our data more usable by our ML algorithms. We removed the columns with no variation, high correlation, and we denoised the dataset. In step 8 we removed the columns with no variation. The method describes the data in several ways. Reviewing the chart showed that many variables do not change at all. Next, we used a heat map to find sensors that had the same data. Finally, we used a rolling average to smooth the data from our original dataset into a new one.

There's more...

So far we have just looked at training data. But we will also need to look at testing the data. There is a test dataset and a RUL dataset. These datasets will help us test our models. To import them you would run 2 additional import steps:

1. Importing test data: Relying on the schema from the training set the test set is imported and put in a table called engine_test:

# File location and type
file_location = "/FileStore/tables/test_FD001.txt"
df = spark.read.option("delimiter"," ").csv(file_location, 
                                            schema=schema, 
                                            header=False)
df.write.mode("overwrite").saveAsTable("engine_test")

2. Importing the RUL Dataset: The next step is to import the remaining useful life dataset and save that to a table as well:

file_location = "/FileStore/tables/RUL_FD001.txt"
RULschema = StructType([StructField("RUL", IntegerType...

Using keras for fall detection

One strategy for predictive maintenance is to look at patterns of device failures for a given record. In this recipe, we will classify the data that exhibits a pattern that happens before the device fails.

We will be using keras, which is a fairly powerful machine learning library. Keras strips away some of the complexity of TensorFlow and PyTorch. Keras is a great framework for beginners in machine learning as it is easy to get started on and the concepts learned in Keras transfer to more expressive machine learning libraries such as TensorFlow and PyTorch.

Getting ready

This recipe expands on the predictive maintenance dataset we feature engineered in the previous recipe. If you have not already done so you will need to import the keras, tensorflow, sklearn, pandas, and numpy libraries into your Databricks cluster.

How to do it...

Please observe the following steps:

1. Firstly, import the required libraries. We import pandas, pyspark.sql, and numpy for data manipulation, keras for machine learning, and sklearn for evaluating the model. After evaluating the model we use io, pickle, and mlflow to save the model and results so that it can be evaluated against other models:

from pyspark.sql.functions import *
from pyspark.sql.window import Window

import pandas as pd
import numpy as np
import io
import keras
from sklearn.model_selection import train_test_split
from sklearn.metrics import precision_score
from sklearn.preprocessing import MinMaxScaler

from keras.models import Sequential
from keras.layers import Dense, Activation, LeakyReLU, Dropout

import pickle
import mlflow

2. Next, we import training and testing data. Out training data will be used to train our models and our testing data will be used to evaluate the models:

X_train = spark.sql("select rolling_average_s2, rolling_average_s3, 
          ...

How it works...

There are typically three tasks that neural networks does:

• Import data
• Recognize the patterns of the data by training 
• Predicting the outcomes of new data

Neural networks take in data, trains themselves to recognize the patterns of the data, and then are used to predict the outcomes of new data. This recipe uses the cleaned and feature engineered dataset saved in the previous recipe. The X_train dataset is pulled in from the spark data table into a Panda DataFrame. The training DataFrames, X_train, and y_train are used for training. X_test gives us a list of devices that have failed and y_test gives us the real-time failure of those machines. Those datasets are used to train models and test the results.

First, we have the input layer. The data is fed to each of our 32 input neurons. The neurons are connected through channels. The channel is assigned a numerical value known as weight. The inputs are multiplied by the corresponding...

There's more...

In this recipe, we used LeakyReLU as our activation function, rmsprop as our optimizer, and binary_crossentropy as our loss function. We then saved the results to mlflow. We can tune parameters in this experiment by trying different combinations such as the number of neurons or the number of layers. We could also change the activation function to use ReLU or TanH. We could also use Adam as our optimizer. Saving those results to mlflow allows us to improve our model.

Implementing LSTM to predict device failure

Recurrent neural networks predict sequences of data. In the previous recipe, we looked at 1 point in time and determined to determine if maintenance was needed. As we saw in the first recipe when we did the data analysis the turbofan run to failure dataset is highly variable. The data reading at any point in time might indicate a need for maintenance while the next indicates that there is no need for maintenance. When determining whether or not to send a technician out having an oscillating signal can be problematic. Long Short Term Memory (LSTM) is often used with time-series data such as the turbofan run to failure dataset.

With the LSTM, we look at a series of data, similar to windowing. LSTM uses an ordered sequence to help determine, in our case, if a turbofan engine is about to fail based on the previous sequence of data. 

Getting ready

For this recipe we will use the NASA Turbofan run to failure dataset. For this recipe we will be using a Databricks notebook. This recipe requires a few libraries to be installed. For data processing we need to install numpy and pandas, keras for creating a LSTM model, and sklearn and mlflow for evaluating and saving the results of our model.

Even though in previous recipes we added windowing and preprocessed the data, in this recipe we will use the raw data. LSTMs window the data and also have a good deal of extraction and transformation that is unique to this type of ML Algorithm.

How to do it...

We will execute the following steps for this recipe:

1. First, we will import all of the libraries which we will need later. We will import pandas and numpy for data processing, keras for the ML models, sklearn for evaluations, and pickel and mlflow for storing the results:

import pandas as pd
import numpy as np

import keras
from keras.models import Sequential
from keras.layers import Dense, Dropout, LSTM, Activation

from sklearn import preprocessing
from sklearn.metrics import confusion_matrix, recall_score, precision_score
import pickle
import mlflow

2. Next we will set the variables. We will set 2 cycles periods. In addition we use a sequence length variable. The sequence length allows the LSTM to look back over 5 cycles. This is similar to windowing that was discussed in Chapter 1, Setting Up the IoT and AI Environment. We are also going to get a list of data columns:

week1 = 7
week2 = 14
sequence_length = 100
sensor_cols = ['s' + str(i) for i in range(1,22)]
sequence_cols...

How it works...

A LSTM is a special type of recurrent neural network (RNN). A RNN is a neural network architecture that deal with sequenced data by keeping the sequence in memory. Conversely, a typical feed-forward neural does not keep the information about the sequences and do not allow for flexible inputs and outputs. A recursive neural network uses recursion to call from one output back to its input thereby generating a sequence. It passes a copy of the state of the network at any given time. In our case we are using two layers for our RNN. This additional layer helps with accuracy.

LSTMs solve a problem of vanilla RNNs by dropping out data to solve the vanishing gradient problem. The vanishing gradient problem is when the neural network stops training early but is inaccurate. By using dropout data we can help solve that problem. The LSTM does this by using gating functions.

Deploying models to web services

Deployment of the model can be different depending on the capabilities of the device. Some devices with extra compute can handle having the machine learning models run directly on the device. While others require assistance. In this chapter, we are going to deploy the model to a simple web service. With modern cloud web apps or Kubernetes these web services can scale to meet the needs of the fleet of devices. In the next chapter, we will show how to run the model on the device.

Getting ready

So far in this book, we have looked at three different machine learning algorithms to solve the predictive maintenance problem with the NASA Turbofan run to failure dataset. We recorded the results to MLflow. We can see that our XGBoost notebook outperformed the more complex neural networks. The following screenshot shows the MLflow result set showing the parameters and their associated scores.

From here we can download our model and put it in our web service. To do this we are going to use a Python Flask web service and Docker to make the service portable. Before we start, pip install the python Flask package. Also install Docker onto your local computer. Docker is a tool that allows you to build out complex deployments.

How to do it...

In this project, you will need to create three files for testing the predictor web service and one file to scale it to production. First create app.py for our web server, requirements.txt for the dependencies, and the XGBoost model you downloaded from mlflow. These files will allow you to test the web service. Next, to put it into production you will need to dockerize the application. Dockerizing the file allow you to deploy it to services such as cloud-based web application or Kubernetes services. These services scale easily making onboarding new IoT devices seamless. Then execute the following steps:

1. The app.py file is the Flask application. Import Flask for the web service, os and pickle for reading the model into memory, pandas for data manipulation, and xgboost to run our model:

from flask import Flask, request, jsonify
import os
import pickle
import pandas as pd
import xgboost as xgb

2. Next is to initialize our variables. By loading the Flask application...

How it works...

Flask is a lightweight web server. We pull in the model that we saved to disk using pickle to rehydrate the model. We then create an http endpoint to call into.

There's more...

Modern cloud-based web applications such as Azure Web Apps can automatically pull new Docker images into production. There is also a great deal of DevOps tools that can pull images and run them through various tests before deploying them with Docker container instances or docker orchestration tools such as Kubernetes. But for them to do this one must first put them into a container registry such as Azure Container Registry or Docker Hub. To do this we will need to do a few steps. First, we will build our container. Next, we can run our container to ensure that it works. Then we log into our container registry service and push the container to it. The detailed steps are as follows:

1. First, we build the container. To do it we navigate to the folder with the docker file and run docker build. We are going to tag it with the -t command to ch4.  We then specify that the docker file is in the local folder with the period .: 

docker build -t ch4 .

2. Now that we...