Working with Data and Analytics

Managing data and performing analytics on it is a crucial aspect of any Internet of Things (IoT) deployment. It allows you to gain insights based on the large amounts of data generated by IoT devices and make appropriate decisions based on data to improve operations, increase efficiency, and reduce costs. With Amazon Web Services (AWS) and other cloud providers, there are a variety of services that you can use to analyze and visualize data that you have obtained from your IoT devices, from simple data storage and retrieval options that you can configure without much difficulty to more complex analytics and machine learning (ML) tools, which you may have to learn and fine-tune, that you perform as part of the analysis.

Often, data analytics is the piece of the puzzle that completes the picture that we are trying to architect with our IoT networks, as even with edge networks in which we process data on the edge nodes to reduce costs, there is usually...

Technical requirements

This chapter will require you to have the following software installed

• Arduino IDE
• AWS account

We will be running our programs on Python and have a bit of Structured Query Language (SQL) syntax, a standardized programming language used for managing and manipulating relational databases, that we need to use as part of querying data in this chapter; again, don’t worry if you don’t understand some of the code — we will walk you through it and get you down to understanding how each part of the code works in no time.

You can access the GitHub folder for the code that is used in this chapter at https://github.com/PacktPublishing/IoT-Made-Easy-for-Beginners/tree/main/Chapter10.

Introduction to data analysis at scale

Data analysis is often done at scale to analyze large sets of data using the capabilities of cloud computing services such as AWS. Designing a workflow for the data analysis to follow is the pivotal starting point for this to be performed. This will follow five main categories: collection, storage, processing, visualization, and data security.

In this section, we will be introducing you to data analysis on AWS, discussing which services we can use as part of AWS to perform the data analytics workloads we need it to, and walking through the best practices that are part of this. We will understand how to design and incorporate workflows into the IoT network that we currently have and work with it to better power our capabilities.

Data analysis on AWS

Data analysis on AWS can be summarized in five main steps. These steps can be seen in the following diagram:

Figure 10.1 – Data analysis workflow on AWS

...

Analysis on the cloud and outside

When working with data, it often is necessary to visualize data to gain insights and make informed decisions. With services such as Amazon QuickSight, you are able to do this and create interactive dashboards and reports. However, some organizations' requirements may necessitate the use of third-party services alongside AWS’ native tools.

Many third-party services can be used. In this section, we will discuss some of these third-party services, alongside how we can start architecting workloads within the cloud for data workflows and quickly ensure that they adhere to best practices, both when creating and evaluating them.

Third-party data services

In this section, we will talk about two different third-party data analytics and visualization services: Datadog and Prometheus.

Datadog

Datadog is a cloud-based monitoring and analytics platform that provides many features for monitoring and troubleshooting many aspects of an organization...

Practical – smart home insights with AWS IoT Analytics

In this practical exercise, we will explore IoT data analytics using AWS. Specifically, we will use AWS services such as S3, Glue, Athena, and QuickSight to analyze a dataset of IoT sensor readings collected from a smart home over a period of 1 month.

You will need the following software components as part of the practical:

• An AWS account (you can create one for free if you don’t have one already)
• A dataset of IoT sensor readings (you can create a sample dataset or use a publicly available dataset)

Let’s move to the various steps of the practical, as follows:

1. Download the occupancy detection dataset:
   1. We can obtain a dataset from https://github.com/PacktPublishing/IoT-Made-Easy-for-Beginners/tree/main/Chapter10/analyzing_smart_home_sensor_readings/datatest.csv.
   2. Open the dataset and take note of the fields inside it.
2. To start off, we will have to load our dataset into an Amazon...

Industrial data analytics

We have seen the usage of data analytics in the past two sections and how it can be beneficial for our workloads. Now, let’s look at how it can benefit industry cases and how we can accordingly evaluate our deployments based on the best practices that are set out for us.

Evaluating performance

Use services such as CloudWatch metrics to monitor the performance of the IoT Analytics pipeline, such as the number of messages processed, the time it takes to process each message, and the number of errors that are encountered. This will be critical for use in further analysis and eventual optimization. The following are factors to consider in evaluating performance:

• Analyze your data: We can use IoT Analytics SQL or other data analytics tools to identify any patterns or issues that we may need to address if they affect system performance.
• Optimize your pipeline: From the analysis of the data, we can optimize the pipeline by adding data normalization...

Practical – creating a data pipeline for end-to-end data ingestion and analysis

In this practical, we will look to create a data pipeline based on the AWS console. This will follow the architecture shown in the following diagram:

Figure 10.5 – Data pipeline workflow for data ingestion

We will have a device send data to a channel. The channel will receive the data and send it through the pipeline, which will pipe the data through to the data store. From the data store, we can then make SQL queries to create a dataset from which we will read the data.

We can now go ahead and start off by creating a channel.

Creating a channel

Let's create our channel as part of the analytics workflow:

1. If you have not already, sign in to the AWS console. Afterward, navigate to the IoT Analytics console. For your convenience, here is a link to the console: https://console.aws.amazon.com/iotanalytics/.
2. In the IoT Analytics dashboard,...

Summary

In this chapter, we have covered the fundamentals of data analytics within IoT workloads, discussing how different services within AWS can handle analysis and storage loads that are required as part of our IoT cloud workflow. We then learned more about how we can design and develop the implementation of our architecture according to our use case and learned how to practically use the offerings from AWS IoT Analytics to provision an end-to-end data pipeline.

In the next chapter, we will be discussing security and privacy within IoT, which is an imperative topic to talk about, given how it has become even more prevalent as more and more people have shifted their workloads into the cloud.

Further reading

For more information on what was covered in this chapter, please refer to the following links:

• Understand more on data lakes and analytics solutions provided by AWS: https://aws.amazon.com/big-data/datalakes-and-analytics/
• Review more IoT Analytics success cases provided by AWS: https://aws.amazon.com/iot-analytics/
• Look at another reference architecture for an AWS serverless data analytics pipeline: https://aws.amazon.com/blogs/big-data/aws-serverless-data-analytics-pipeline-reference-architecture/
• Take a look at an implementation of real-time monitoring of industrial machines that utilizes AWS IoT: https://ieeexplore.ieee.org/document/10016452/
• Look at how a connected factory was able to leverage its offerings based on AWS IoT: https://aws.amazon.com/blogs/iot/connected-factory-offering-based-on-aws-iot-for-industry-4-0-success/
• Explore more on architecting industrial IoT workloads based on the cloud: https://us.nttdata.com/en/blog/2022...