Chapter 7: In the End, It Is All about Data, Isn't it?

Open any modern big data or advanced data analytics book or reference and without any doubt, you will find the phrase Internet of Things (IoT) mentioned frequently. Do you know why? Because IoT is considered one of the greatest data sources for the big data domain.

As stated in the earlier chapters, no one buys technologies just for the sake of playing with them. Technologies are invented and keep evolving to solve business problems. The main purpose of IoT technologies, in the end, is to use valuable real-world data to gain business value and insights; this is what IoT technologies provide to business stakeholders.

Collecting data is one benefit of IoT technologies, and another is managing and controlling IoT devices remotely. For the latter benefit, some might argue that managing devices remotely has already been available for decades now – mobile phone devices are a great example – so accordingly, we...

An IoT data analytics overview

Traditional data analytics or Business Intelligence (BI) solutions have been available for decades now. They follow the standard and well-known data analytics or data mining process that starts with data extraction from source systems. This is followed by the data transformation process and then loading data into purpose-fit data analytics stores, such as SQL-based data warehouses or a relational database. This process is usually called Extract, Transform, and Load (ETL).

Traditional data analytics is different from modern or advanced data analytics; in traditional data analytics, a business analyst or business owner starts by already having data from different data sources. Then, they will ask the question, OK I have all the data – what kind of information will I get out of such raw data? Then, they ask the question, Now that I have the information from the raw data, what kind of business insights will I get out of such valuable information...

Data analytics solution building blocks

Let's go step by step here to understand the different solution building blocks required to build a data analytics solution.

Data sources

I know this book is about IoT, and we mentioned earlier that IoT is one of the greatest data sources for any business organization, but based on the analytics questions you have to hand, you might need some other data sources to do a proper analysis and answer those business data analytics questions.

For example, you might want data from an external data source such as weather data or an IoT device's metadata, as the data coming from IoT devices is usually raw and small due to bandwidth and power limitations; hence, you can enrich raw data in the cloud by using another IoT device data repository, such as IoT device metadata or IoT device registry data.

To conclude, you might require more data sources besides data coming from IoT devices, so identifying those data sources should be the...

AWS IoT analytics

AWS IoT Analytics is a fully managed IoT analytics service that helps you collect, preprocess, enrich, store, and analyze IoT data at scale.

The AWS IoT Analytics service is fully integrated with AWS IoT Core; this is a great feature, as you can build IoT data analytics solutions more quickly using this managed service.

The AWS IoT Analytics service also integrates with other AWS and non-AWS services, and it accepts data from many data sources, so you might use a different IoT core from a different vendor and use the AWS IoT Analytics service for analytics purposes. In other words, you don't need to use AWS IoT Core to use AWS IoT Analytics; however, if you already use AWS IoT Core, then it is much easier to go with AWS IoT Analytics, as it is fully integrated with AWS IoT Core.

Let's see AWS IoT Analytics' main solution building blocks:

Figure 7.3 – The AWS IoT Analytics service

The architecture is simple, as...

Other IoT analytics-related AWS services

The AWS IoT platform offers some services that cover business and technical areas needed for building a complete E2E IoT solution in AWS. In the next section, we will start with the AWS IoT Events service.

AWS IoT events

AWS IoT Events is a fully managed service that detects and responds to events from different IoT sensors, services, and applications.

You can say that AWS IoT Events is a fully managed implementation of a Complex Event Processing (CEP) architecture where the CEP engine aggregates, processes, and analyzes a massive stream of data or events to gain real-time insights from the events that occurred.

The definition of event here means a business event, such as a failure in equipment or fraud detection.

So, business domain experts can build a detection model for their business using the AWS IoT Events console by defining the input that the detection model needs to work upon, the logic that they need to apply on the...

Summary

In this chapter, you learned about IoT data analytics, and the true and ultimate business benefits of IoT technologies, which is getting data from the physical world to gain more insights and act upon that data. You also learned about traditional data analytics versus modern or advanced data analytics, and the different types of data analytics (descriptive, diagnostic, predictive, and prescriptive).

You also learned about the different building blocks of IoT data analytics solutions such as Lambda architecture, data lakes, batch and streaming data processing, query engines, different analytics data stores, visualization, data exploration tools, and technologies used in data analytics solutions.

You learned about the fully managed AWS IoT Analytics service, and how it works and integrates with other related AWS services such as AWS IoT Events and AWS IoT SiteWise.

In the next chapter, we will cover some of the architecture and design paradigms and concepts used in building...