Implementing a Custom Industrial IoT Platform

In Chapter 7, Developing Industrial IoT and Architecture, we described the general flow of processing I-IoT data. In this chapter, we are going to update our flow, using the most common open source technologies to develop our platform from scratch. These include the following:

• KairosDB as a TSDB
• Neo4j as an asset DB
• Kafka and Mosquitto to collect data and adapt
• Airflow as an analytics platform

The purpose of our exercise is not to develop a real I-IoT platform, but instead to discuss some key topics related to I-IoT and to introduce various technologies that we can use in our on-premise I-IoT stack. At the end of this chapter, we will compare this solution with other open source solutions such as Kaa IoT or Eclipse IoT.

In this chapter, we will cover the following topics:

• Developing a platform based on open source technologies
• Using...

Technical requirements

In this chapter, we need the following:

• Docker Community Edition: https://store.docker.com
• JDK 1.8: http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html
• Git: https://git-scm.com/downloads
• Node.js 8+: https://nodejs.org
• Python 3.7: https://www.python.org/downloads/

To get a complete solution of the exercises, take a look at the repository at https://github.com/PacktPublishing/Hands-On-Industrial-Internet-of-Things .

An open source platform in practice

According to surveys carried out by the Eclipse Foundation, 20% of IoT platforms have been developed as on-premise solutions. This shows a positive trend with regard to the use of the public cloud, but there are dozens of reasons why we might want to continue to develop using on-premise or open-source solutions:

• Return of investment: We might have a budget that is too small to justify a big investment
• Technology: We might want to use technology that does not depend strictly on a supplier
• Privacy: We might want to export data outside our country
• Data shadowing: We might need a copy of the data on our legacy platform, either to experiment with or just as a backup
• Integration: We might be implementing an inter-cloud platform, meaning we need an an integration layer
• Experience: We might be developing our first I-IoT platform, so we want to start...

Mosquitto as MQTT connector

To install Mosquitto, we can follow the instructions from https://mosquitto.org/download/, or we can use Docker as usual.

1. From the command console, run the following command:

$ docker run -it -p 1883:1883 -p 9001:9001 eclipse-mosquitto

2. We are going to deploy the official Mosquito distribution, so we can test our MQTT broker. We need to install an MQTT client. Run the following command with an administrative user:

$ npm install mqtt-cli -g

3. Subscribe to a topic as follows:

$ mqtt-cli localhost topic/hello payload -w

4. Publish our first message as follows:

$ mqtt-cli localhost topic/hello "Test hello message"

The outcome should look like this:

$ mqtt-cli localhost topic/hello payload -w
Topic: topic/hello Message: payload
Topic: topic/hello Message: Test hello message

Apache Kafka as a data dispatcher

For our proposed architecture, we need to decouple acquisition from processing, improving the scalability and the independence of the layers. To achieve this goal, we can use a queue. We could either use Java Message Service (JMS) or Advanced Message Queuing Protocol (AMQP), but in this case we are going to use Apache Kafka. This is supported by most common analytics platforms, it has a very high performance and scalability, and it also has a good analytics framework.

In Kafka, each topic is divided into a set of logs called partitions. The producers write to the tail of Kafka's logs and consumers read the logs. Apache Kafka scales topic consumption by distributing partitions among a consumer group. A consumer group is a set of consumers which share a common group identifier. The following diagram shows a topic with three partitions and two...

Storing time-series data on Apache Cassandra

In the previous sections, we learned how to work with an MQTT stream and Kafka to apply rule-based analytics. Now we want to store our data in a scalable and fault-tolerant database. We can use various NoSQL databases, such as HBase, Parquet, or Kudu, but for our platform we are going to use Apache Cassandra.

Apache Cassandra

Apache Cassandra is a decentralized NoSQL database that has a good level of scalability and high availability without compromising performance. Apache Cassandra supports thousands of nodes and different levels of replicas and consistency. It also has a high level of data sharding.

Apache Cassandra is organized as a ring of nodes. Each node takes care of a portion...

Building an asset registry to store asset information

In Chapter 7, Developing Industrial IoT and Architecture, we highlighted the importance of the asset registry and its role within the context of the I-IoT. The asset registry can maintain a lot of information, including the class of the model of our equipment, the ID of the equipment and its properties, the components of the equipment, the measures we have to monitor, the thresholds and functional parameters of our measures, and the hierarchy of our fleet. A typical hierarchy of our assets might look as follows:

Asset Instance ID:Asset Class ID -> Asset Instance Child ID:Asset Class ID

In the following example, the ACME: Enterprise manages SydneyPlant. This in turn hosts a subsystem, which includes CT001:

ACME:Enterprise -> SydneyPlant:Plant -> Train1:Segment -> CT001:Pump

CT001 is a pump, which we can use to...

Pro and cons of the proposed platform

The platform we have proposed in this chapter has highlighted several important concepts about I-IoT and data processing. Let's take a look at some of its advantages and disadvantages.

Pros are as follows:

• The data is stored locally or in an on-premise cloud that is vendor-agnostic
• We can use a federated architecture to grant the right ownership of the intellectual property
• We can plug in new applications, avoiding the limitation of a specific technology
• Reliability can be improved by configuring multiple nodes of Docker instances

Cons are as follows:

• Edge management is completely missing here. This includes features, such as patching, security, or certificate renewal which are normally supported by a vendor.
• The current solution doesn't include an Manufacturing Execution System (MES) and Enterprise Resource Planning (ERP). A...

Summary

In this chapter, we learned how to develop an I-IoT platform from scratch. The purpose of this was to highlight the issues we might encounter during the development of an I-IoT platform.

In the next chapter, we will discover the most common IoT and I-IoT platform.

Questions

1. What is the most important benefit of using Kafka instead of RabbitMQ or AMQP?
   1. Performance and scalability
   2. Routing and protocols
   3. Reliability
2. What are the most important differences between cold-path analytics and hot-path analytics?
   1. Cold-path analytics use real-time processing
   2. In hot-path analytics, data is processed before storage
   3. In cold-path analytics, data is stored and sent to the hot path
3. Why would you use a TSDB database rather than a SQL/NoSQL standard database to store time-series?
   1. Data-sharding
   2. It has a specific API to interpolate data and aggregate data
   3. Scalability and reliability

Further reading

Read the following articles for more information:

• Apache Kafka 1.0 Cookbook: https://www.packtpub.com/big-data-and-business-intelligence/apache-kafka-10-cookbook
• Learning Apache Cassandra - Second Edition : https://www.packtpub.com/big-data-and-business-intelligence/learning-apache-cassandra-second-edition
• Learning Neo4j 3.x - Second Edition: https://www.packtpub.com/big-data-and-business-intelligence/learning-neo4j-3x-second-edition
• The Predix Meridium Asset model: https://www.meridium.com/secure/documentation/Help/Unified_V1020/Default/Subsystems/PredixAPM_Assets/Content/am_apm_ingest_assets3/c_apm_asset_about_asset_model.htm
• ANSI ISA-95: https://www.isa.org/isa95/
• IoT Eclipse: https://iot.eclipse.org/white-papers/
• Apache Airflow integration: https://airflow.apache.org/integration.html