The first three chapters of this book all dealt with batch data. Having learned about the installation of Hadoop, data ingestion tools and techniques, and data stores, let's turn to data streaming. Not only will we look at how we can handle real-time data streams, but also how to design pipelines around them.
In this chapter, we will cover the following topics:
Let's understand a few key concepts relating to real-time streaming applications in the following sections.
The data stream is a continuous flow of data from one end to another end, from sender to receiver, from producer to consumer. The speed and volume of the data may vary; it may be 1 GB of data per second or it may be 1 KB of data per second or per minute.
In batch processing, data is collected in batches and each batch is sent for processing. The batch interval can be anything...
In the following sections we will walk through some important real-time streaming components.
The message queue lets you publish and subscribe to a stream of events/records. There are various alternatives we can use as a message queue in our real-time stream architecture. For example, there is RabbitMQ, ActiveMQ, and Kafka. Out of these, Kafka has gained tremendous popularity due to its various unique features. Hence, we will discuss the architecture of Kafka in detail. A discussion of RabbitMQ and ActiveMQ is beyond the scope of this book.
Apache Storm is a free and open source distributed real-time stream processing framework. At the time of writing this book, the stable release version of Apache Storm is 1.0.5. The Storm framework is predominantly written in the Clojure programming language. Originally, it was created and developed by Nathan Marz and the team at Backtype. The project was later acquired by Twitter.
During one of his talks on the Storm framework, Nathan Marz talked about stream processing applications using any framework, such as Storm. These applications involved queues and worker threads. Some of the data source threads write messages to queues and other threads pick up these messages and write to target data stores. The main drawback here is that source threads and targets threads do not match the data load of each other and this results in data pileup. It also results in data loss...
Apart from Apache Storm, there are quite a few other open source real-time data streaming frameworks. In this section, I will discuss in brief only open source non-commercial frameworks. But, at the end of this section, I will provide a few URLs for a few commercial vendor products that offer some very interesting features.
Kafka Streams is a library for building streaming applications. Kafka Streams is a client library for building applications and microservices, where the input and output data are stored in Kafka Clusters. The Kafka Streams API transforms and enriches the data.
The following are the important features of the Kafka Streams API:
The main focus of Spark Streaming is stream-batching operation, called micro-batching. This programming model suits many use cases, but not all use cases require real-time stream processing with sub-second latency. For example, a use case such as credit card fraud prevention requires millisecond latency. Hence, the micro-batching programming model is not suited there. (But, the latest version of Spark, 2.4, supports millisecond data latency).
Apache Flink supports millisecond latency and is suited for use cases such as fraud detection and like.
Spark uses micro-batches to process events while Storm processes events one by one. It means that Spark has a latency of seconds while Storm provides a millisecond of latency. Spark Streaming provides a high-level abstraction called a Discretized Stream or DStream, which represents a continuous sequence of RDDs. (But, the latest version of Spark, 2.4 supports millisecond data latency.) The latest Spark version supports DataFrames.
Almost the same code (API) can be used for Spark Streaming and Spark batch jobs. That helps to reuse most of the code base for both programming models. Also, Spark supports Machine learning and the Graph API. So, again, the same codebase can be used for those use cases as well.
In this chapter, we started with a detailed understanding of real-time stream processing concepts, including data stream, batch vs. real-time processing, CEP, low latency, continuous availability, horizontal scalability, storage, and so on. Later, we learned about Apache Kafka, which is a very important component of modern real-time stream data pipelines. The main features of Kafka are scalability, durability, reliability, and high throughput.
We also learned about Kafka Connect; its architecture, data flow, sources, and connectors. We studied case studies to design a data pipeline with Kafka Connect using file source, file Sink, JDBC source, and file Sink Connectors.
In the later sections, we learned about various open source real-time stream-processing frameworks, such as the Apache Storm framework. We have seen a few practical examples, as well. Apache Storm is distributed...
© 2018 Packt Publishing Limited All Rights Reserved
