In this chapter, we will extend our deployment of machine learning models beyond batch processing in order to learn from data, make predictions, and identify trends in real time! We will develop and deploy a real-time stream processing and machine learning application comprised of the following high-level technologies:
So far in this book, we have been performing batch processing—that is, we have been provided with bounded raw data files and processed that data as a group. As we saw in Chapter 1, The Big Data Ecosystem, stream processing differs from batch processing in the fact that data is processed as and when individual units, or streams, of data arrive. We also saw in Chapter 1, The Big Data Ecosystem, how Apache Kafka, as a distributed streaming platform, allows us to move real-time data between systems and applications in a fault-tolerant and reliable manner via a logical streaming architecture comprising of the following components:
Apache Kafka allows us to move real-time data reliably between systems and applications. But we still need some sort of processing engine to process and transform that real-time data in order ultimately to derive value from it based on the use case in question. Fortunately, there are a number of stream processing engines available to allow us to do this, including—but not limited—to the following:
Though a detailed comparison of the available stream processing engines is beyond the scope of this book, you are encouraged to explore the preceding links...
In this section, we will develop an end-to-end stream processing pipeline that is capable of streaming data from a source system that generates continuous data, and thereafter able to publish those streams to an Apache Kafka distributed cluster. Our stream processing pipeline will then use Apache Spark to both consume data from Apache Kafka, using its Structured Streaming engine, and apply trained machine learning models to these streams in order to derive insights in real time using MLlib. The end-to-end stream processing pipeline that we will develop is illustrated in Figure 8.4:
In the case...
In this chapter, we developed Apache Kafka producer and consumer applications and utilized Spark's Structured Streaming engine to process streaming data consumed from a Kafka topic. In our real-world case study, we designed, developed, and deployed an end-to-end stream processing pipeline that was capable of consuming real tweets being authored across the world and then classified their underlying sentiment using machine learning, all of which was done in real time.
In this book, we went on both a theoretical and a hands-on journey through some of the most important and exciting technologies and frameworks that underpin the data-intelligence-driven revolution being seen across industry today. We started out by describing a new breed of distributed and scalable technologies that allow us to store, process, and analyze huge volumes of structured, semi-structured, and...
© 2018 Packt Publishing Limited All Rights Reserved
