Chapter 5. Spark Streaming

Spark Streaming adds the holy grail of big data processing—that is, real-time analytics—to Apache Spark. It enables Spark to ingest live data streams and provides real-time intelligence at a very low latency of a few seconds.

In this chapter, we are going to cover the following recipes:

• WordCount using Structured Streaming
• Diving into Structured Streaming
• Streaming Twitter data
• Streaming using Kafka
• Understanding streaming challenges

Introduction

Streaming is the process of dividing continuously flowing input data into discrete units so that it can be processed easily. Familiar examples in real life are streaming video and audio content (though a user can download the full movie before he/she can watch it, a faster solution is to stream data in small chunks that start playing for the user while the rest of the data is being downloaded in the background).

Real-world examples of streaming, besides multimedia, are the processing of market feeds, weather data, electronic stock trading data, and so on. All these applications produce large volumes of data at very fast rates and require special handling of the data so that you can derive insight from the data in real time.

Streaming has a few basic concepts; it'll be better if we discuss them before we focus on Spark Streaming. The rate at which a streaming application receives data is called data rate and is expressed in the form of kilobytes per second (Kbps) or megabytes per...

WordCount using Structured Streaming

Let's start with a simple example of streaming in which in one terminal, we will type some text and the streaming application will capture it in another window.

How to do it...

1. Start the Spark shell:

$ spark-shell  

2. Create a DataFrame to read what's coming on port 8585:

scala> val lines = spark.readStream.format("socket").option("host","localhost").option("port",8585).load

3. Cast the lines from DataFrame to Dataset with the String datatype and then flatten it:

scala> val words = lines.as[String].flatMap(_.split(" "))

4.  Do the word count:

scala> val wordCounts = words.groupBy("value").count()

5. Start the netcat server in a separate window:

$ nc -lk 8585

6. Come back to the previous terminal and print the complete set of counts to the console every time it is updated:

scala> val query = wordCounts.writeStream.outputMode("complete").format("console").start()

7. Now go back to the terminal where you started netcat and enter different lines, such as to be or not to be...

Taking a closer look at Structured Streaming

Structured Streaming has been introduced in various places in this chapter, but let's use this recipe to discuss some more details. Structured Streaming is essentially a stream-processing engine built on top of the Spark SQL engine. 

An alternative way to look at streaming data is to think of it as an infinite/unbounded table that gets continuously appended as new data arrives.

The four fundamental concepts in Structured Streaming are:

• Input table: To input the table
• Trigger: How often the table gets updated
• Result table: The final table after every trigger update
• Output table: What part of the result to write to storage after every trigger

A query may be interested in only newly appended data (since the last query), all of the data that has been updated (including appended obviously), or the whole table; this leads to the three output modes in Structured Streaming, as follows:

• Append
• Update
• Complete

The DataFrame/Dataset API that is used for bounded tables...

Streaming Twitter data

Twitter is a famous microblogging platform. It produces a massive amount of data with around 500 million tweets sent each day. Twitter allows its data to be accessed by APIs, and that makes it the poster child of testing any big data streaming application.

In this recipe, we will see how we can live stream data in Spark using Twitter-streaming libraries. Twitter is just one source of providing streaming data to Spark and has no special status. Therefore, there are no built-in libraries for Twitter. Spark does provide some APIs to facilitate the integration with Twitter libraries, though.

An example use of a live Twitter data feed can be to find trending tweets in the last 5 minutes.

How to do it...

1. Create a Twitter account if you do not already have one.
2. Go to http://apps.twitter.com.
3. Click on Create New App.
4. Fill out the Name, Description, Website, and Callback URL fields and then click on Create your Twitter Application. You will receive a screen like this:

5. You will reach...

Streaming using Kafka

Kafka is a distributed, partitioned, and replicated commit log service. In simple words, it is a distributed messaging server. Kafka maintains the message feed in categories called topics. An example of a topic can be the ticker symbol of a company you would like to get news about, for example, CSCO for Cisco.

Processes that produce messages are called producers and those that consume messages are called consumers. In traditional messaging, the messaging service has one central messaging server, also called the broker. Since Kafka is a distributed messaging service, it has a cluster of brokers, which functionally acts as one Kafka broker, as shown here:

For each topic, Kafka maintains the partitioned log. This partitioned log consists of one or more partitions spread across the cluster, as shown in the following figure:

Kafka borrows a lot of concepts from Hadoop and other big data frameworks. The concept of partition is very similar to the concept of InputSplit in Hadoop...

Understanding streaming challenges

There are certain challenges every streaming application faces. In this recipe, we will develop some understanding of these challenges.

Late arriving/out-of-order data

If there is leader selection in streaming challenges, it would go to the late data. This is such a streaming-specific issue that folks not very familiar with streaming find it surprising that this issue is so prevalent. 

There are two notions of time in streaming:

• Event time: This is the time when an event actually happened, for example, measuring the temperature on a drive to an industrial site. Almost always, this event will contain this time as part of the record.
• Processing time: This is measured by the program that processed the event, for example, if the time series IoT event is processed in the cloud, then the processing time is the time this event reached the component (like Kinesis), which is doing the processing. 

In stream-processing applications, this time lag between the event time...