Chapter 8. Batch Analytics with Apache Flink

This chapter will introduce the reader to Apache Flink, illustrating how to use Flink for big data analysis, based on the batch processing model. We will look at DataSet APIs, which provide easy-to-use methods for performing batch analysis on big data.

In this chapter, we will cover the following topics:

• Introduction to Apache Flink
• Installing Flink
• Using the Scala shell
• Using the Flink cluster UI
• Batch Analytics using Flink

Introduction to Apache Flink

Flink is an open source framework for distributed stream processing, and has the following features:

• It provides results that are accurate, even in the case of out-of-order or late-arriving data
• It is stateful and fault tolerant, and can seamlessly recover from failures while maintaining an exactly-once application state
• It performs at a large scale, running on thousands of nodes with very good throughput and latency characteristics

The following is a screenshot from the official documentation that shows how Apache Flink can be used:

Another way of viewing the Apache Flink framework is shown in the following screenshot:

All Flink programs are executed lazily, when the program’s main method is executed, the data loading and transformations do not happen directly. Rather, each operation is created and added to the program’s plan. The operations are actually executed when the execution is explicitly triggered by an execute() call on the execution environment. Whether...

Installing Flink

In this section, we will download and install Apache Flink.

Flink runs on Linux, OS X, and Windows. To be able to run Flink, the only requirement is having a working Java 7.x (or higher) installation. If you are using Windows, please take a look at the Flink on Windows guide at https://ci.apache.org/projects/flink/flink-docs-release-1.4/start/flink_on_windows.html, which describes how to run Flink on Windows for local setups.

You can check your version of Java by issuing the following command:

java -version

If you have Java 8, the output will look something like this:

java version "1.8.0_111"
Java(TM) SE Runtime Environment (build 1.8.0_111-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.111-b14, mixed mode)

Downloading Flink

Download the Apache Flink binaries relevant to your platform at https://flink.apache.org/downloads.html:

Figure: Screenshot showing Apache Flink libraries

Download Hadoop version 2.8 by clicking on Download. You will see the download page in your browser, as...

Using the Flink cluster UI

Using the Flink cluster UI, you can understand and monitor what's running in your cluster and dig deeply into various jobs and tasks. You can monitor the job statuses, cancel jobs, or debug any problems with the jobs. By looking at logs, you can also diagnose problems with your code, and fix them.

The following is a list of Completed Jobs:

You can drill down into any particular job to see more details about the job's execution:

Figure: Drilling down a particular job to see job's execution

You can look at the Timeline of the job to get more details:

Figure: Screenshot to see Timeline of a job

The following screenshot shows the Task Managers tab, showing all of the task managers. This helps you understand the number and status of the task managers:

You can also check the Logs, as shown in the following screenshot:

The Metrics tab gives you details of the memory and CPU resources:

Figure: Screenshot showing details of the memory and CPU resources in Metrics tab

You can also...

Batch analytics

Batch Analytics in Apache Flink are quite similar to the streaming analytics in the way Flink handles both types of analytics using same APIs. This gives a lot of flexibility and allows code reuse across both the different types of analytics.

In this section, we will look at some analytical jobs on the sample data we are using OnlineRetail.csv. We will also be loading cities.csv and temperature.csv to do some more join operations.

Reading file

Flink comes with several built-in formats to create data sets from common file formats. Many of them have shortcut methods on the execution environment.

File-based

File based sources can be read using APIs which are listed as follows:

• readTextFile(path)/TextInputFormat: Reads files line wise and returns them as strings.
• readTextFileWithValue(path)/TextValueInputFormat: Reads files line wise and returns them as StringValues. StringValues are mutable strings.
• readCsvFile(path)/CsvInputFormat: Parses files of comma (or another char) delimited...

Summary

In this chapter, we have discussed Apache Flink and how Flink can be used to perform batch analysis on a large amount of data. We explored Flink and inner workings of Flink. Then we loaded and analyzed data performing transformations and aggregation operations. Then we explored how to perform Join operations on big data.

In the next chapter, we will discuss real-time analytics using Apache Flink.