Chapter case study

We will start this chapter with a case study example. In the previous chapter, we created our first Oozie Workflow to delete a given directory; we will build on top of that.

In this chapter, our use case is as follows.

On a daily basis we get incoming data in a HDFS directory. Our Workflow comes into action to process it via a simple Pig script. If we find the directory empty, we send a mail to the support team stating we did not get any data today. This is a very common data ingestion pattern in Hadoop for file-based loads.

There are many concepts, which will be introduced by use of this example; I thought to do it this way rather than sharing the concept first and sharing the example later. Using this example, we will cover the following concepts:

• Decision nodes

• Expression language

• Oozie command-line execution

Let's get started. The data ingestion pipeline for our use case can be represented as follows:

Pig Preprocess Decision node

Open Hue and go to Editor | Workflows to create...