Chapter 3. Oozie Fundamentals

In this chapter, we will see how to create Oozie Workflow to solve a given business problem. Remember we are learning the what part of the data pipeline solution using Workflow in this part of book. We will eventually move to the when (time) part using Coordinators in the coming chapters.

In this chapter, we will do the following:

• Create an Oozie Workflow using Hue and by manual XML writing

• Run Oozie applications with and without Hue

• Submit Oozie jobs from command line

• Understand the concept of Control nodes

• Understand Workflow states

• Use expression language functions

• Use the Oozie Email action

Chapter case study

We will start this chapter with a case study example. In the previous chapter, we created our first Oozie Workflow to delete a given directory; we will build on top of that.

In this chapter, our use case is as follows.

On a daily basis we get incoming data in a HDFS directory. Our Workflow comes into action to process it via a simple Pig script. If we find the directory empty, we send a mail to the support team stating we did not get any data today. This is a very common data ingestion pattern in Hadoop for file-based loads.

There are many concepts, which will be introduced by use of this example; I thought to do it this way rather than sharing the concept first and sharing the example later. Using this example, we will cover the following concepts:

• Decision nodes

• Expression language

• Oozie command-line execution

Let's get started. The data ingestion pipeline for our use case can be represented as follows:

Pig Preprocess Decision node

Open Hue and go to Editor | Workflows to create...

Summary

In this chapter, we covered a lot of ground. We saw how to write Oozie workflow.xml and run it via the Oozie command line. We also discussed how to check the status of the same using command line. Besides this, we covered the concepts of Decision and Fork nodes, EL functions, and Workflow states.

In the next chapter, we will discuss Coordinators that decide the when component of data processing pipeline scheduling via Oozie. We will also see how to run MapReduce jobs using Oozie.