Search icon
Arrow left icon
All Products
Best Sellers
New Releases
Books
Videos
Audiobooks
Learning Hub
Newsletters
Free Learning
Arrow right icon
Apache Oozie Essentials

You're reading from  Apache Oozie Essentials

Product type Book
Published in Dec 2015
Publisher
ISBN-13 9781785880384
Pages 164 pages
Edition 1st Edition
Languages
Author (1):
Jagat Singh Jagat Singh
Profile icon Jagat Singh

Data pipelines


In real Big Data projects, the Coordinators are scheduled tasks that are part of the data pipeline. For example, get data from some system and process it (this forms one Coordinator), and then another sub process can send the processed data to a database (this forms another Coordinator). Finally, both of them are abstracted to form Bundle. To think in terms of how to solve your job using Oozie, start by drawing the job Workflow on a whiteboard/paper. Then discuss with your team how you can create unit abstractions to run individually and in isolation.

Check out the following example.

The database has a record of daily rainfall in Melbourne. We import that data to Hadoop using a regular Coordinator job (Coordinator 1). Using another scheduled job, we send the results back to the database as shown in the following figure:

Data pipelines

Note

Exercise: Take the preceding example and make one Bundle that processes our rainfall data in the first Coordinator (using Pig script) and sends...

lock icon The rest of the chapter is locked
Register for a free Packt account to unlock a world of extra content!
A free Packt account unlocks extra newsletters, articles, discounted offers, and much more. Start advancing your knowledge today.
Unlock this book and the full library FREE for 7 days
Get unlimited access to 7000+ expert-authored eBooks and videos courses covering every tech area you can think of
Renews at $15.99/month. Cancel anytime}