Executing PDI jobs from a filesystem (Simple)

Exclusive offer: get 50% off this eBook here
Instant Pentaho Data Integration Kitchen [Instant]

Instant Pentaho Data Integration Kitchen [Instant] — Save 50%

Explore the world of Pentaho Data Integration command-line tools which will help you use the Kitchen with this book and ebook

$19.99    $10.00
by Sergio Ramazzina | September 2013 | Open Source

In this article by Sergio Ramazzina, the author of the book Instant Pentaho Data Integration Kitchen, will guide you through simply starting a PDI job using the script Kitchen. In this case, the PDI job we are going to start is stored locally in the computer filesystem, but it could be anywhere in the network in any place that is directly accessible. You will learn how to start simple jobs both with and without a set of input parameters previously defined in the job.

Using command-line scripts is a fast way to start batches, but it is also the easiest way to schedule our jobs using our operating system's scheduler. The script accepts a set of inline arguments to pass the proper options required by the program to run our job in any specific situation.

(For more resources related to this topic, see here.)

Getting ready

To get ready for this article, we first need to check that our Java environment is configured properly; to do this, check that the JAVA_HOME environment variable is set. Even if all the PDI scripts, when started, call other scripts that try to find out about our Java execution environment to get the values of the JAVA_HOME variable, it is always a good rule of thumb to have that variable set properly anytime we work with a Java application.

The Kitchen script is in the PDI home directory, so the best thing to do to launch the script in the easiest way is to add the path to the PDI home directory to the PATH variable. This gives you the ability to start the Kitchen script from any place without specifying the absolute path to the Kitchen file location. If you do not do this, you will always have to specify the complete path to the Kitchen script file.

To play with this article, we will use the samples in the directory <book_samples>/sample1; here, <book_samples> is the directory where you unpacked all the samples of the article.

How to do it…

For starting a PDI job in Linux or Mac, use the following steps:

  1. Open the command-line terminal and go to the <book_samples>/sample1 directory.
  2. Let's start the sample job. To identify which job file needs to be started by Kitchen, we need to use the –file argument with the following syntax:

    –file: <complete_filename_to_job_file>

    Remember to specify either an absolute path or a relative path by properly setting the correct path to the file. The simplest way to start the job is with the following syntax:

    $ kitchen.sh –file:./export-job.kjb

  3. If you're not positioned locally in the directory where the job files are located, you must specify the complete path to the job file as follows:

    $ kitchen.sh –file:/home/sramazzina/tmp/samples/export-job.kjb

  4. Another option to start our job is to separately specify the name of the directory where the job file is located and then give the name of the job file. To do this, we need to use the –dir argument together with the –file argument. The –dir argument lets you specify the location of the job file directory using the following syntax:

    –dir: <complete_path_to_ job_file_directory>

    So, if we're located in the same directory where the job resides, to start the job, we can use the following new syntax:

    $ kitchen.sh – dir:. –file:export-job.kjb

  5. If we're starting the job from a different directory than the directory where the job resides, we can use the absolute path and the –dir argument to set the job's directory as follows:

    $ kitchen.sh –dir:/home/sramazzina/tmp/samples –file:export-job.kjb

For starting a PDI job with parameters in Linux or Mac, perform the following steps:

  1. Normally, PDI manages input parameters for the executing job. To set parameters using the command-line script, we need to use a proper argument. We use the –param argument to specify the parameters for the job we are going to launch. The syntax is as follows:

    -param: <parameter_name>= <parameter_value>

  2. Our sample job and transformation does accept a sample parameter called p_country that specifies the name of the country we want to export the customers to a file. Let's suppose we are positioned in the same directory where the job file resides and we want to call our job to extract all the customers for the country U.S.A. In this case, we can call the Kitchen script using the following syntax:

    $ kitchen.sh –param:p_country=USA -file=./export-job.kjb

    Of course, you can apply the –param switch to all the other three cases we detailed previously.

For starting a PDI job in Windows, use the following steps:

  1. In Windows, a PDI job from the filesystem can be started by following the same rules that we saw previously, using the same arguments in the same way. The only difference is in the way we specify the command-line arguments.
  2. Any time we start the PDI jobs from Windows, we need to specify the arguments using the / character instead of the – character we used for Linux or Mac. Therefore, this means that:

    -file: <complete_filename_to_job_file>

    Will become:

    /file: <complete_filename_to_job_file>

    And:

    –dir: <complete_path_to_ job_file_directory>

    Will become:

    /dir: <complete_path_to_ job_file_directory>

  3. From the directory <book_samples>/sample1, if you want to start the job, you can run the Kitchen script using the following syntax:

    C:\temp\samples>Kitchen.bat /file:./export-job.kjb

  4. Regarding the use of PDI parameters in command-line arguments, the second important difference on Windows is that we need to substitute the = character in the parameter assignment syntax with the : character. Therefore, this means that:

    –param: <parameter_name>= <parameter_value>

    Will become:

    /param: <parameter_name>: <parameter_value>

  5. From the directory <book_samples>/sample1, if you want to extract all the customers for the country U. S. A, you can start the job using the following syntax:

    C:\temp\samples>Kitchen.bat /param:p_country:USA /file:./exportjob.
    kjb

For starting the PDI transformations, perform the following steps:

  1. The Pan script starts PDI transformations. On Linux or Mac, you can find the pan.sh script in the PDI home directory. Assuming that you are in the same directory, <book_samples>/sample1, where the transformation is located, you can start a simple transformation with a command in the following way:

    $ pan.sh –file:./read-customers.ktr

  2. If you want to start a transformation by specifying some parameters, you can use the following command:

    $ pan.sh –param:p_country=USA –file:./read-customers.ktr

  3. In Windows, you can use the Pan.bat script, and the sample commands will be as follows:

    C:\temp\samples>Pan.bat /file:./read-customers.ktr

  4. Again, if you want to start a transformation by specifying some parameters, you can use the following command:

    C:\temp\samples>Pan.bat /param:p_country=USA /file:./readcustomers.
    ktr

Summary

IIn this article, you were guided through simply starting a PDI job using the script Kitchen. In this case, the PDI job we started were stored locally in the computer filesystem, but it could be anywhere in the network in any place that is directly accessible. You learned how to start simple jobs both with and without a set of input parameters previously defined in the job.

Using command-line scripts was a fast way to start batches, but it was also the easiest way to schedule our jobs using our operating system's scheduler. The script accepted a set of inline arguments to pass the proper options required by the program to run our job in any specific situation.

Resources for Article :


Further resources on this subject:


Instant Pentaho Data Integration Kitchen [Instant] Explore the world of Pentaho Data Integration command-line tools which will help you use the Kitchen with this book and ebook
Published: July 2013
eBook Price: $19.99
See more
Select your format and quantity:

About the Author :


Sergio Ramazzina

Sergio Ramazzina is an experienced software architect/trainer with 25 plus years of experience in the IT field on a broad number of projects for banks and major Italian companies, designing complex enterprise solutions in Java/JavaEE and Ruby. He started using Pentaho products from the very beginning in late 2003. He gained thorough experience by deploying Pentaho as Open Source BI solution, standalone, or deeply integrated in other applications as the analytical engine of choice.

In 2009, due to his experience in the Java/JavaEE world and the appreciation for the Open Source world and its main ideas, he began participating actively as a contributor to some of the Pentaho projects: JPivot, Saiku, CDF, CDA and gained the Pentaho Active Contributor level. At that time,he started participating as a BI architect and Pentaho expert on a wide number of projects where Open Source BI and Pentaho were the main players. In late 2010, he founded Serasoft, a young Italian consulting firm specialized in delivering high value Open Source Business Intelligence solutions. With the team in Serasoft, he used to share his passion and experience in designing and delivering highlyinnovative enterprise solutions to help users make their work more effective. In July 2013, he published his first book Pentaho: Instant Pentaho Data Integration Kitchen How-To. He is also passionate about skiing, tennis, and photography, and has a deep love for his young daughter Camilla.

Books From Packt


Pentaho Data Integration Cookbook Second Edition
Pentaho Data Integration Cookbook Second Edition

Pentaho Data Integration Beginner's Guide, Second Edition
Pentaho Data Integration Beginner's Guide, Second Edition

Pentaho Reporting 3.5 for Java Developers
Pentaho Reporting 3.5 for Java Developers

 Pentaho Data Integration 4 Cookbook
Pentaho Data Integration 4 Cookbook

Pentaho 5.0 Reporting by Example: Beginner’s Guide
Pentaho 5.0 Reporting by Example: Beginner’s Guide

Pentaho 3.2 Data Integration: Beginner's Guide
Pentaho 3.2 Data Integration: Beginner's Guide

OSWorkflow: A guide for Java developers and architects to integrating open-source Business Process Management
OSWorkflow: A guide for Java developers and architects to integrating open-source Business Process Management

Microsoft Windows Workflow Foundation 4.0 Cookbook
Microsoft Windows Workflow Foundation 4.0 Cookbook


No votes yet

Post new comment

CAPTCHA
This question is for testing whether you are a human visitor and to prevent automated spam submissions.
a
c
d
B
g
w
Enter the code without spaces and pay attention to upper/lower case.
Code Download and Errata
Packt Anytime, Anywhere
Register Books
Print Upgrades
eBook Downloads
Video Support
Contact Us
Awards Voting Nominations Previous Winners
Judges Open Source CMS Hall Of Fame CMS Most Promising Open Source Project Open Source E-Commerce Applications Open Source JavaScript Library Open Source Graphics Software
Resources
Open Source CMS Hall Of Fame CMS Most Promising Open Source Project Open Source E-Commerce Applications Open Source JavaScript Library Open Source Graphics Software