Chapter 5. Extract, Transform, and Load Basics

We're moving along nicely into the process of designing and building a data warehouse. If you've been reading all the way through to here, you'll recall how we've introduced the Warehouse Builder software (how to install it along with the Oracle Database), looked at its architecture, and covered a short overview of the analysis and design phases for implementing a data warehouse project. We've defined our data sources and imported the metadata for them. We've designed our target structure into which we'll load the data. Congratulations for having read this far—don't give up now because we're not done yet. We still have to get data from our sources into our target. We will do that by:

1. Designing mappings in OWB.

2. Deploying the mappings to the database.

3. Running the mappings.

This chapter will expose ETL (Extract, Transform, and Load) for the first time in this book. ETL is the first step in building the mappings from source to target. We have sources...

ETL

The process of extracting, transforming, and loading data can appear rather complicated. We do have a special term to describe it, ETL, which contains the three steps mentioned. We're dealing with source data on different database systems from our target and a database from a vendor other than Oracle. Let's look from a high level at what is involved in getting that data from a source system to our target, and then take a look at whether to stage the data or not. We will then see how to automate that process in Warehouse Builder, which will relieve us of much of the work.

Manual ETL processes

First of all, we need to be able to get data out of that source system and move it over to the target system. We can't begin to do anything until that is accomplished, but what means can we use to do so? We know that the Oracle Database provides various methods to load data into it. There is an application that Oracle provides called SQL*Loader, which is a utility to load data from flat files. This...

Staging

We need to consider a practical aspect to this process that is related to ETL, as well as to the structure in our target database. This practical aspect is the question of whether to stage the source data in a temporary location before performing the transformations on it and loading it into the target structure. Staging is the process of copying the source data temporarily into a table(s) in our target database. Here we can perform any transformations that are required before loading the source data into the final target tables. The source data could actually be copied to a table in another database that we create just for this purpose, but it doesn't have to be. This process involves saving data to storage at any step along the way to the final target structure, and it can incorporate a number of intermediate staging steps. The source and target designations will be affected during the intermediate steps of staging. So we'll need to decide on a staging strategy, if any, before...

Mappings and operators in OWB

We are now going to look at the Warehouse Builder and its features for designing and building our ETL process. OWB handles this with what are called mappings. A mapping is composed of a series of operators that describe the sources, targets, and a series of operations that flow from source to target to load the data. It is all designed in a graphical manner using the Mapping Editor, which is available from the Design Center. Let's run the Design Center now and take a look at the Mapping Editor, its features, and some of the operators that are available to us. Launch the Design Center as we discussed in Chapter 2 in the Overview of Warehouse Builder Design Center section.

In the Design Center | Project Explorer window, expand the ACME_DW_PROJECT project (if it is not already expanded) by clicking on the plus sign beside it. To access the Mapping Editor, we need a mapping to work on. So to begin with, we can create an empty mapping at this point.

Note

There is...

Summary

This chapter has given us an overview of the Extract, Transform, and Load (ETL) process as well as the Warehouse Builder's support for designing our ETL process. We discussed the process of mapping and a little of what that involves in OWB. We took a look at the OWB Mapping Editor to get a feel for the windows available to us, and also looked at a list of some of the operators OWB provides for us to use in our mappings.

We're laying the groundwork here for the real fun that comes in the next chapter where we get to put this knowledge to use in designing a mapping. In the next chapter, we will also get to use some of these operaors.