Data Profiling with IBM Information Analyzer

Data profiling is essentially data mining, but for a different purpose. You mine data to understand, to gain better knowledge about the data. While the more common use of data mining is for gaining the data insights for business purpose (e.g. customer buying characteristics), data profiling is for technical purpose. To be more precise, you do data profiling to gather and analyze the technical metadata characteristics of the data. Information Analyzer, data profiling software from IBM, helps you gain insight into such technical metadata characteristics as, for example, column data type and size (length).

This article is based on a case where a database table grew its size unexpectedly and its initial disk space allocation got strained. Looking at the growth pattern of the table, such as the number of new records (which was not that huge), didn’t give us any clue about the cause of the problem. In this article, Djoni Darmawikarta will step through the Information Analyzer process, run one of its functions called Column Analysis on a simple table (a scaled-down version of the real table), and show how the profiling output help solve the problem.

Information Analyzer is a client-server software. A data profiling user (metadata analyst) works on its GUI client, so to make it easier to show you how I solve the problem I’ll use a lot of screenshots.

Our example data is an Oracle table that has two columns and three rows (In real life, they can typically be more than 50 and a few millions, respectively).

When you start the Information Analyzer client, called Information Server Console, you’ll be shown its start-up screen; and then, its log-in window.

When your log-in is successful, the console main window will show up.

Assuming the Oracle table that we’d like to profile is new; we must identify it to the Analyzer, which technically means importing its metadata.

Make sure you have connected the Oracle database to the Information Analyzer server before you import the metadata of its tables.

Expand Metadata Management from the HOME drop-down menu.

Then, click Import Metadata.

Our example Oracle data (table) is in the CLROPER database (hosted in DDOM02), so select CLROPER and then click Identify Next Level.

It might take a while, particularly for a database that has many tables and many columns; so just wait.

On the completion message screen, click OK to close the screen.

All tables in CLROPER database will be identified (listed) including our example table named SPACE1. We’ll next identify the columns of our SPACE1 table; so select SPACE1 and then click Identify Next Level.

The result shows that Analyzer has correctly identified the two columns of the table.

Now, import metadata of all columns of the table by selecting the table and then clicking Import.

Click OK to continue.

Wait for completion.

Click OK on the successful completion screen.

We’re now done with the metadata of the data; we’re now ready to start our profiling task.

In Information Analyzer (as in most other software of these days) we group our profiling works into projects. Here, I just use an existing project (DJONI_TEST), so select Open Project from the drop-down arrow on the right of NO PROJECT SELECTED.

You’ll be shown the list of existing projects. Select your project, and click Open.

Our previous (existing) profiling works are shown.

Next, open click Project Properties from the OVERVIEW drop-down menu.

Go to the Data Sources tab. Our SPACE1 table is not in the list yet, as we haven’t identified it specifically in our project (we did in the previous steps at the server-wide level); so we need to add it into our project, click Add.

Expand the SPACE1 table to see its columns. Select all of the columns as we want to profile all of them, and then click OK.

When completed, click Save All, and then close the Project Properties window.

Now, we’re ready to profile our SPACE1 data, to analyze its columns. On the main toolbar select Investigate | Column analysis.