Data Profiling with IBM Information Analyzer

Information Analyzer is a client-server software. A data profiling user (metadata analyst) works on its GUI client, so to make it easier to show you how I solve the problem I’ll use a lot of screenshots.

Our example data is an Oracle table that has two columns and three rows (In real life, they can typically be more than 50 and a few millions, respectively).

When you start the Information Analyzer client, called Information Server Console, you’ll be shown its start-up screen; and then, its log-in window.

Data Profiling with IBM Information Analyzer

Data Profiling with IBM Information Analyzer

When your log-in is successful, the console main window will show up.

Data Profiling with IBM Information Analyzer

Assuming the Oracle table that we’d like to profile is new; we must identify it to the Analyzer, which technically means importing its metadata.

Make sure you have connected the Oracle database to the Information Analyzer server before you import the metadata of its tables.

Expand Metadata Management from the HOME drop-down menu.

Data Profiling with IBM Information Analyzer

Then, click Import Metadata.

Data Profiling with IBM Information Analyzer

Our example Oracle data (table) is in the CLROPER database (hosted in DDOM02), so select CLROPER and then click Identify Next Level.

Data Profiling with IBM Information Analyzer

It might take a while, particularly for a database that has many tables and many columns; so just wait.

Data Profiling with IBM Information Analyzer

On the completion message screen, click OK to close the screen.

Data Profiling with IBM Information Analyzer

All tables in CLROPER database will be identified (listed) including our example table named SPACE1. We’ll next identify the columns of our SPACE1 table; so select SPACE1 and then click Identify Next Level.

Data Profiling with IBM Information Analyzer

The result shows that Analyzer has correctly identified the two columns of the table.

Data Profiling with IBM Information Analyzer

Now, import metadata of all columns of the table by selecting the table and then clicking Import.

Click OK to continue.

Data Profiling with IBM Information Analyzer

Wait for completion.

Data Profiling with IBM Information Analyzer

Click OK on the successful completion screen.

Data Profiling with IBM Information Analyzer

We’re now done with the metadata of the data; we’re now ready to start our profiling task.

In Information Analyzer (as in most other software of these days) we group our profiling works into projects. Here, I just use an existing project (DJONI_TEST), so select Open Project from the drop-down arrow on the right of NO PROJECT SELECTED.

Data Profiling with IBM Information Analyzer

You’ll be shown the list of existing projects. Select your project, and click Open.

Data Profiling with IBM Information Analyzer

Our previous (existing) profiling works are shown.

Data Profiling with IBM Information Analyzer

Next, open click Project Properties from the OVERVIEW drop-down menu.

Data Profiling with IBM Information Analyzer

Go to the Data Sources tab. Our SPACE1 table is not in the list yet, as we haven’t identified it specifically in our project (we did in the previous steps at the server-wide level); so we need to add it into our project, click Add.

Data Profiling with IBM Information Analyzer

Expand the SPACE1 table to see its columns. Select all of the columns as we want to profile all of them, and then click OK.

Data Profiling with IBM Information Analyzer

When completed, click Save All, and then close the Project Properties window.

Data Profiling with IBM Information Analyzer

Now, we’re ready to profile our SPACE1 data, to analyze its columns. On the main toolbar select Investigate | Column analysis.

Data Profiling with IBM Information Analyzer

Select all columns of the SPACE1 table to analyze, and click Run Column Analysis.

Data Profiling with IBM Information Analyzer

Click Submit.

Data Profiling with IBM Information Analyzer

Check status by clicking Details.

Data Profiling with IBM Information Analyzer

When the job status shows Schedule Complete, click Close to close the Activity Status (job status) window.

Data Profiling with IBM Information Analyzer

Close the Column Analysis window as well.

Data Profiling with IBM Information Analyzer

To check result, click Open Column Analysis.

Data Profiling with IBM Information Analyzer

Our profiling output shows the metadata characteristics of the two columns. Our focus is on their sizes; so if necessary scroll to the right to see the Length columns.

Data Profiling with IBM Information Analyzer

The Length has three columns: Defined, Inferred, and Selected. The Defined length of the first column (INTEGER1) is as defined in the metadata of the table we imported, which is 38. The Inferred length, which is 3, is produced, by Information Analyzer, by computing statistically the data lengths of all rows, based on the actual data values of the column; and then, it suggests (Selected) that 3 should be the length of this column. Similarly, Information Analyzer did the Inferred and Suggested on the other column, the LARGECHAR1.

Based on these output produced by Information Analyzer, we can decide how much we’d to reduce the length of the columns, which will certainly reduce the disk space needed for the data.

Summary

Using a data profiling tool, such as the IBM Information Analyzer, we can analyze and gain knowledge particularly large amount of data that otherwise would not be apparent. The Information Analyzer has much more functionalities; this article discussed only the basics of one of them (Column Analysis).

Books to Consider

comments powered by Disqus