Data Discovery – Understanding Our Data before Ingesting It

As you may already have noticed, data ingestion is not just retrieving data from a source and inserting it in another place. It involves understanding some business concepts, secure access to the data, and how to store it, and now it is essential to discover our data.

Data discovery is the process of understanding our data’s patterns and behaviors, ensuring the whole data pipeline will be successful. In this process, we will understand how our data is modeled and used, so we can set up and plan our ingestion using the best fit.

In this chapter, you will learn about the following:

• Documenting the data discovery process
• Configuring OpenMetadata
• Connecting OpenMetadata to our database

Technical requirements

You can also find the code from this chapter in its GitHub repository here: https://github.com/PacktPublishing/Data-Ingestion-with-Python-Cookbook.

Documenting the data discovery process

In recent years, manual data discovery has been rapidly deprecated, giving rise to machine learning and other automated solutions, bringing fast insights into data in storage or online spreadsheets, such as Google Sheets.

Nevertheless, many small companies are just starting out their businesses or data areas, so implementing a paid or cost-related solution might not be a good idea right away. As data professionals, we also need to be malleable when applying the first solution to a problem – there will always be space to improve it later.

Getting ready

This recipe will cover the steps to start the data discovery process effectively. Even though, here, the process is more related to the manual discovery steps, you will see it also applies to the automated ones.

Let’s start by downloading the datasets.

For this recipe, we are going to use the The evolution of genes in viruses and bacteria dataset (https://www.kaggle.com...

Configuring OpenMetadata

OpenMetadata is an open source tool used for metadata management, allowing the process of data discovery and governance. You can find more about it here: https://open-metadata.org/.

By performing a few steps, it is possible to create a local or production instance using Docker or Kubernetes. OpenMetadata can connect to multiple resources, such as MySQL, Redis, Redshift, BigQuery, and others, to bring the information needed to build a data catalog.

Getting ready

Before starting our configuration, we must install OpenMetadata and ensure the Docker containers are running correctly. Let us see how it is done:

Note

At the time this book was written, the application was in the 0.12 version and with some documentation and installation improvements. This means the best approach to installing it may change over time. Please refer to the official documentation for it here: https://docs.open-metadata.org/quick-start/local-deployment.

1. Let’...

Connecting OpenMetadata to our database

Now that we have configured our Data Discovery tool, let’s create a sample connection to our local database instance. Let’s try to use PostgreSQL to do an easy integration and practice another database usage.

Getting ready

First, ensure our application runs appropriately by accessing the http://localhost:8585/my-data address.

Note

Inside OpenMetadata, the user must have the Data Steward or Administration role to create connections. You can switch to the admin user using the previous credentials we saw.

You can check the Docker status here:

Figure 3.16 – Active containers are shown in the Docker desktop application

Use PostgreSQL for testing. Since we already have a Google project ready, let us create a SQL instance using the PostgreSQL engine.

As we kept the queries to create the database and tables in Chapter 2, we can build it again in Postgres. The queries can also be found...

Further reading

• https://nira.com/data-discovery/
• https://coresignal.com/blog/data-discovery/
• https://www.polymerhq.io/blog/diligence/what-is-data-discovery-guide/
• https://bi-survey.com/data-discovery
• https://www.heavy.ai/technical-glossary/data-discovery
• https://www.datapine.com/blog/what-are-data-discovery-tools/
• https://www.knowsolution.com.br/data-discovery-como-relaciona-bi-descubra/ (in Portuguese)

Other tools

If you are interested in learning more about other data discovery tools available on the market, here are some:

• Tableau: Tableau (https://www.tableau.com/) is more extensively used for data visualizations and dashboards but comes with some features to discover and catalog data. You can read more about how to use Tableau for data discovery on their resources page here: https://www.tableau.com/learn/whitepapers/data-driven-organization-7-keys-data-discovery.
• OpenDataDiscovery (free and open source): OpenDataDiscovery has recently...