Chapter 6: Building a 311 Data Pipeline

In the previous three chapters, you learned how to use Python, Airflow, and NiFi to build data pipelines. In this chapter, you will use those skills to create a pipeline that connects to SeeClickFix and downloads all the issues for a city, and then loads it in Elasticsearch. I am currently running this pipeline every 8 hours. I use this pipeline as a source of open source intelligence – using it to monitor quality of life issues in neighborhoods, as well as reports of abandoned vehicles, graffiti, and needles. Also, it's really interesting to see what kinds of things people complain to their city about – during the COVID-19 pandemic, my city has seen several reports of people not social distancing at clubs.

In this chapter, we're going to cover the following main topics:

• Building the data pipeline
• Building a Kibana dashboard

Building the data pipeline

This data pipeline will be slightly different from the previous pipelines in that we will need to use a trick to start it off. We will have two paths to the same database – one of which we will turn off once it has run the first time, and we will have a processor that connects to itself for the success relationship. The following screenshot shows the completed pipeline:

Figure 6.1 – The complete pipeline

The preceding screenshot may look complicated, but I assure you that it will make sense by the end of this chapter.

Mapping a data type

Before you can build the pipeline, you need to map a field in Elasticsearch so that you get the benefit of the coordinates by mapping them as the geopoint data type. To do that, open Kibana at http://localhost:5601. At the toolbar, select Dev Tools (the wrench icon) and enter the code shown in the left panel of the following screenshot, and then click the run arrow. If it was successful...

Building a Kibana dashboard

Now that your SeeClickFix data pipeline has loaded data in Elasticsearch, it would be nice to see the results of the data, as would an analyst. Using Kibana, you can do just that. In this section, you will build a Kibana dashboard for your data pipeline.

To open Kibana, browse to http://localhost:5601 and you will see the main window. At the bottom of the toolbar (on the left of the screen; you may need to expand it), click the management icon at the bottom. You need to select Create new Index Pattern and enter scf*, as shown in the following screenshot:

Figure 6.4 – Creating the index pattern in Kibana

When you click the next step, you will be asked to select a Time Filter field name. Because there are several fields with times in them, and they are in a format that is already recognizable by Elasticsearch, they will be indexed as such, and you can select a primary time filter. The field selected will be the default...

Summary

In this chapter, you learned how to build a data pipeline using data from a REST API. You also added a flow to the data pipeline to allow you to backfill the data, or to recreate a database with all of the data using a single pipeline.

The second half of the chapter provided a basic overview of how to build a dashboard using Kibana. Dashboards will usually be outside the responsibilities of a data engineer. In smaller firms, however, this could very well be your job. Furthermore, being able to quickly build a dashboard can help validate your data pipeline and look for any possible errors in the data.

In the next chapter, we begin a new section of this book, where you will take the skills you have learned and improve them by making your pipelines ready for production. You will learn about deployment, better validation techniques, and other skills needed when you are running pipelines in a production environment.