Embedding Data Quality in Organizations

At the end of Chapter 8, we established how to go about calculating and publicizing the benefits of the work that’s done in the organization to remediate data. This chapter is about how to make those benefits sustainable in the long term. If the organization remediates the data as a “one-off exercise,” there will be benefits, but in the medium to long term, the data will return to a low state of quality.

Essentially, sustaining the benefits comes from two areas – firstly, making changes to the way that data is collected in the first place and secondly, continuing the activities outlined in Chapters 3 to 8, but on a smaller scale, in a business-as-usual context.

I will refer regularly to the term “business as usual” throughout this chapter. The term means the day-to-day operational work to keep an organization running smoothly, excluding all project and one-off activities. For example, one of the...

Preventing issue re-occurrence

In Chapter 8, we provided a table of the key governance activities that are required during the remediation phase. The last of these was the prevention of re-occurrence. This starts in the remediation phase but becomes a key activity as your initiative transitions from a project-style activity to a business-as-usual activity.

If remediation is completed as a one-off activity without understanding why the data quality issue arose in the first place, the issues will simply re-occur in the future. The remediation effort will eventually need to be repeated. It is possible to avoid this with a proper understanding of the cause, a change in systems or processes to resolve that cause, and then ongoing monitoring to ensure the quality remains sufficiently high.

One organization that I worked with used a Big Four consulting firm to complete and correct their supplier data. The work was completed on an entirely manual basis (from detecting the issues to remediation...

Ongoing data quality rule improvement

Once a data quality initiative is completed and a set of valuable rules are in place, it is critical to maintain these. It would be wonderful if the rules could remain consistent for a few years at least, but in my experience, this is never the case. Some rules may stay consistent for 10 years, while others will change within months (or even weeks!) of being initially established. The rules that stay consistent for longer are generally those that address long-standing legislative requirements such as taxation. Those that change more regularly are those most closely tied to how the business operates.

For example, product data tends to evolve quite quickly. If you consider an organization that makes technology products such as mobile phones, the level of change will be high. For example, in 2010, the network capabilities of a phone were limited to 2G and 3G and a rule might have checked that every handset had one of these values. In 2023, 4G and...

Transitioning to day-to-day remediation

Sometimes, organizations that have just completed their first data quality initiative lack the building blocks required to make remediation part of their employees’ day-to-day responsibilities. When the intensive project-based remediation ends, there is no mechanism to “pick up the baton” and continue. As described in Chapter 8, often, it is not practical to complete all the data correction required against a particular rule, so some proportion of the work remains. The hope is that the amount of work remaining is small enough (or of a reduced urgency) so that a business-as-usual team could manage it.

This section is about how the work should be transitioned from a project phase into business as usual and what mechanisms and building blocks must be set up to accommodate this. The starting point is to outline what is required for success and how this might be put in place.

Requirements for success

For a team to be...

Continuing the data quality journey

Chapter 2 showed that data quality work usually takes place through multiple iterations of a cyclical process. We have just described the final step in that process – the transition to business as usual.

The next step is to return to the beginning again (apologies to anyone who thought that they were finished!) and start to scope further data quality work in a new initiative.

This section describes how this can be approached.

Roadmap of data quality initiatives

A single data quality initiative (as described in Chapters 3 to 8) will include a range of different people – from project managers to data quality rule developers. These initiative-based resources will typically leave the organization or return to their original roles when the initiative ends. If there is only ever a need for one initiative, then this is fine, but if it is expected that many initiatives will be needed, then this adds additional cost to the next initiative...

Summary

In this chapter, we learned about what needs to happen with data quality after the intense effort of a budgeted data quality initiative. We learned what causes data quality issues to re-occur and how we can minimize that recurrence. We also learned about the need to keep up with business change and manage the baseline of rules effectively as time passes. Then, we learned about how to transition data quality remediation from a fully managed initiative-based process to an embedded activity in a business as usual team. Finally, we learned how to transition from a single initiative into a longer-term roadmap of activity that fully transforms the data quality of your organization.

We’ve now been through the entire data quality improvement cycle that we outlined in Chapter 2. In the final chapter, we will highlight the key best practices and the most commonly made mistakes in data quality work before finishing this book by looking at how innovation might change the field...