Reader small image

You're reading from  Data Observability for Data Engineering

Product typeBook
Published inDec 2023
PublisherPackt
ISBN-139781804616024
Edition1st Edition
Right arrow
Authors (2):
Michele Pinto
Michele Pinto
author image
Michele Pinto

Michele Pinto is the Head of Engineering at Kensu. With over 15 years of experience, Michele has a great knack for understanding how data observability and data engineering are closely linked. He started his career as a software engineer and has worked since then in various roles, such as big data engineer, big data architect, head of data and until recently he was a Head of Engineering. He has a great community presence and believes in giving back to the community. He has also been a teacher for Digital Product Management Master TAG Innovation School in Milan, Italy. His collaboration on the book has been prompt, swift, eager, and very invested.
Read more about Michele Pinto

Sammy El Khammal
Sammy El Khammal
author image
Sammy El Khammal

Sammy El Khammal works at Kensu. He started off as a field engineer and worked his way up to the position of product manager. In the past, he has also worked with Mercedes as their Business Development Analyst – Intern. He has also been an O'Reilly teacher for 3 workshops on data quality, lineage monitoring, and data observability. During that time, he provided some brilliant insights, very responsive behaviour, and immense talent and determination.
Read more about Sammy El Khammal

View More author details
Right arrow

Anomaly detection

In this section, you’ll learn how an anomaly can be detected based on the metrics you have gathered and can be the basis for rules starting from the SLIs.

Anomalies can be inferred from the following:

  • Simple indicator deterministic cases
  • Multiple indicators deterministic cases
  • Time series analysis

Let’s dig into these categories to explain how the observability metrics can be leveraged to find out the root cause and, in fine, new points of attention and rules for your data sources.

Simple indicator deterministic cases

Anomalies can be detected by adding a series of basic checks to the rules based on the type of metrics you gather, as well as the business logic.

By handling missing values effectively, organizations can prevent potential misinterpretations or errors in data analysis. For example, if the data producer or consumer expects no missing values in the data source, a deterministic rule addressing the number of...

lock icon
The rest of the page is locked
Previous PageNext Page
You have been reading a chapter from
Data Observability for Data Engineering
Published in: Dec 2023Publisher: PacktISBN-13: 9781804616024

Authors (2)

author image
Michele Pinto

Michele Pinto is the Head of Engineering at Kensu. With over 15 years of experience, Michele has a great knack for understanding how data observability and data engineering are closely linked. He started his career as a software engineer and has worked since then in various roles, such as big data engineer, big data architect, head of data and until recently he was a Head of Engineering. He has a great community presence and believes in giving back to the community. He has also been a teacher for Digital Product Management Master TAG Innovation School in Milan, Italy. His collaboration on the book has been prompt, swift, eager, and very invested.
Read more about Michele Pinto

author image
Sammy El Khammal

Sammy El Khammal works at Kensu. He started off as a field engineer and worked his way up to the position of product manager. In the past, he has also worked with Mercedes as their Business Development Analyst – Intern. He has also been an O'Reilly teacher for 3 workshops on data quality, lineage monitoring, and data observability. During that time, he provided some brilliant insights, very responsive behaviour, and immense talent and determination.
Read more about Sammy El Khammal