Monitoring our data ingest file size
When ingesting data, we can track a few items to ensure the incoming information is what we expect. One of the most important of these items is the data size we are ingesting, which can mean file size or the size of chunks of streaming data.
Logging the size of incoming data allows the creation of intelligent and efficient monitoring. If at some point the size of incoming data diverges from what is expected, we can take action to investigate and resolve the issue.
In this recipe, we will create simple Python code that logs the size of ingested files, which is very valuable in data monitoring.
Getting ready
We will use only Python code. Make sure you have Python version 3.7 or above. You can use the following command on your CLI to check your version:
$ python3 –-version Python 3.8.10
The following code execution can be done using a Python shell or a Jupyter notebook.
How to do it…
This exercise will create...
