Storing Data Collected from Sensors

In the previous two chapters, we learned how to collect data from various sensors. However, the best we have done is display the information on a screen. It’s a waste to collect all of that data and not store it. Let’s look at how we can store the data that we are collecting. By the end of this chapter, you will know how to store data in files and various file formats that are in common use, set up a database, and write queries to read data out of a database. The data that you store will be useful for analysis in the future, usually by data analysts. For example, if you collect and store temperature and humidity data from different weather stations over a certain period, then that data can be used to analyze weather patterns.

In this chapter, we are going to cover the following main topics:

• Storing data
• Working with flat files
• Working with databases

We’ll start by listing what you will need to complete...

Technical requirements

You will require the following to complete this chapter:

• The Arduino IDE
• Arduino MKR WiFi 1010
• A micro-USB cable
• Arduino MKR ENV Shield or MKR IoT Carrier
• A micro-SD card
• An SD card reader
• A Raspberry Pi 3 or 4, or a virtual machine

If you have got everything you need, then let’s proceed to collect and store some data.

The source code for this chapter can be found in this book’s GitHub repository: https://github.com/securetorobert/Arduino-Data-Communications/tree/main/chapter-4/.

Storing data

The easiest way to store data is offline, where a connection to the internet or a local network is not required. Where a small amount of data is concerned, this might be done on the memory of the device, the Electronic Erasable Programmable Read-Only Memory (EEPROM). However, we will be dealing with a lot of data, so we need to store it somewhere with a reasonable amount of space. Also, while some microcontrollers have an EEPROM, the MKR board doesn’t.

MKR boards do not have expandable storage but some shields do. The MKR ENV Shield is one of them. We will attach a micro-SD card to the MKR ENV shield and store data on it.

We store data in files. Each file has a format. One quick way of knowing what format a file is stored in is by looking at the name of the file and the extension.

In the following project, we will create a sketch that will write temperature and humidity readings to a file. To make it fun and easy to read the file, we will use the TSV format...

Understanding flat files

A flat file contains all of the information in one file. We saw this example of how all of the temperature and humidity information is stored in one file. Let’s consider some formats for storing data in a flat file.

Getting to know the TXT file format

One common file format is TXT. This format represents a text file. This simply means that the file is not specially formatted, and you can read it using a bunch of tools and editors. If you are on Microsoft Windows, you can read this type of file using an application called Notepad. On Mac, you can use TextEdit. On Linux, you can use Nano, Vim, Emacs, and a host of other editors.

This file can hold any body of text without any special formatting.

Getting to know the CSV file format

Another common file format is Comma-Separated Values (CSV). It is used to store information in which the different values are separated by a comma and are all stored on one line. Every line is terminated with a...

Working with databases

You can use a database to keep information organized. Databases are frequently backed by an engine called the database server. How a Database server does its job is not as important as what a database server empowers you to do.

While we have been able to store data on a file, we need to ask ourselves, “What happens if the file gets corrupted? How do we provide backup for this data?” database servers can provide redundancy, replication, and high availability for our data when we need it.

You can use a database server to store and retrieve information when you need to. When you retrieve information from a database, you specify what you want and not how to get it. This is different from how we have done things so far, where we have to program the microcontroller to do exactly what we want the way we want it. When we tell a microcontroller how to do something, that style of programming is called imperative. When we let the database determine how...

Summary

In this chapter, you learned about different approaches and formats for storing data and learned how to store data in a flat file on the MKR board. You also learned about databases and set up two different DBMSs: an RDBMS called MySQL and a time series DBMS called InfluxDB. The skills you have learned will come in handy when you set up databases that you can use to store data.

In Chapter 5, you will learn how to move data into and out of the DMBSs that you’ve configured.

Further reading

You can learn about a library for working with SQLite on an Arduino at https://github.com/siara-cc/sqlite_micro_logger_arduino.