A typical need of any software application is to input/output data by reading/writing data files or data streams or by querying/manipulating a database. Regarding files and streams, unstructured data, or even binary data, is hard to manipulate, and so they are not recommended.
Also, proprietary data formats are not recommended because of the vendor lock-in risk, and so only standard data formats should be used. Fortunately, there are free Rust libraries that come to the rescue in these situations. There are Rust crates available to manipulate some of the most popular file formats, such as TOML, JSON, and XML.
In terms of databases, there are Rust crates to manipulate data using some of the most popular databases, such as SQLite, PostgreSQL, and Redis.
In this chapter, you will learn about the following:
- How to read configuration data from a TOML file...
Technical requirements
It is required for you to install the SQLite runtime library when you're running the SQLite code. However, it is also useful (although not required) to install a SQLite interactive manager. You can download the precompiled binaries of SQLite tools from https://www.sqlite.org/download.html. However, version 3.11 or higher would be ideal.
Please note that if you're using Debian-derived Linux distribution, the libsqlite3-dev package should be installed.
It is also required for you to install and run the PostgreSQL Database Management System (DBMS) when you're running the PostgreSQL code. As with SQLite, it is useful but not required to install a PostgreSQL interactive manager. You can download the precompiled binary of PostgreSQL DBMS from https://www.postgresql.org/download/. However, version 7.4 or higher would be acceptable.
Installing and running the Redis server is necessary when you're running the Redis code. You can download it from https...
Project overview
In this chapter, we'll look at how to build a program that loads a JSON file and an XML file into three databases: a SQLite database, a PostgreSQL database, and a Redis key-value store. To avoid hardwiring the names and positions of the files and the database credentials into the program, we are going to load them from a TOML configuration file.
The final project is named transformer, but we'll explain this through several preliminary small projects:
- toml_dynamic and toml_static: These read a TOML file in two different ways.
- json_dynamic and json_static: These read a JSON file in two different ways.
- xml_example: This reads an XML file.
- sqlite_example: This creates two tables in a SQLite database, inserts records into them, and queries them.
- postgresql_example: This creates two tables in a PostgreSQL database, inserts records into them, and queries them.
- redis_example: This adds some data to a key-value store and queries it.
Reading a TOML file
One simple and maintainable way to store information in a filesystem is to use a text file. This is also very efficient for data spanning no more than 100 KB. However, there are several competing standards for storing information in text files, such as INI, CSV, JSON, XML, YAML, and others.
The one used by Cargo is TOML. This is a really powerful format that is used by many Rust developers to store the configuration data of their apps. It is designed to be written by hand, using a text editor, but it can also be written by an application very easily.
The toml_dynamic and toml_static projects (using the toml crate) load data from a TOML file. Reading a TOML file is useful when configuring a software application, and this is what we'll do. We will use the data/config.toml file, which contains all of the parameters for the projects of this chapter.
You can also create or modify a TOML file by using code, but we are not going to do that. Being able to modify a TOML...
Reading and writing a JSON file
For storing data that is more complex than that which is stored in a configuration file, JSON format is more appropriate. This format is quite popular, particularly among those who use the JavaScript language.
We are going to read and parse the data/sales.json file. This file contains a single anonymous object, which contains two arrays—"products" and "sales".
The "products" array contains two objects, each one having three fields:
"products": [
{
"id": 591,
"category": "fruit",
"name": "orange"
},
{
"id": 190,
"category": "furniture",
"name": "chair"
}
],
The "sales" array contains three objects, each one containing five fields:
"sales": [
{
"id": "2020-7110",
"product_id": 190,
"date":...
Reading an XML file
Another very text format is XML. Unfortunately, there is no stable serialization/deserialization library to manage XML format. However, this is not necessarily a shortcoming. In actual fact, XML format is often used to store large datasets; so large, in fact, that it would be inefficient to load them all before we start converting the data into an internal format. In these cases, it may be more efficient to scan the file or incoming stream and process it as long as it is read.
The xml_example project is a rather convoluted program that scans the XML file specified on the command line and, in a procedural fashion, loads information from the file into a Rust data structure. It is meant to read the ../data/sales.xml file. This file has a structure corresponding to the JSON file we sought in the previous section. The following lines show an excerpt of that file:
<?xml version="1.0" encoding="utf-8"?>
<sales-and-products>
<product>...
Accessing databases
Text files are good when they are small and when they don't need to be changed often. Actually, the only way that a text file can be changed is if you append something to the end of it or rewrite it completely. If you want to change the information in a large dataset quickly, the only way to do so is to use a database manager. In this section, we are going to learn how to manipulate a SQLite database with a simple example.
But first, let's look at three popular, broad categories of database managers:
- Single-user databases: These store all of the databases in a single file, which must be accessible by the application code. The database code is linked into the application (it may be a static-link library or a dynamic-link library). Only one user at a time is allowed to access it, and all users have administrative privileges. To move the database anywhere, you simply move the file. The most popular choices in this category are SQLite and Microsoft Access.
- DBMS...
Accessing a SQLite database
The source code for this section is found in the sqlite_example project. To run it, open its folder and type in cargo run.
This will create the sales.db file in the current folder. This file contains a SQLite database. Then, it will create the Products and Sales tables in this database, it will insert a row into each of these tables, and it will perform a query on the database. The query asks for all the sales, joining each of them with its associated product. For each extracted row, a line will be printed onto the console, showing the timestamp of the sale, the weight of the sale, and the name of the associated product. As there is only one sale in the database, you will see just the following line printed:
At instant 1234567890, 7.439 Kg of pears were sold.
This project only uses the rusqlite crate. Its name is a contraction of Rust SQLite. To use this crate, the Cargo.toml file must contain the following line:
rusqlite = "0.23"
Implementing the...
Accessing a PostgreSQL database
What we did in the SQLite database is similar to what we will be doing in the PostgreSQL database. This is because they are both based on the SQL language, but mostly because SQLite is designed to be similar to PostgreSQL. It may be harder to convert an application from PostgreSQL into SQLite because the former has many advanced features that are not available in the latter.
In this section, we are going to convert the example from the previous section so that it works with a PostgreSQL database instead of SQLite. So, we'll explain the differences.
The source code for this section can be found in the postgresql_example folder. To run it, open its folder and type in cargo run. This will carry out essentially the same operations that we saw for sqlite_example, and so after creating and populating the database, it will print the following:
At instant 1234567890, 7.439 Kg of pears were sold.
Implementation of the project
This project only uses the crate...
Storing and retrieving data from a Redis store
Some applications need a very fast response time for certain kinds of data; faster than what a DBMS can offer. Usually, a DBMS dedicated to one user would be fast enough, but for some applications (typically large-scale web applications) there are hundreds of concurrent queries and many concurrent updates. You can use many computers, but the data must be kept coherent among them, and keeping coherence can cause a bottleneck of performance.
A solution to this problem is to use a key-value store, which is a very simple database that can be replicated across a network. This keeps the data in memory to maximize the speed, but it also supports the option to save the data in a file. This avoids losing information if the server is stopped.
A key-value store is similar to the HashMap collection of the Rust standard library, but it is managed by a server process, which could possibly be running on a different computer. A query is a message exchanged...
Putting it all together
You should now know enough to build an example that does what we described at the beginning of the chapter. We have learned the following:
- How to read a TOML file to parameterize the program
- How to load the data regarding products and sales into memory, specified in a JSON file and in an XML file
- How to store all of this data in three places: a SQLite DB file, a PostgreSQL database, and a Redis key-value store
The source code of the complete example is found in the transformer project. To run it, open its folder and type in cargo run ../data/config.toml. If everything is successful, it will recreate and populate the SQLite database contained in the data/sales.db file, the PostgreSQL database, which can be accessed from localhost on port 5432 and is named Rust2018, and the Redis store, which can be accessed from localhost. Then, it will query the SQLite and PostgreSQL databases for the number of rows in their tables, and it will print the following:
SQLite #Products...
Summary
In this chapter, we looked at some basic techniques to access data in popular text formats (TOML, JSON, and XML) or data managed by popular database managers (SQLite, PostgreSQL, and Redis). Of course, many other file formats and database managers exist, and there is still a lot to be learned about these formats and these database managers. Nevertheless, you should now have a grasp of what they do. These techniques are useful for many kinds of applications.
In the next chapter, we will learn how to build a web backend service using the REST architecture. To keep that chapter self-contained, we will only use a framework to receive and respond to web requests, and not use a database. Of course, that is quite unrealistic; but by combining those web techniques with the ones introduced in this chapter, you can build a real-world web service.
Questions
- Why is it not a good idea to change programmatically a TOML file edited by a user?
- When is it better to use a dynamically typed parsing of TOML or JSON files and when is it better to use statically typed parsing?
- When is it required to derive a structure from the Serialize and the Deserialize trait?
- What is a pretty generation of a JSON string?
- Why could it be better to use a stream parser, rather than a single-call parser?
- When is SQLite a better choice and when is it better to use PostgreSQL?
- Which is the type of the parameters passed with a SQL command to a SQLite database manager?
- What does the query method do on a PostgreSQL database?
- What are the names of the functions to read and write values in a Redis key-value store?
- Can you try to write a program that gets an ID from the command line, queries SQLite, PostgreSQL, or the Redis database for the ID, and prints some information regarding the data found?