Automating with Shell Scripts

In this chapter, we’re going to demonstrate system administration task automation with shell scripts. We are going to illustrate several ways of handling scripting using Bash shell. The plan is to create a script that will automate the creation of the database dump. This task, while easy, will demonstrate how things can go sideways and how to handle those situations.

In this chapter, we will cover the following topics:

• Backing up a database
• Understanding scripting
• Understanding Bash built-ins and grammar
• Understanding the backup script – first steps
• Handling errors and debugging

Technical requirements

For this chapter, you will require a Linux system where you can install packages and are not afraid to break things in the process. To that end, a virtual machine would be most preferred, typically running on an old computer that you can reinstall from scratch. We do not expect to break anything, but during learning, this may happen.

Backing up a database

In terms of most common databases, such as MySQL and PostgreSQL, there are at least two different ways to back up a database:

• Take a database dump by extracting all current data, along with the database schema
• Copy replication logs

In cloud environments, you can also take a snapshot of the disk database where the backup is being saved.

A database dump can also be used as a full backup. Replication logs aren’t self-sufficient database dumps, so you will need to combine them with a full backup. This is called an incremental backup.

Doing a full backup can take a long time, especially for big databases. While it’s running, the database puts a lock on its data files, so it doesn’t save new data on the disk; instead, it stores everything in the replication logs until the database lock is released. For large databases, this operation can take hours. Because of that, we will be creating a full backup once a week and copying...

Understanding scripting

A shell script is a simple text file filled with commands. Unlike compiled programs, shell scripts are not evaluated before execution, but rather while they are being executed. This makes for a very quick development process – there’s no compilation at all. But at the same time, the execution is a bit slower. Also, the errors that the compiler would have caught surface during execution and can often lead to script exiting.

On the upside, there’s not much to learn when you are writing a script – much less than when you are writing a program in C or Python. Interacting with system commands is as simple as just typing their names.

Bash lacks a lot of sophistication in programming languages: there are not many data types and structures, there’s very rudimentary control of scope, and the memory implementation is not meant to be efficient with scale.

There’s not one good rule of thumb for choosing when to write a...

Understanding Bash built-ins and grammar

Let’s get back to the fundamentals before we start creating a script. First, we will look into the Bash scripting language syntax and its limitations.

Built-in commands are commands that are integral to Bash and are the main scripting syntax we are going to use. Bash will try to execute any other commands from the system it runs on.

Just like any other interpreted language (for example, Python or Ruby), Bash has unique syntax and grammar. Let’s look into it.

Bash, similar to other programming languages, interprets files from top to bottom and from left to right. Each line usually contains one or more commands. You can glue several commands together in one line using a pipe (|) or double pipe character (||), semicolon (;), or double ampersands (&&). It’s useful to remember that double pipes have the same function as logical OR and double ampersands have the same function as logical AND. This way, you can...

Understanding the backup script – first steps

Now that we know what a script can look like, we can start writing one. You can use your favorite console editor or IDE to do this. Let’s create an empty file named run_backups.sh and change its permissions so that they’re executable:

admin@myhome:~$ touch run_backups.sh && chmod +x run_backups.sh
admin@myhome:~$ ls -l run_backups.sh
-rwxr-xr-x  1 admin  admin  0 Dec  1 15:56 run_backups.sh

It’s an empty file, so we’ll need to add a basic database backup command and proceed from there. We won’t be covering granting this script access to a database. We will be backing up a PostgreSQL database and using the pg_dump tool for that purpose.

Let’s input a shebang line and a pg_dump command call in our base script:

#!/usr/bin/env bash
pg_dump mydatabase > mydatabase.sql

To execute this script, we’ll need to start the following...

Handling errors and debugging

While running our backup script, we can encounter several errors: access to the database might be blocked, the pg_dump process might get killed, we may be out of disk space, or any other error preventing us from completing a full database dump.

In any of those cases, we will need to catch the error and handle it gracefully.

Additionally, we might want to refactor the script to make it configurable, make use of functions, and debug the script. Debugging will prove very useful, especially when dealing with larger scripts.

Let’s dive right into it and start with adding a function:

#!/usr/bin/env bash
function run_dump() {
  database_name=$1
  pg_dump -U postgres $database_name > $database_name.sql
}
run_dump mydatabase

We’ve added a run_dump function that takes one argument and sets a local variable called database_name with the content of this argument. It then uses this local variable to pass options to...

Summary

Shell scripting is a very common way of automating periodically running tasks in a Linux system. Sometimes, it evolves to a bigger system chained together with multiple Bash scripts and Python programs to complete complex tasks using multiple smaller tasks that do one thing at the same time in a very reliable way.

In modern systems, you will probably see as much Bash as Python scripts.

In this chapter, we learned how to create an executable script, as well as how to create a simple backup script that handles errors and generates a new filename on each run. We also added a function that removes old backups so that we can avoid filling the disk space. Additionally, as a side effect, we learned how to create a new PostgreSQL database and allow access to it from a local system.

In the next chapter, we’ll learn how to create Linux services and how to manage them.

Exercises

Try out the following exercises to test your knowledge of this chapter:

1. Write a function that will list all databases and feed that list to the for loop we’ve created.
2. Change a date timestamp into another format of your choosing.
3. Catch any errors the find function could return (for example, it couldn’t delete a file).