Home Cloud & Networking Linux Shell Scripting Cookbook, Second Edition - Second Edition

Linux Shell Scripting Cookbook, Second Edition - Second Edition

books-svg-icon Book
eBook $28.99 $19.99
Print $48.99
Subscription $15.99 $10 p/m for three months
$10 p/m for first 3 months. $15.99 p/m after that. Cancel Anytime!
What do you get with a Packt Subscription?
This book & 7000+ ebooks & video courses on 1000+ technologies
60+ curated reading lists for various learning paths
50+ new titles added every month on new and emerging tech
Early Access to eBooks as they are being written
Personalised content suggestions
Customised display settings for better reading experience
50+ new titles added every month on new and emerging tech
Playlists, Notes and Bookmarks to easily manage your learning
Mobile App with offline access
What do you get with a Packt Subscription?
This book & 6500+ ebooks & video courses on 1000+ technologies
60+ curated reading lists for various learning paths
50+ new titles added every month on new and emerging tech
Early Access to eBooks as they are being written
Personalised content suggestions
Customised display settings for better reading experience
50+ new titles added every month on new and emerging tech
Playlists, Notes and Bookmarks to easily manage your learning
Mobile App with offline access
What do you get with eBook + Subscription?
Download this book in EPUB and PDF formats, plus a monthly download credit
This book & 6500+ ebooks & video courses on 1000+ technologies
60+ curated reading lists for various learning paths
50+ new titles added every month on new and emerging tech
Early Access to eBooks as they are being written
Personalised content suggestions
Customised display settings for better reading experience
50+ new titles added every month on new and emerging tech
Playlists, Notes and Bookmarks to easily manage your learning
Mobile App with offline access
What do you get with a Packt Subscription?
This book & 6500+ ebooks & video courses on 1000+ technologies
60+ curated reading lists for various learning paths
50+ new titles added every month on new and emerging tech
Early Access to eBooks as they are being written
Personalised content suggestions
Customised display settings for better reading experience
50+ new titles added every month on new and emerging tech
Playlists, Notes and Bookmarks to easily manage your learning
Mobile App with offline access
What do you get with eBook?
Download this book in EPUB and PDF formats
Access this title in our online reader
DRM FREE - Read whenever, wherever and however you want
Online reader with customised display settings for better reading experience
What do you get with video?
Download this video in MP4 format
Access this title in our online reader
DRM FREE - Watch whenever, wherever and however you want
Online reader with customised display settings for better learning experience
What do you get with video?
Stream this video
Access this title in our online reader
DRM FREE - Watch whenever, wherever and however you want
Online reader with customised display settings for better learning experience
What do you get with Audiobook?
Download a zip folder consisting of audio files (in MP3 Format) along with supplementary PDF
What do you get with Exam Trainer?
Flashcards, Mock exams, Exam Tips, Practice Questions
Access these resources with our interactive certification platform
Mobile compatible-Practice whenever, wherever, however you want
BUY NOW $10 p/m for first 3 months. $15.99 p/m after that. Cancel Anytime!
eBook $28.99 $19.99
Print $48.99
Subscription $15.99 $10 p/m for three months
What do you get with a Packt Subscription?
This book & 7000+ ebooks & video courses on 1000+ technologies
60+ curated reading lists for various learning paths
50+ new titles added every month on new and emerging tech
Early Access to eBooks as they are being written
Personalised content suggestions
Customised display settings for better reading experience
50+ new titles added every month on new and emerging tech
Playlists, Notes and Bookmarks to easily manage your learning
Mobile App with offline access
What do you get with a Packt Subscription?
This book & 6500+ ebooks & video courses on 1000+ technologies
60+ curated reading lists for various learning paths
50+ new titles added every month on new and emerging tech
Early Access to eBooks as they are being written
Personalised content suggestions
Customised display settings for better reading experience
50+ new titles added every month on new and emerging tech
Playlists, Notes and Bookmarks to easily manage your learning
Mobile App with offline access
What do you get with eBook + Subscription?
Download this book in EPUB and PDF formats, plus a monthly download credit
This book & 6500+ ebooks & video courses on 1000+ technologies
60+ curated reading lists for various learning paths
50+ new titles added every month on new and emerging tech
Early Access to eBooks as they are being written
Personalised content suggestions
Customised display settings for better reading experience
50+ new titles added every month on new and emerging tech
Playlists, Notes and Bookmarks to easily manage your learning
Mobile App with offline access
What do you get with a Packt Subscription?
This book & 6500+ ebooks & video courses on 1000+ technologies
60+ curated reading lists for various learning paths
50+ new titles added every month on new and emerging tech
Early Access to eBooks as they are being written
Personalised content suggestions
Customised display settings for better reading experience
50+ new titles added every month on new and emerging tech
Playlists, Notes and Bookmarks to easily manage your learning
Mobile App with offline access
What do you get with eBook?
Download this book in EPUB and PDF formats
Access this title in our online reader
DRM FREE - Read whenever, wherever and however you want
Online reader with customised display settings for better reading experience
What do you get with video?
Download this video in MP4 format
Access this title in our online reader
DRM FREE - Watch whenever, wherever and however you want
Online reader with customised display settings for better learning experience
What do you get with video?
Stream this video
Access this title in our online reader
DRM FREE - Watch whenever, wherever and however you want
Online reader with customised display settings for better learning experience
What do you get with Audiobook?
Download a zip folder consisting of audio files (in MP3 Format) along with supplementary PDF
What do you get with Exam Trainer?
Flashcards, Mock exams, Exam Tips, Practice Questions
Access these resources with our interactive certification platform
Mobile compatible-Practice whenever, wherever, however you want
  1. Free Chapter
    Shell Something Out
About this book
The shell remains one of the most powerful tools on a computer system — yet a large number of users are unaware of how much one can accomplish with it. Using a combination of simple commands, we will see how to solve complex problems in day to day computer usage.Linux Shell Scripting Cookbook, Second Edition will take you through useful real-world recipes designed to make your daily life easy when working with the shell. The book shows the reader how to effectively use the shell to accomplish complex tasks with ease.The book discusses basics of using the shell, general commands and proceeds to show the reader how to use them to perform complex tasks with ease.Starting with the basics of the shell, we will learn simple commands with their usages allowing us to perform operations on files of different kind. The book then proceeds to explain text processing, web interaction and concludes with backups, monitoring and other sysadmin tasks.Linux Shell Scripting Cookbook, Second Edition serves as an excellent guide to solving day to day problems using the shell and few powerful commands together to create solutions.
Publication date:
May 2013
Publisher
Packt
Pages
384
ISBN
9781782162742

 

Chapter 1. Shell Something Out

In this chapter, we will cover:

  • Printing in the terminal

  • Playing with variables and environment variables

  • Function to prepend to environment variables

  • Math with the shell

  • Playing with file descriptors and redirection

  • Arrays and associative array

  • Visiting aliases

  • Grabbing information about the terminal

  • Getting and setting dates and delays

  • Debugging the script

  • Functions and arguments

  • Reading output of a sequence of commands in a variable

  • Reading n characters without pressing the return key

  • Running a command until it succeeds

  • Field separators and iterators

  • Comparisons and tests

 

Introduction


Unix-like systems are amazing operating system designs. Even after many decades, Unix-style architecture for operating systems serves as one of the best designs. One of the important features of this architecture is the command-line interface, or the shell. The shell environment helps users to interact with and access core functions of the operating system. The term scripting is more relevant in this context. Scripting is usually supported by interpreter-based programming languages. Shell scripts are files in which we write a sequence of commands that we need to perform and are executed using the shell utility.

In this book we are dealing with Bash (Bourne Again Shell), which is the default shell environment for most GNU/Linux systems. Since GNU/Linux is the most prominent operating system on Unix-style architecture, most of the examples and discussions are written by keeping Linux systems in mind.

The primary purpose of this chapter is to give readers an insight into the shell environment and become familiar with the basic features that the shell offers. Commands are typed and executed in a shell terminal. When a terminal is opened, a prompt is available which usually has the following format:

username@hostname$

Or:

root@hostname #

or simply as $ or #.

$ represents regular users and # represents the administrative user root. Root is the most privileged user in a Linux system.

Note

It is usually a bad idea to directly use the shell as the root user (administrator) to perform tasks. This is because typing errors in your commands have the potential to do more damage when your shell has more privileges. So, it is recommended to log in as a regular user (your shell will denote that as $ in the prompt, and # when running as root), and then use tools such as `sudo' to run privileged commands. Running a command such as sudo <command> <arguments> will run it as root.

A shell script is a text file that typically begins with a shebang, as follows:

#!/bin/bash

Shebang is a line on which #! is prefixed to the interpreter path. /bin/bash is the interpreter command path for Bash.

Execution of a script can be done in two ways. Either we can run the script as a command-line argument to bash or we can grant execution permission to the script so it becomes executable.

The script can be run with the filename as a command-line argument as follows (the text that starts with # is a comment, you don't have to type it out):

$ bash script.sh # Assuming script is in the current directory.

Or:

$ bash /home/path/script.sh # Using full path of script.sh.

If a script is run as a command-line argument for bash, the shebang in the script is not required.

If required, we can utilize the shebang to facilitate running the script on its own. For this, we have to set executable permissions for the script and it will run using the interpreter path that is appended to #! to the shebang. This can be set as follows:

$ chmod a+x script.sh

This command gives the script.sh file the executable permission for all users. The script can be executed as:

$ ./script.sh #./ represents the current directory

Or:

$ /home/path/script.sh # Full path of the script is used

The kernel will read the first line and see that the shebang is #!/bin/bash. It will identify /bin/bash and execute the script internally as:

$ /bin/bash script.sh

When a shell is started, it initially executes a set of commands to define various settings such as prompt text, colors, and much more. This set of commands are read from a shell script at ~/.bashrc (or ~/.bash_profile for login shells) located in the home directory of the user. The Bash shell also maintains a history of commands run by the user. It is available in the ~/.bash_history file.

Note

~ denotes your home directory, which is usually /home/user where user is your username or /root for the root user.

A login shell is the shell which you get just after logging in to a machine. However, if you open up a shell while logged in to a graphical environment (such as GNOME, KDE, and so on), then it is not a login shell.

In Bash, each command or command sequence is delimited by using a semicolon or a new line. For example:

$ cmd1 ; cmd2

This is equivalent to:

$ cmd1
$ cmd2

Finally, the # character is used to denote the beginning of unprocessed comments. A comment section starts with # and proceeds up to the end of that line. The comment lines are most often used to provide comments about the code in the file or to stop a line of code from being executed.

Now let us move on to the basic recipes in this chapter.

 

Printing in the terminal


The terminal is an interactive utility by which a user interacts with the shell environment. Printing text in the terminal is a basic task that most shell scripts and utilities need to perform regularly. As we will see in this recipe, this can be performed via various methods and in different formats.

How to do it...

echo is the basic command for printing in the terminal.

echo puts a newline at the end of every echo invocation by default:

$ echo "Welcome to Bash"
Welcome to Bash

Simply, using double-quoted text with the echo command prints the text in the terminal. Similarly, text without double quotes also gives the same output:

$ echo Welcome to Bash
Welcome to Bash

Another way to do the same task is by using single quotes:

$ echo 'text in quotes'

These methods may look similar, but some of them have a specific purpose and side effects too. Consider the following command:

$ echo "cannot include exclamation - ! within double quotes"

This will return the following output:

bash: !: event not found error

Hence, if you want to print special characters such as !, either do not use them within double quotes or escape them with a special escape character (\) prefixed with it, like so:

$ echo Hello world !

Or:

$ echo 'Hello world !'

Or:

$ echo "Hello world \!" #Escape character \ prefixed.

The side effects of each of the methods are as follows:

  • When using echo without quotes, we cannot use a semicolon, as it acts as a delimiter between commands in the Bash shell

  • echo hello; hello takes echo hello as one command and the second hello as the second command

  • Variable substitution, which is discussed in the next recipe, will not work within single quotes

Another command for printing in the terminal is printf. It uses the same arguments as the printf command in the C programming language. For example:

$ printf "Hello world"

printf takes quoted text or arguments delimited by spaces. We can use formatted strings with printf. We can specify string width, left or right alignment, and so on. By default, printf does not have newline as in the echo command. We have to specify a newline when required, as shown in the following script:

#!/bin/bash 
#Filename: printf.sh

printf  "%-5s %-10s %-4s\n" No Name  Mark 
printf  "%-5s %-10s %-4.2f\n" 1 Sarath 80.3456 
printf  "%-5s %-10s %-4.2f\n" 2 James 90.9989 
printf  "%-5s %-10s %-4.2f\n" 3 Jeff 77.564

We will receive the formatted output:

No    Name       Mark
1     Sarath     80.35
2     James      91.00
3     Jeff       77.56

How it works...

%s, %c, %d, and %f are format substitution characters for which an argument can be placed after the quoted format string.

%-5s can be described as a string substitution with left alignment (- represents left alignment) with width equal to 5. If - was not specified, the string would have been aligned to the right. The width specifies the number of characters reserved for that variable. For Name, the width reserved is 10. Hence, any name will reside within the 10-character width reserved for it and the rest of the characters will be filled with space up to 10 characters in total.

For floating point numbers, we can pass additional parameters to round off the decimal places.

For marks, we have formatted the string as %-4.2f, where .2 specifies rounding off to two decimal places. Note that for every line of the format string a newline (\n) is issued.

There's more...

While using flags for echo and printf, always make sure that the flags appear before any strings in the command, otherwise Bash will consider the flags as another string.

Escaping newline in echo

By default, echo has a newline appended at the end of its output text. This can be avoided by using the -n flag. echo can also accept escape sequences in double-quoted strings as an argument. When using escape sequences, use echo as echo -e "string containing escape sequences". For example:

echo -e "1\t2\t3"
1  2  3

Printing a colored output

Producing a colored output on the terminal is very interesting and is achieved by using escape sequences.

Colors are represented by color codes, some examples being, reset = 0, black = 30, red = 31, green = 32, yellow = 33, blue = 34, magenta = 35, cyan = 36, and white = 37.

To print a colored text, enter the following command:

echo -e "\e[1;31m This is red text \e[0m"

Here, \e[1;31m is the escape string that sets the color to red and \e[0m resets the color back. Replace 31 with the required color code.

For a colored background, reset = 0, black = 40, red = 41, green = 42, yellow = 43, blue = 44, magenta = 45, cyan = 46, and white=47, are the color codes that are commonly used.

To print a colored background, enter the following command:

echo -e "\e[1;42m Green Background \e[0m"
 

Playing with variables and environment variables


Variables are essential components of every programming language and are used to hold varying data. Scripting languages usually do not require variable type declaration before its use as they can be assigned directly. In Bash, the value for every variable is string, regardless of whether we assign variables with quotes or without quotes. Furthermore, there are variables used by the shell environment and the operating environment to store special values, which are called environment variables. Let us look at how to play with some of these variables in this recipe.

Getting ready

Variables are named with the usual naming constructs. When an application is executing, it will be passed a set of variables called environment variables. To view all the environment variables related to a terminal, issue the env command. For every process, environment variables in its runtime can be viewed by:

cat /proc/$PID/environ

Set PID with a process ID of the process (PID always takes an integer value).

For example, assume that an application called gedit is running. We can obtain the process ID of gedit with the pgrep command as follows:

$ pgrep gedit
12501

You can obtain the environment variables associated with the process by executing the following command:

$ cat /proc/12501/environ
GDM_KEYBOARD_LAYOUT=usGNOME_KEYRING_PID=1560USER=slynuxHOME=/home/slynux

Note

Note that many environment variables are stripped off for convenience. The actual output may contain numerous variables.

The aforementioned command returns a list of environment variables and their values. Each variable is represented as a name=value pair and are separated by a null character (\0). If you can substitute the \0 character with \n, you can reformat the output to show each variable=value pair in each line. Substitution can be made using the tr command as follows:

$ cat /proc/12501/environ  | tr '\0' '\n'

Now, let us see how to assign and manipulate variables and environment variables.

How to do it...

A variable can be assigned as follows:

var=value

var is the name of a variable and value is the value to be assigned. If value does not contain any space character (such as space), it need not be enclosed in quotes, Otherwise it is to be enclosed in single or double quotes.

Note that var = value and var=value are different. It is a usual mistake to write var =value instead of var=value. The later one is the assignment operation, whereas the earlier one is an equality operation.

Printing contents of a variable is done using by prefixing $ with the variable name as follows:

var="value" #Assignment of value to variable var.

echo $var

Or:

echo ${var}

We will receive an output as follows:

value

We can use variable values inside printf or echo in double quotes:

#!/bin/bash
#Filename :variables.sh
fruit=apple
count=5
echo "We have $count ${fruit}(s)"

The output will be as follows:

We have 5 apple(s)

Environment variables are variables that are not defined in the current process, but are received from the parent processes. For example, HTTP_PROXY is an environment variable. This variable defines which proxy server should be used for an Internet connection.

Usually, it is set as:

HTTP_PROXY=192.168.1.23:3128
export HTTP_PROXY

The export command is used to set the env variable. Now any application, executed from the current shell script, will receive this variable. We can export custom variables for our own purposes in an application or shell script that is executed. There are many standard environment variables that are available for the shell by default.

For example, PATH. A typical PATH variable will contain:

$ echo $PATH
/home/slynux/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games

When given a command for execution, the shell automatically searches for the executable in the list of directories in the PATH environment variable (directory paths are delimited by the ":" character). Usually, $PATH is defined in /etc/environment or /etc/profile or ~/.bashrc. When we need to add a new path to the PATH environment, we use:

export PATH="$PATH:/home/user/bin"

Or, alternately, we can use:

$ PATH="$PATH:/home/user/bin"
$ export PATH

$ echo $PATH
/home/slynux/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/home/user/bin

Here we have added /home/user/bin to PATH.

Some of the well-known environment variables are HOME, PWD, USER, UID, SHELL, and so on.

Note

When using single quotes, variables will not be expanded and will be displayed as is. This means:

$ echo '$var' will print $var

Whereas, $ echo "$var" will print the value of the $var variable if defined or nothing at all if it is not defined.

There's more...

Let us see more tips associated with standard and environment variables.

Finding the length of a string

Get the length of a variable value using the following command:

length=${#var}

For example:

$ var=12345678901234567890$
echo ${#var}
20

The length parameter will bear the number of characters in the string.

Identifying the current shell

To identify the shell which is currently being used, we can use the SHELL variable, like so:

echo $SHELL

Or:

echo $0

For example:

$ echo $SHELL
/bin/bash

$ echo $0
/bin/bash

Checking for super user

UID is an important environment variable that can be used to check whether the current script has been run as a root user or regular user. For example:

If [ $UID -ne 0 ]; then
    echo Non root user. Please run as root.
else
    echo Root user
fi

The UID value for the root user is 0.

Modifying the Bash prompt string (username@hostname:~$)

When we open a terminal or run a shell, we see a prompt string such as user@hostname: /home/$. Different GNU/Linux distributions have slightly different prompts and different colors. We can customize the prompt text using the PS1 environment variable. The default prompt text for the shell is set using a line in the ~/.bashrc file.

  • We can list the line used to set the PS1 variable as follows:

    $ cat ~/.bashrc | grep PS1
    PS1='${debian_chroot:+($debian_chroot)}\u@\h:\w\$ '
    
  • To set a custom prompt string, enter the following command:

    slynux@localhost: ~$ PS1="PROMPT>"
    PROMPT> Type commands here # Prompt string changed.
    
  • We can use colored text using the special escape sequences such as \e[1;31 (refer to the Printing in the terminal recipe of this chapter).

There are also certain special characters that expand to system parameters. For example, \u expands to username, \h expands to hostname, and \w expands to the current working directory.

 

Function to prepend to environment variables


Environment variables are often used to store a list of paths of where to search for executables, libraries, and so on. Examples are $PATH, $LD_LIBRARY_PATH, which will typically look like this:

PATH=/usr/bin;/bin
LD_LIBRARY_PATH=/usr/lib;/lib

This essentially means that whenever the shell has to execute binaries, it will first look into /usr/bin followed by /bin.

A very common task that one has to do when building a program from source and installing to a custom path is to add its bin directory to the PATH environment variable. Let's say in this case we install myapp to /opt/myapp, which has binaries in a directory called bin and libraries in lib.

How to do it...

A way to do this is to say it as follows:

export PATH=/opt/myapp/bin:$PATH
export LD_LIBRARY_PATH=/opt/myapp/lib;$LD_LIBRARY_PATH

PATH and LD_LIBRARY_PATH should now look something like this:

PATH=/opt/myapp/bin:/usr/bin:/bin
LD_LIBRARY_PATH=/opt/myapp/lib:/usr/lib;/lib

However, we can make this easier by adding this function in .bashrc-:

prepend() { [ -d "$2" ] && eval $1=\"$2':'\$$1\" && export $1; }

This can be used in the following way:

prepend PATH /opt/myapp/bin
prepend LD_LIBRARY_PATH /opt/myapp/lib

How it works...

We define a function called prepend(), which first checks if the directory specified by the second parameter to the function exists. If it does, the eval expression sets the variable with the name in the first parameter equal to the second parameter string followed by : (the path separator) and then the original value for the variable.

However, there is one caveat, if the variable is empty when we try to prepend, there will be a trailing : at the end. To fix this, we can modify the function to look like this:

prepend() { [ -d "$2" ] && eval $1=\"$2\$\{$1:+':'\$$1\}\" && export $1 ; }

Note

In this form of the function, we introduce a shell parameter expansion of the form:

${parameter:+expression}

This expands to expression if parameter is set and is not null.

With this change, we take care to try to append : and the old value if, and only if, the old value existed when trying to prepend.

 

Math with the shell


Arithmetic operations are an essential requirement for every programming language. In this recipe, we will explore various methods for performing arithmetic operations in shell.

Getting ready

The Bash shell environment can perform basic arithmetic operations using the commands let, (( )), and []. The two utilities expr and bc are also very helpful in performing advanced operations.

How to do it...

  1. A numeric value can be assigned as a regular variable assignment, which is stored as a string. However, we use methods to manipulate as numbers:

    #!/bin/bash
    no1=4;
    no2=5;
    
  2. The let command can be used to perform basic operations directly. While using let, we use variable names without the $ prefix, for example:

    let result=no1+no2
    echo $result
    
    • Increment operation:

      	$ let no1++
      
    • Decrement operation:

      	$ let no1--
      
    • Shorthands:

      	let no+=6
      	let no-=6
      

    These are equal to let no=no+6 and let no=no-6 respectively.

    • Alternate methods:

      The [] operator can be used in the same way as the let command as follows:

      	result=$[ no1 + no2 ]
      

      Using the $ prefix inside [] operators are legal, for example:

      	result=$[ $no1 + 5 ]
      

      (( )) can also be used. $ prefixed with a variable name is used when (( )) operator is used, as follows:

      	result=$(( no1 + 50 ))
      

      expr can also be used for basic operations:

      	result=`expr 3 + 4`
      	result=$(expr $no1 + 5)
      

      All of the preceding methods do not support floating point numbers, and operate on integers only.

  3. bc , the precision calculator is an advanced utility for mathematical operations. It has a wide range of options. We can perform floating point operations and use advanced functions as follows:

    echo "4 * 0.56" | bc
    2.24
    
    no=54; 
    result=`echo "$no * 1.5" | bc`
    echo $result
    81.0
    

    Additional parameters can be passed to bc with prefixes to the operation with semicolon as delimiters through stdin.

    • Decimal places scale with bc: In the following example the scale=2 parameter sets the number of decimal places to 2. Hence, the output of bc will contain a number with two decimal places:

      	echo "scale=2;3/8" | bc
      	0.37
      
    • Base conversion with bc: We can convert from one base number system to another one. Let us convert from decimal to binary, and binary to octal:

      	#!/bin/bash
      	Desc: Number conversion
      	
      	no=100
      	echo "obase=2;$no" | bc
      	1100100
      	no=1100100
      	echo "obase=10;ibase=2;$no" | bc
      	100
      
    • Calculating squares and square roots can be done as follows:

      	echo "sqrt(100)" | bc #Square root
      	echo "10^10" | bc #Square
      
 

Playing with file descriptors and redirection


File descriptors are integers that are associated with file input and output. They keep track of opened files. The best-known file descriptors are stdin, stdout, and stderr. We even can redirect the contents of one file descriptor to another. This recipe shows examples on how to manipulate and redirect with file descriptors.

Getting ready

While writing scripts we use standard input (stdin), standard output (stdout), and standard error (stderr) frequently. Redirection of an output to a file by filtering the contents is one of the essential things we need to perform. While a command outputs some text, it can be either an error or an output (nonerror) message. We cannot distinguish whether it is output text or an error text by just looking at it. However, we can handle them with file descriptors. We can extract text that is attached to a specific descriptor.

File descriptors are integers associated with an opened file or data stream. File descriptors 0, 1, and 2 are reserved as follows:

  • 0: stdin (standard input)

  • 1: stdout (standard output)

  • 2: stderr (standard error)

How to do it...

  1. Redirecting or saving output text to a file can be done as follows:

    $ echo "This is a sample text 1" > temp.txt
    

    This would store the echoed text in temp.txt by truncating the file, the contents will be emptied before writing.

  2. To append text to a file, consider the following example:

    $ echo "This is sample text 2" >> temp.txt
    
  3. You can view the contents of the file as follows:

    $ cat temp.txt
    This is sample text 1
    This is sample text 2
    
  4. Let us see what a standard error is and how you can redirect it. stderr messages are printed when commands output an error message. Consider the following example:

    $ ls +
    ls: cannot access +: No such file or directory
    

    Here + is an invalid argument and hence an error is returned.

    Tip

    Successful and unsuccessful commands

    When a command returns after an error, it returns a nonzero exit status. The command returns zero when it terminates after successful completion. The return status can be read from special variable $? (run echo $? immediately after the command execution statement to print the exit status).

    The following command prints the stderr text to the screen rather than to a file (and because there is no stdout output, out.txt will be empty):

    $ ls + > out.txt 
    ls: cannot access +: No such file or directory 
    

    In the following command, we redirect stderr to out.txt:

    $ ls + 2> out.txt # works
    

    You can redirect stderr exclusively to a file and stdout to another file as follows:

    $ cmd 2>stderr.txt 1>stdout.txt
    

    It is also possible to redirect stderr and stdout to a single file by converting stderr to stdout using this preferred method:

    $ cmd 2>&1 output.txt
    

    Or the alternate approach:

    $ cmd &> output.txt 
    
  5. Sometimes, the output may contain unnecessary information (such as debug messages). If you don't want the output terminal burdened with the stderr details then you should redirect the stderr output to /dev/null, which removes it completely. For example, consider that we have three files a1, a2, and a3. However, a1 does not have the read-write-execute permission for the user. When you need to print the contents of files starting with a, we use the cat command. Set up the test files as follows:

    $ echo a1 > a1 
    $ cp a1 a2 ; cp a2 a3;
    $ chmod 000 a1  #Deny all permissions
    

    While displaying contents of the files using wildcards (a*), it will show an error message for file a1 as it does not have the proper read permission:

    $ cat a*
    cat: a1: Permission denied
    a1
    a1
    

    Here, cat: a1: Permission denied belongs to the stderr data. We can redirect the stderr data into a file, whereas stdout remains printed in the terminal. Consider the following code:

    $ cat a* 2> err.txt #stderr is redirected to err.txt
    a1
    a1
    
    $ cat err.txt
    cat: a1: Permission denied
    

    Take a look at the following code:

    $ cmd 2>/dev/null
    

    When redirection is performed for stderr or stdout, the redirected text flows into a file. As the text has already been redirected and has gone into the file, no text remains to flow to the next command through pipe (|), and it appears to the next set of command sequences through stdin.

  6. However, there is a way to redirect data to a file, as well as provide a copy of redirected data as stdin for the next set of commands. This can be done using the tee command. For example, to print stdout in the terminal as well as redirect stdout into a file, the syntax for tee is as follows:

    command | tee FILE1 FILE2
    

    In the following code, the stdin data is received by the tee command. It writes a copy of stdout to the out.txt file and sends another copy as stdin for the next command. The cat -n command puts a line number for each line received from stdin and writes it into stdout:

    $ cat a* | tee out.txt | cat -n
    cat: a1: Permission denied
         1a1
         2a1
    

    Examine the contents of out.txt as follows:

    $ cat out.txt
    a1
    a1
    

    Note that cat: a1: Permission denied does not appear because it belongs to stderr. The tee command can read from stdin only.

    By default, the tee command overwrites the file, but it can be used with appended options by providing the -a option, for example, $ cat a* | tee -a out.txt | cat -n.

    Commands appear with arguments in the format: command FILE1 FILE2 … or simply command FILE.

  7. We can use stdin as a command argument. It can be done by using - as the filename argument for the command as follows:

    $ cmd1 | cmd2 | cmd -
    

    For example:

    $ echo who is this | tee -
    who is this
    who is this
    

    Alternately, we can use /dev/stdin as the output filename to use stdin.

    Similarly, use /dev/stderr for standard error and /dev/stdout for standard output. These are special device files that correspond to stdin, stderr, and stdout.

How it works...

For output redirection, > and >> operators are different. Both of them redirect text to a file, but the first one empties the file and then writes to it, whereas the later one adds the output to the end of the existing file.

When we use a redirection operator, the output won't print in the terminal but it is directed to a file. When redirection operators are used, by default, they operate on standard output. To explicitly take a specific file descriptor, you must prefix the descriptor number to the operator.

> is equivalent to 1> and similarly it applies for >> (equivalent to 1>>).

When working with errors, the stderr output is dumped to the /dev/null file. ./dev/null is a special device file where any data received by the file is discarded. The null device is often known as a black hole as all the data that goes into it is lost forever.

There's more...

A command that reads stdin for input can receive data in multiple ways. Also, it is possible to specify file descriptors of our own using cat and pipes, for example:

$ cat file | cmd
$ cmd1 | cmd

Redirection from a file to a command

By using redirection, we can read data from a file as stdin as follows:

$ cmd < file

Redirecting from a text block enclosed within a script

Sometimes we need to redirect a block of text (multiple lines of text) as standard input. Consider a particular case where the source text is placed within the shell script. A practical usage example is writing a logfile header data. It can be performed as follows:

#!/bin/bash
cat<<EOF>log.txt
LOG FILE HEADER
This is a test log file
Function: System statistics
EOF

The lines that appear between cat <<EOF >log.txt and the next EOF line will appear as the stdin data. Print the contents of log.txt as follows:

$ cat log.txt
LOG FILE HEADER
This is a test log file
Function: System statistics

Custom file descriptors

A file descriptor is an abstract indicator for accessing a file. Each file access is associated with a special number called a file descriptor. 0, 1, and 2 are reserved descriptor numbers for stdin, stdout, and stderr.

We can create our own custom file descriptors using the exec command. If you are already familiar with file programming with any other programming language, you might have noticed modes for opening files. Usually, the following three modes are used:

  • Read mode

  • Write with truncate mode

  • Write with append mode

< is an operator used to read from the file to stdin. > is the operator used to write to a file with truncation (data is written to the target file after truncating the contents). >> is an operator used to write to a file by appending (data is appended to the existing file contents and the contents of the target file will not be lost). File descriptors can be created with one of the three modes.

Create a file descriptor for reading a file, as follows:

$ exec 3<input.txt # open for reading with descriptor number 3

We could use it in the following way:

$ echo this is a test line > input.txt
$ exec 3<input.txt

Now you can use file descriptor 3 with commands. For example, we will use cat <&3 as follows:

$ cat<&3
this is a test line

If a second read is required, we cannot re-use the file descriptor 3. It is required that we reassign the file descriptor 3 for read using exec for making a second read.

Create a file descriptor for writing (truncate mode) as follows:

$ exec 4>output.txt # open for writing

For example:

$ exec 4>output.txt
$ echo newline >&4
$ cat output.txt
newline

Create a file descriptor for writing (append mode) as follows:

$ exec 5>>input.txt

For example:

$ exec 5>>input.txt
$ echo appended line >&5
$ cat input.txt 
newline
appended line
 

Arrays and associative arrays


Arrays are a very important component for storing a collection of data as separate entities using indexes. Regular arrays can use only integers as their array index. On the other hand, Bash also supports associative arrays that can take a string as their array index. Associative arrays are very useful in many types of manipulations where having a string index makes more sense. In this recipe, we will see how to use both of these.

Getting ready

To use associate arrays, you must have Bash Version 4 or higher.

How to do it...

  1. An array can be defined in many ways. Define an array using a list of values in a line as follows:

    array_var=(1 2 3 4 5 6)
    #Values will be stored in consecutive locations starting from index 0.
    

    Alternately, define an array as a set of index-value pairs as follows:

    array_var[0]="test1"
    array_var[1]="test2"
    array_var[2]="test3"
    array_var[3]="test4"
    array_var[4]="test5"
    array_var[5]="test6"
    
  2. Print the contents of an array at a given index using the following commands:

    echo ${array_var[0]}
    test1
    index=5
    echo ${array_var[$index]}
    test6
    
  3. Print all of the values in an array as a list using the following commands:

    $ echo ${array_var[*]}
    test1 test2 test3 test4 test5 test6
    

    Alternately, you could use:

    $ echo ${array_var[@]}
    test1 test2 test3 test4 test5 test6
    
  4. Print the length of an array (the number of elements in an array) as follows:

    $ echo ${#array_var[*]}
    6
    

There's more...

Associative arrays have been introduced to Bash from Version 4.0 and they are useful entities to solve many problems using the hashing technique. Let us go into more detail.

Defining associative arrays

In an associative array, we can use any text data as an array index. Initially, a declaration statement is required to declare a variable name as an associative array. This can be done as follows:

$ declare -A ass_array

After the declaration, elements can be added to the associative array using two methods as follows:

  • By using inline index-value list method, we can provide a list of index-value pairs:

    $ ass_array=([index1]=val1 [index2]=val2)
    
  • Alternately, you could use separate index-value assignments:

    $ ass_array[index1]=val1
    $ ass_array'index2]=val2
    

For example, consider the assignment of price for fruits using an associative array:

$ declare -A fruits_value
$ fruits_value=([apple]='100dollars' [orange]='150 dollars')

Display the content of an array as follows:

$ echo "Apple costs ${fruits_value[apple]}"
Apple costs 100 dollars

Listing of array indexes

Arrays have indexes for indexing each of the elements. Ordinary and associative arrays differ in terms of index type. We can obtain the list of indexes in an array as follows:

$ echo ${!array_var[*]}

Or, we can also use:

$ echo ${!array_var[@]

In the previous fruits_value array example, consider the following command:

$ echo ${!fruits_value[*]}
orange apple

This will work for ordinary arrays too.

 

Visiting aliases


An alias is basically a shortcut that takes the place of typing a long-command sequence. In this recipe, we will see how to create aliases using the alias command.

How to do it...

There are various operations you can perform on aliases, these are as follows:

  1. An alias can be created as follows:

    $ alias new_command='command sequence'
    

    Giving a shortcut to the install command, apt-get install, can be done as follows:

    $ alias install='sudo apt-get install'
    

    Therefore, we can use install pidgin instead of sudo apt-get install pidgin.

  2. The alias command is temporary; aliasing exists until we close the current terminal only. To keep these shortcuts permanent, add this statement to the ~/.bashrc file. Commands in ~/.bashrc are always executed when a new shell process is spawned:

    $ echo 'alias cmd="command seq"' >> ~/.bashrc
    
  3. To remove an alias, remove its entry from ~/.bashrc (if any) or use the unalias command. Alternatively, alias example= should unset the alias named example.

  4. As an example, we can create an alias for rm so that it will delete the original and keep a copy in a backup directory:

    alias rm='cp $@ ~/backup && rm $@'
    

Note

When you create an alias, if the item being aliased already exists, it will be replaced by this newly aliased command for that user.

There's more...

There are situations when aliasing can also be a security breach. See how to identify them.

Escaping aliases

The alias command can be used to alias any important command, and you may not always want to run the command using the alias. We can ignore any aliases currently defined by escaping the command we want to run. For example:

$ \command

The \ character escapes the command, running it without any aliased changes. While running privileged commands on an untrusted environment, it is always good security practice to ignore aliases by prefixing the command with \. The attacker might have aliased the privileged command with his/her own custom command to steal the critical information that is provided by the user to the command.

 

Grabbing information about the terminal


While writing command-line shell scripts, we will often need to heavily manipulate information about the current terminal, such as the number of columns, rows, cursor positions, masked password fields, and so on. This recipe helps in collecting and manipulating terminal settings.

Getting ready

tput and stty are utilities that can be used for terminal manipulations. Let us see how to use them to perform different tasks.

How to do it...

There are specific information you can gather about the terminal as shown in the following list:

  • Get the number of columns and rows in a terminal by using the following commands:

    tput cols
    tput lines
    
  • To print the current terminal name, use the following command:

    tput longname
    
  • To move the cursor to a 100,100 position, you can enter:

    tput cup 100 100
    
  • Set the background color for the terminal using the following command:

    tputsetb n
    

    n can be a value in the range of 0 to 7.

  • Set the foreground color for text by using the following command:

    tputsetf n
    

    n can be a value in the range of 0 to 7.

  • To make text bold use this:

    tput bold
    
  • To start and end underlining use this:

    tput smul
    tput rmul
    
  • To delete from the cursor to the end of the line use the following command:

    tputed
    
  • While typing a password, we should not display the characters typed. In the following example, we will see how to do it using stty:

    #!/bin/sh
    #Filename: password.sh
    echo -e "Enter password: "
    stty -echo
    read password
    stty echo
    echo
    echo Password read.
    

    Note

    The -echo option in the preceding command disables the output to the terminal, whereas echo enables output.

 

Getting and setting dates and delays


Many applications require printing dates in different formats, setting date and time, and performing manipulations based on date and time. Delays are commonly used to provide a wait time (such as 1 second) during the program execution. Scripting contexts, such as monitoring a task every 5 seconds, demands the understanding of writing delays in a program. This recipe will show you how to work with dates and time delays.

Getting ready

Dates can be printed in variety of formats. We can also set dates from the command line. In Unix-like systems, dates are stored as an integer, which denotes the number of seconds since 1970-01-01 00:00:00 UTC. This is called epoch or Unix time . Let us see how to read dates and set them.

How to do it...

It is possible to read the dates in different formats and also to set the date. This can be accomplished with these steps:

  1. You can read the date as follows:

    $ date
    Thu May 20 23:09:04 IST 2010
    
  2. The epoch time can be printed as follows:

    $ date +%s
    1290047248
    

    We can find out epoch from a given formatted date string. You can use dates in multiple date formats as input. Usually, you don't need to bother about the date string format that you use if you are collecting the date from a system log or any standard application generated output. Convert the date string into epoch as follows:

    $ date --date "Thu Nov 18 08:07:21 IST 2010" +%s
    1290047841
    

    The --date option is used to provide a date string as input. However, we can use any date formatting options to print the output. Feeding the input date from a string can be used to find out the weekday, given the date.

    For example:

    $ date --date "Jan 20 2001" +%A
    Saturday
    

    The date format strings are listed in the table mentioned in the How it works… section:

  3. Use a combination of format strings prefixed with + as an argument for the date command to print the date in the format of your choice. For example:

    $ date "+%d %B %Y"
    20 May 2010
    
  4. We can set the date and time as follows:

    # date -s "Formatted date string"
    

    For example:

    # date -s "21 June 2009 11:01:22"
    
  5. Sometimes we need to check the time taken by a set of commands. We can display it using the following code:

    #!/bin/bash
    #Filename: time_take.sh
    start=$(date +%s)
    commands;
    statements;
    
    end=$(date +%s)
    difference=$(( end - start))
    echo Time taken to execute commands is $difference seconds.

Note

An alternate method would be to use time <scriptpath> to get the time that it took to execute the script.

How it works...

While considering dates and time, epoch is defined as the number of seconds that have elapsed since midnight proleptic Coordinated Universal Time (UTC) of January 1, 1970, not counting leap seconds. Epoch time is very useful when you need to calculate the difference between two dates or time. You may find out the epoch times for two given timestamps and take the difference between the epoch values. Therefore, you can find out the total number of seconds between two dates.

To write a date format to get the output as required, use the following table:

Date component

Format

Weekday

%a (for example, Sat)

%A (for example, Saturday)

Month

%b (for example, Nov)

%B (for example, November)

Day

%d (for example, 31)

Date in format (mm/dd/yy)

%D (for example, 10/18/10)

Year

%y (for example, 10)

%Y (for example, 2010)

Hour

%I or %H (For example, 08)

Minute

%M (for example, 33)

Second

%S (for example, 10)

Nano second

%N (for example, 695208515)

Epoch Unix time in seconds

%s (for example, 1290049486)

There's more...

Producing time intervals is very essential when writing monitoring scripts that execute in a loop. Let us see how to generate time delays.

Producing delays in a script

To delay execution in a script for a particular period of time, use sleep:$ sleepno_of_seconds. For example, the following script counts from 0 to 40 by using tput and sleep:

#!/bin/bash
#Filename: sleep.sh
echo -n Count:
tput sc

count=0;
while true;
do
    if [ $count -lt 40 ];
    then
        let count++;
        sleep 1;
        tput rc
        tput ed
        echo -n $count;
    else exit 0;
    fi
done

In the preceding example, a variable count is initialized to 0 and is incremented on every loop execution. The echo statement prints the text. We use tput sc to store the cursor position. On every loop execution we write the new count in the terminal by restoring the cursor position for the number. The cursor position is restored using tput rc. This clears text from the current cursor position to the end of the line, so that the older number can be cleared and the count can be written. A delay of 1 second is provided in the loop by using the sleep command.

 

Debugging the script


Debugging is one of the critical features that every programming language should implement to produce race-back information when something unexpected happens. Debugging information can be used to read and understand what caused the program to crash or to act in an unexpected fashion. Bash provides certain debugging options that every sysadmin should know. This recipe shows how to use these.

How to do it...

We can either use Bash's inbuilt debugging tools or write our scripts in such a manner that they become easy to debug, here's how:

  1. Add the -x option to enable debug tracing of a shell script as follows:

    $ bash -x script.sh
    

    Running the script with the -x flag will print each source line with the current status. Note that you can also use sh -x script.

  2. Debug only portions of the script using set -x and set +x. For example:

    #!/bin/bash
    #Filename: debug.sh
    for i in {1..6};
    do
        set -x
        echo $i
        set +x
    done
    echo "Script executed"

    In the preceding script, the debug information for echo $i will only be printed, as debugging is restricted to that section using -x and +x.

  3. The aforementioned debugging methods are provided by Bash built-ins. But they always produce debugging information in a fixed format. In many cases, we need debugging information in our own format. We can set up such a debugging style by passing the _DEBUG environment variable.

    Look at the following example code:

    #!/bin/bash
    function DEBUG()
    {
        [ "$_DEBUG" == "on" ] && $@ || :
    }
    for i in {1..10}
    do
        DEBUG echo $i
    done

    We can run the above script with debugging set to "on" as follows:

    $ _DEBUG=on ./script.sh
    

    We prefix DEBUG before every statement where debug information is to be printed. If _DEBUG=on is not passed to the script, debug information will not be printed. In Bash, the command : tells the shell to do nothing.

How it works...

The -x flag outputs every line of script as it is executed to stdout. However, we may require only some portions of the source lines to be observed such that commands and arguments are to be printed at certain portions. In such conditions we can use set builtin to enable and disable debug printing within the script.

  • set -x: This displays arguments and commands upon their execution

  • set +x: This disables debugging

  • set -v: This displays input when they are read

  • set +v: This disables printing input

There's more...

We can also use other convenient ways to debug scripts. We can make use of shebang in a trickier way to debug scripts.

Shebang hack

The shebang can be changed from #!/bin/bash to #!/bin/bash -xv to enable debugging without any additional flags (-xv flags themselves).

 

Functions and arguments


Like any other scripting languages, Bash also supports functions. Let us see how to define and use functions.

How to do it...

We can create functions to perform tasks and we can also create functions that take parameters (also called arguments) as you can see in the following steps:

  1. A function can be defined as follows:

    function fname()
    {
        statements;
    }
    Or alternately,
    fname()
    {
        statements;
    }
  2. A function can be invoked just by using its name:

    $ fname ; # executes function
    
  3. Arguments can be passed to functions and can be accessed by our script:

    fname arg1 arg2 ; # passing args
    

    Following is the definition of the function fname. In the fname function, we have included various ways of accessing the function arguments.

    fname()
    {
      echo $1, $2; #Accessing arg1 and arg2
      echo "$@"; # Printing all arguments as list at once
      echo "$*"; # Similar to $@, but arguments taken as single entity
      return 0; # Return value
    }

    Similarly, arguments can be passed to scripts and can be accessed by script:$0 (the name of the script):

    • $1 is the first argument

    • $2 is the second argument

    • $n is the nth argument

    • "$@"expands as "$1" "$2" "$3" and so on

    • "$*" expands as "$1c$2c$3", where c is the first character of IFS

    • "$@" is used more often than "$*"since the former provides all arguments as a single string

There's more...

Let us explore through more tips on Bash functions.

The recursive function

Functions in Bash also support recursion (the function that can call itself). For example, F() { echo $1; F hello; sleep 1; }.

Tip

Fork bomb

We can write a recursive function, which is basically a function that calls itself:

:(){ :|:& };:

It infinitely spawns processes and ends up in a denial-of-service attack. & is postfixed with the function call to bring the subprocess into the background. This is a dangerous code as it forks processes and, therefore, it is called a fork bomb.

You may find it difficult to interpret the preceding code. See the Wikipedia page http://en.wikipedia.org/wiki/Fork_bomb for more details and interpretation of the fork bomb.

It can be prevented by restricting the maximum number of processes that can be spawned from the config file at /etc/security/limits.conf.

Exporting functions

A function can be exported—like environment variables—using export, such that the scope of the function can be extended to subprocesses, as follows:

export -f fname

Reading the return value (status) of a command

We can get the return value of a command or function in the following way:

cmd; 
echo $?;

$? will give the return value of the command cmd.

The return value is called exit status . It can be used to analyze whether a command completed its execution successfully or unsuccessfully. If the command exits successfully, the exit status will be zero, otherwise it will be a nonzero value.

We can check whether a command terminated successfully or not by using the following script:

#!/bin/bash
#Filename: success_test.sh
CMD="command" #Substitute with command for which you need to test the exit status
$CMD
if [ $? -eq 0 ];
then
    echo "$CMD executed successfully"
else
    echo "$CMD terminated unsuccessfully"
fi

Passing arguments to commands

Arguments to commands can be passed in different formats. Suppose -p and-v are the options available and -k N is another option that takes a number. Also, the command takes a filename as argument. It can be executed in multiple ways as shown:

  • $ command -p -v -k 1 file

  • $ command -pv -k 1 file

  • $ command -vpk 1 file

  • $ command file -pvk 1

 

Reading the output of a sequence of commands in a variable


One of the best-designed features of shell scripting is the ease of combining many commands or utilities to produce output. The output of one command can appear as the input of another, which passes its output to another command, and so on. The output of this combination can be read in a variable. This recipe illustrates how to combine multiple commands and how its output can be read.

Getting ready

Input is usually fed into a command through stdin or arguments. Output appears as stderr or stdout. While we combine multiple commands, we usually use stdin to give input and stdout to provide an output.

In this context, the commands are called filters . We connect each filter using pipes, the piping operator being |. An example is as follows:

$ cmd1 | cmd2 | cmd3 

Here we combine three commands. The output of cmd1 goes to cmd2 and output of cmd2 goes to cmd3 and the final output (which comes out of cmd3) will be printed, or it can be directed to a file.

How to do it...

We typically use pipes and use them with the subshell method for combining outputs of multiple files. Here's how:

  1. Let us start with combining two commands:

    $ ls | cat -n > out.txt
    

    Here the output of ls (the listing of the current directory) is passed to cat –n, which in turn puts line numbers to the input received through stdin. Therefore, its output is redirected to the out.txt file.

  2. We can read the output of a sequence of commands combined by pipes as follows:

    cmd_output=$(COMMANDS)

    This is called subshell method . For example:

    cmd_output=$(ls | cat -n)
    echo $cmd_output
    

    Another method, called back quotes (some people also refer to it as back tick ) can also be used to store the command output as follows:

    cmd_output=`COMMANDS`
    

    For example:

    cmd_output=`ls | cat -n`
    echo $cmd_output
    

    Back quote is different from the single-quote character. It is the character on the ~ button in the keyboard.

There's more...

There are multiple ways of grouping commands. Let us go through a few of them.

Spawning a separate process with subshell

Subshells are separate processes. A subshell can be defined using the ( )operators as follows:

pwd;
(cd /bin; ls);
pwd;

When some commands are executed in a subshell, none of the changes occur in the current shell; changes are restricted to the subshell. For example, when the current directory in a subshell is changed using the cd command, the directory change is not reflected in the main shell environment.

The pwd command prints the path of the working directory.

The cd command changes the current directory to the given directory path.

Subshell quoting to preserve spacing and the newline character

Suppose we are reading the output of a command to a variable using a subshell or the back quotes method. We always quote them in double quotes to preserve the spacing and newline character (\n). For example:

$ cat text.txt
1
2
3

$ out=$(cat text.txt)
$ echo $out
1 2 3 # Lost \n spacing in 1,2,3 

$ out="$(cat tex.txt)"
$ echo$out
1
2
3
 

Reading n characters without pressing the return key


read is an important Bash command to read text from the keyboard or standard input. We can use read to interactively read an input from the user, but read is capable of much more. Most of the input libraries in any programming language read the input from the keyboard; but string input termination is done when return is pressed. There are certain critical situations when return cannot be pressed, but the termination is done based on a number of characters or a single character. For example, in a game, a ball is moved upward when + is pressed. Pressing + and then pressing return every time to acknowledge the + press is not efficient. In this recipe we will use the read command that provides a way to accomplish this task without having to press return.

How to do it...

You can use various options of the read command to obtain different results as shown in the following steps:

  1. The following statement will read n characters from input into the variable_name variable:

    read -n number_of_chars variable_name
    

    For example:

    $ read -n 2 var
    $ echo $var
    
  2. Read a password in the nonechoed mode as follows:

    read -s var
    
  3. Display a message with read using:

    read -p "Enter input:"  var
    
  4. Read the input after a timeout as follows:

    read -t timeout var
    

    For example:

    $ read -t 2 var
    #Read the string that is typed within 2 seconds into variable var.
    
  5. Use a delimiter character to end the input line as follows:

    read -d delim_char var
    

    For example:

    $ read -d ":" var
    hello:#var is set to hello
    
 

Running a command until it succeeds


When using your shell for everyday tasks, there will be cases where a command might succeed only after some conditions are met, or the operation depends on an external event (such as a file being available to download). In such cases, one might want to run a command repeatedly until it succeeds.

How to do it...

Define a function in the following way:

repeat()
{
  while true
  do
    $@ && return
  done
}

Or, add this to your shell's rc file for ease of use:

repeat() { while true; do $@ && return; done }

How it works...

We create a function called repeat that has an infinite while loop, which attempts to run the command passed as a parameter (accessed by $@) to the function. It then returns if the command was successful, thereby exiting the loop.

There's more...

We saw a basic way to run commands until they succeed. Let us see what we can do to make things more efficient.

A faster approach

On most modern systems, true is implemented as a binary in /bin. This means that each time the aforementioned while loop runs, the shell has to spawn a process. To avoid this, we can use the : shell built-in, which always returns an exit code 0:

repeat() { while :; do $@ && return; done }

Though not as readable, this is certainly faster than the first approach.

Adding a delay

Let's say you are using repeat() to download a file from the Internet which is not available right now, but will be after some time. An example would be:

repeat wget -c http://www.example.com/software-0.1.tar.gz

In the current form, we will be sending too much traffic to the web server at www.example.com, which causes problems to the server (and maybe even to you, if say the server blacklists your IP for spam). To solve this, we can modify the function and add a small delay as follows:

repeat() { while :; do $@ && return; sleep 30; done }

This will cause the command to run every 30 seconds.

 

Field separators and iterators


The internal field separator (IFS) is an important concept in shell scripting. It is very useful while manipulating text data. We will now discuss delimiters that separate different data elements from single data stream. An internal field separator is a delimiter for a special purpose. An internal field separator is an environment variable that stores delimiting characters. It is the default delimiter string used by a running shell environment.

Consider the case where we need to iterate through words in a string or comma separated values (CSV). In the first case we will use IFS=" " and in the second, IFS=",". Let us see how to do it.

Getting ready

Consider the case of CSV data:

data="name,sex,rollno,location"
To read each of the item in a variable, we can use IFS.
oldIFS=$IFS
IFS=, now,
for item in $data;
do
    echo Item: $item
done

IFS=$oldIFS

The output is as follows:

Item: name
Item: sex
Item: rollno
Item: location

The default value of IFS is a space component (newline, tab, or a space character).

When IFS is set as , the shell interprets the comma as a delimiter character, therefore, the $item variable takes substrings separated by a comma as its value during the iteration.

If IFS is not set as , then it would print the entire data as a single string.

How to do it...

Let us go through another example usage of IFS by taking the /etc/passwd file into consideration. In the /etc/passwd file, every line contains items delimited by ":". Each line in the file corresponds to an attribute related to a user.

Consider the input: root:x:0:0:root:/root:/bin/bash. The last entry on each line specifies the default shell for the user. To print users and their default shells, we can use the IFS hack as follows:

#!/bin/bash
#Desc: Illustration of IFS
line="root:x:0:0:root:/root:/bin/bash" 
oldIFS=$IFS;
IFS=":"
count=0
for item in $line;
do

     [ $count -eq 0 ]  && user=$item;
     [ $count -eq 6 ]  && shell=$item;
    let count++
done;
IFS=$oldIFS
echo $user\'s shell is $shell;

The output will be:

root's shell is /bin/bash

Loops are very useful in iterating through a sequence of values. Bash provides many types of loops. Let us see how to use them:

  • Using a for loop:

    for var in list;
    do
        commands; # use $var
    done
    list can be a string, or a sequence.

    We can generate different sequences easily.

    echo {1..50}can generate a list of numbers from 1 to 50. echo {a..z}or{A..Z} or {a..h} can generate lists of alphabets. Also, by combining these we can concatenate data.

    In the following code, in each iteration, the variable i will hold a character in the range a to z:

    for i in {a..z}; do actions; done;

    The for loop can also take the format of the for loop in C. For example:

    for((i=0;i<10;i++))
    {
        commands; # Use $i
    }
  • Using a while loop:

    while condition
    do
        commands;
    done

    For an infinite loop, use true as the condition.

  • Using a until loop:

    A special loop called until is available with Bash. This executes the loop until the given condition becomes true. For example:

    x=0;
    until [ $x -eq 9 ]; # [ $x -eq 9 ] is the condition
    do
        let x++; echo $x;
    done
 

Comparisons and tests


Flow control in a program is handled by comparison and test statements. Bash also comes with several options to perform tests that are compatible with the Unix system-level features. We can use if, if else, and logical operators to perform tests and certain comparison operators to compare data items. There is also a command called test available to perform tests. Let us see how to use these.

How to do it...

We will have a look at all the different methods used for comparisons and performing tests:

  • Using an if condition:

    if condition;
    then
        commands;
    fi
  • Using else if and else:

    if condition; 
    then
        commands;
    else if condition; then
        commands;
    else
        commands;
    fi

    Note

    Nesting is also possible with if and else. The if conditions can be lengthy, to make them shorter we can use logical operators as follows:

    • [ condition ] && action; # action executes if the condition is true

    • [ condition ] || action; # action executes if the condition is false

    && is the logical AND operation and || is the logical OR operation. This is a very helpful trick while writing Bash scripts.

  • Performing mathematical comparisons: Usually conditions are enclosed in square brackets []. Note that there is a space between [ or ] and operands. It will show an error if no space is provided. An example is as follows:

    [$var -eq 0 ] or [ $var -eq 0]

    Performing mathematical conditions over variables or values can be done as follows:

    [ $var -eq 0 ]  # It returns true when $var equal to 0.
    [ $var -ne 0 ] # It returns true when $var is not equal to 0

    Other important operators are as follows:

    • -gt: Greater than

    • -lt: Less than

    • -ge: Greater than or equal to

    • -le: Less than or equal to

    Multiple test conditions can be combined as follows:

    [ $var1 -ne 0 -a $var2 -gt 2 ]  # using and -a
    [ $var1 -ne 0 -o var2 -gt 2 ] # OR -o
  • Filesystem related tests: We can test different filesystem-related attributes using different condition flags as follows:

    • [ -f $file_var ]: This returns true if the given variable holds a regular file path or filename

    • [ -x $var ]: This returns true if the given variable holds a file path or filename that is executable

    • [ -d $var ]: This returns true if the given variable holds a directory path or directory name

    • [ -e $var ]: This returns true if the given variable holds an existing file

    • [ -c $var ]: This returns true if the given variable holds the path of a character device file

    • [ -b $var ]: This returns true if the given variable holds the path of a block device file

    • [ -w $var ]: This returns true if the given variable holds the path of a file that is writable

    • [ -r $var ]: This returns true if the given variable holds the path of a file that is readable

    • [ -L $var ]: This returns true if the given variable holds the path of a symlink

    An example of the usage is as follows:

    fpath="/etc/passwd"
    if [ -e $fpath ]; then
        echo File exists; 
    else
        echo Does not exist; 
    fi
  • String comparisons: While using string comparison, it is best to use double square brackets, since the use of single brackets can sometimes lead to errors.

    Two strings can be compared to check whether they are the same in the following manner:

    • [[ $str1 = $str2 ]]: This returns true when str1 equals str2, that is, the text contents of str1 and str2 are the same

    • [[ $str1 == $str2 ]]: It is an alternative method for string equality check

    We can check whether two strings are not the same as follows:

    • [[ $str1 != $str2 ]]: This returns true when str1 and str2 mismatch

    We can find out the alphabetically smaller or larger string as follows:

    • [[ $str1 > $str2 ]]: This returns true when str1 is alphabetically greater than str2

    • [[ $str1 < $str2 ]]: This returns true when str1 is alphabetically lesser than str2

      Note

      Note that a space is provided after and before =, if it is not provided, it is not a comparison, but it becomes an assignment statement.

    • [[ -z $str1 ]]: This returns true if str1 holds an empty string

    • [[ -n $str1 ]]: This returns true if str1 holds a nonempty string

    It is easier to combine multiple conditions using logical operators such as && and || in the following code:

    if [[ -n $str1 ]] && [[ -z $str2 ]] ;
    then
        commands;
    fi

    For example:

    str1="Not empty "
    str2=""
    if [[ -n $str1 ]] && [[ -z $str2 ]];
    then
        echo str1 is nonempty and str2 is empty string.
    fi

    Output:

    str1 is nonempty and str2 is empty string.

The test command can be used for performing condition checks. It helps to avoid usage of many braces. The same set of test conditions enclosed within [] can be used for the test command.

For example:

if  [ $var -eq 0 ]; then echo "True"; fi
can be written as
if  test $var -eq 0 ; then echo "True"; fi
Latest Reviews (15 reviews total)
Very basic and understandable. Was easy to use.
Good book. A lot of very useful information for Linux admins.
Nice tricks found. Learned new ways.
Linux Shell Scripting Cookbook, Second Edition - Second Edition
Unlock this book and the full library FREE for 7 days
Start now