In this chapter, we will cover:
Printing in the terminal
Playing with variables and environment variables
Function to prepend to environment variables
Math with the shell
Playing with file descriptors and redirection
Arrays and associative array
Visiting aliases
Grabbing information about the terminal
Getting and setting dates and delays
Debugging the script
Functions and arguments
Reading output of a sequence of commands in a variable
Reading n characters without pressing the return key
Running a command until it succeeds
Field separators and iterators
Comparisons and tests
Unix-like systems are amazing operating system designs. Even after many decades, Unix-style architecture for operating systems serves as one of the best designs. One of the important features of this architecture is the command-line interface, or the shell. The shell environment helps users to interact with and access core functions of the operating system. The term scripting is more relevant in this context. Scripting is usually supported by interpreter-based programming languages. Shell scripts are files in which we write a sequence of commands that we need to perform and are executed using the shell utility.
In this book we are dealing with Bash (Bourne Again Shell), which is the default shell environment for most GNU/Linux systems. Since GNU/Linux is the most prominent operating system on Unix-style architecture, most of the examples and discussions are written by keeping Linux systems in mind.
The primary purpose of this chapter is to give readers an insight into the shell environment and become familiar with the basic features that the shell offers. Commands are typed and executed in a shell terminal. When a terminal is opened, a prompt is available which usually has the following format:
username@hostname$
Or:
root@hostname #
or simply as $
or #
.
$
represents regular users and #
represents the administrative user root. Root is the most privileged user in a Linux system.
Note
It is usually a bad idea to directly use the shell as the root user (administrator) to perform tasks. This is because typing errors in your commands have the potential to do more damage when your shell has more privileges. So, it is recommended to log in as a regular user (your shell will denote that as $
in the prompt, and #
when running as root), and then use tools such as `sudo'
to run privileged commands. Running a command such as sudo <command> <arguments>
will run it as root.
A shell script is a text file that typically begins with a shebang, as follows:
#!/bin/bash
Shebang is a line on which #!
is prefixed to the interpreter path. /bin/bash
is the interpreter command path for Bash.
Execution of a script can be done in two ways. Either we can run the script as a command-line argument to bash
or we can grant execution permission to the script so it becomes executable.
The script can be run with the filename as a command-line argument as follows (the text that starts with #
is a comment, you don't have to type it out):
$ bash script.sh # Assuming script is in the current directory.
Or:
$ bash /home/path/script.sh # Using full path of script.sh.
If a script is run as a command-line argument for bash
, the shebang in the script is not required.
If required, we can utilize the shebang to facilitate running the script on its own. For this, we have to set executable permissions for the script and it will run using the interpreter path that is appended to #!
to the shebang. This can be set as follows:
$ chmod a+x script.sh
This command gives the script.sh
file the executable permission for all users. The script can be executed as:
$ ./script.sh #./ represents the current directory
Or:
$ /home/path/script.sh # Full path of the script is used
The kernel will read the first line and see that the shebang is #!/bin/bash
. It will identify /bin/bash
and execute the script internally as:
$ /bin/bash script.sh
When a shell is started, it initially executes a set of commands to define various settings such as prompt text, colors, and much more. This set of commands are read from a shell script at ~/.bashrc
(or ~/.bash_profile
for login shells) located in the home directory of the user. The Bash shell also maintains a history of commands run by the user. It is available in the ~/.bash_history
file.
Note
~
denotes your home directory, which is usually /home/user
where user
is your username or /root
for the root user.
A login shell is the shell which you get just after logging in to a machine. However, if you open up a shell while logged in to a graphical environment (such as GNOME, KDE, and so on), then it is not a login shell.
In Bash, each command or command sequence is delimited by using a semicolon or a new line. For example:
$ cmd1 ; cmd2
This is equivalent to:
$ cmd1 $ cmd2
Finally, the #
character is used to denote the beginning of unprocessed comments. A comment section starts with #
and proceeds up to the end of that line. The comment lines are most often used to provide comments about the code in the file or to stop a line of code from being executed.
Now let us move on to the basic recipes in this chapter.
The terminal is an interactive utility by which a user interacts with the shell environment. Printing text in the terminal is a basic task that most shell scripts and utilities need to perform regularly. As we will see in this recipe, this can be performed via various methods and in different formats.
echo
is the basic command for printing in the terminal.
echo
puts a newline at the end of every echo invocation by default:
$ echo "Welcome to Bash" Welcome to Bash
Simply, using double-quoted text with the echo
command prints the text in the terminal. Similarly, text without double quotes also gives the same output:
$ echo Welcome to Bash Welcome to Bash
Another way to do the same task is by using single quotes:
$ echo 'text in quotes'
These methods may look similar, but some of them have a specific purpose and side effects too. Consider the following command:
$ echo "cannot include exclamation - ! within double quotes"
This will return the following output:
bash: !: event not found error
Hence, if you want to print special characters such as !
, either do not use them within double quotes or escape them with a special escape character (\
) prefixed with it, like so:
$ echo Hello world !
Or:
$ echo 'Hello world !'
Or:
$ echo "Hello world \!" #Escape character \ prefixed.
The side effects of each of the methods are as follows:
When using
echo
without quotes, we cannot use a semicolon, as it acts as a delimiter between commands in the Bash shellecho hello; hello
takesecho hello
as one command and the secondhello
as the second commandVariable substitution, which is discussed in the next recipe, will not work within single quotes
Another command for printing in the terminal is printf
. It uses the same arguments as the printf
command in the C programming language. For example:
$ printf "Hello world"
printf
takes quoted text or arguments delimited by spaces. We can use formatted strings with printf
. We can specify string width, left or right alignment, and so on. By default, printf
does not have newline as in the echo
command. We have to specify a newline when required, as shown in the following script:
#!/bin/bash #Filename: printf.sh printf "%-5s %-10s %-4s\n" No Name Mark printf "%-5s %-10s %-4.2f\n" 1 Sarath 80.3456 printf "%-5s %-10s %-4.2f\n" 2 James 90.9989 printf "%-5s %-10s %-4.2f\n" 3 Jeff 77.564
We will receive the formatted output:
No Name Mark 1 Sarath 80.35 2 James 91.00 3 Jeff 77.56
%s
, %c
, %d
, and %f
are format substitution characters for which an argument can be placed after the quoted format string.
%-5s
can be described as a string substitution with left alignment (-
represents left alignment) with width equal to 5
. If -
was not specified, the string would have been aligned to the right. The width specifies the number of characters reserved for that variable. For Name
, the width reserved is 10
. Hence, any name will reside within the 10-character width reserved for it and the rest of the characters will be filled with space up to 10 characters in total.
For floating point numbers, we can pass additional parameters to round off the decimal places.
For marks, we have formatted the string as %-4.2f
, where .2
specifies rounding off to two decimal places. Note that for every line of the format string a newline (\n
) is issued.
While using flags for echo
and printf
, always make sure that the flags appear before any strings in the command, otherwise Bash will consider the flags as another string.
By default, echo
has a newline appended at the end of its output text. This can be avoided by using the -n
flag. echo
can also accept escape sequences in double-quoted strings as an argument. When using escape sequences, use echo
as echo -e "string containing escape sequences"
. For example:
echo -e "1\t2\t3" 1 2 3
Producing a colored output on the terminal is very interesting and is achieved by using escape sequences.
Colors are represented by color codes, some examples being, reset = 0, black = 30, red = 31, green = 32, yellow = 33, blue = 34, magenta = 35, cyan = 36, and white = 37.
To print a colored text, enter the following command:
echo -e "\e[1;31m This is red text \e[0m"
Here, \e[1;31m
is the escape string that sets the color to red and \e[0m
resets the color back. Replace 31
with the required color code.
For a colored background, reset = 0, black = 40, red = 41, green = 42, yellow = 43, blue = 44, magenta = 45, cyan = 46, and white=47, are the color codes that are commonly used.
To print a colored background, enter the following command:
echo -e "\e[1;42m Green Background \e[0m"
Variables are essential components of every programming language and are used to hold varying data. Scripting languages usually do not require variable type declaration before its use as they can be assigned directly. In Bash, the value for every variable is string, regardless of whether we assign variables with quotes or without quotes. Furthermore, there are variables used by the shell environment and the operating environment to store special values, which are called environment variables. Let us look at how to play with some of these variables in this recipe.
Variables are named with the usual naming constructs. When an application is executing, it will be passed a set of variables called environment variables. To view all the environment variables related to a terminal, issue the env
command. For every process, environment variables in its runtime can be viewed by:
cat /proc/$PID/environ
Set PID
with a process ID of the process (PID
always takes an integer value).
For example, assume that an application called gedit
is running. We can obtain the process ID of gedit with the pgrep
command as follows:
$ pgrep gedit 12501
You can obtain the environment variables associated with the process by executing the following command:
$ cat /proc/12501/environ GDM_KEYBOARD_LAYOUT=usGNOME_KEYRING_PID=1560USER=slynuxHOME=/home/slynux
Note
Note that many environment variables are stripped off for convenience. The actual output may contain numerous variables.
The aforementioned command returns a list of environment variables and their values. Each variable is represented as a name=value pair and are separated by a null character (\0
). If you can substitute the \0
character with \n
, you can reformat the output to show each variable=value pair in each line. Substitution can be made using the tr
command as follows:
$ cat /proc/12501/environ | tr '\0' '\n'
Now, let us see how to assign and manipulate variables and environment variables.
A variable can be assigned as follows:
var=value
var
is the name of a variable and value
is the value to be assigned. If value
does not contain any space character (such as space), it need not be enclosed in quotes, Otherwise it is to be enclosed in single or double quotes.
Note that var = value
and var=value
are different. It is a usual mistake to write var =value
instead of var=value
. The later one is the assignment operation, whereas the earlier one is an equality operation.
Printing contents of a variable is done using by prefixing $
with the variable name as follows:
var="value" #Assignment of value to variable var. echo $var
Or:
echo ${var}
We will receive an output as follows:
value
We can use variable values inside printf
or echo
in double quotes:
#!/bin/bash #Filename :variables.sh fruit=apple count=5 echo "We have $count ${fruit}(s)"
The output will be as follows:
We have 5 apple(s)
Environment variables are variables that are not defined in the current process, but are received from the parent processes. For example, HTTP_PROXY
is an environment variable. This variable defines which proxy server should be used for an Internet connection.
Usually, it is set as:
HTTP_PROXY=192.168.1.23:3128 export HTTP_PROXY
The export
command is used to set the env
variable. Now any application, executed from the current shell script, will receive this variable. We can export custom variables for our own purposes in an application or shell script that is executed. There are many standard environment variables that are available for the shell by default.
For example, PATH
. A typical PATH
variable will contain:
$ echo $PATH /home/slynux/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games
When given a command for execution, the shell automatically searches for the executable in the list of directories in the PATH
environment variable (directory paths are delimited by the ":" character). Usually, $PATH
is defined in /etc/environment
or /etc/profile
or ~/.bashrc
. When we need to add a new path to the PATH
environment, we use:
export PATH="$PATH:/home/user/bin"
Or, alternately, we can use:
$ PATH="$PATH:/home/user/bin" $ export PATH $ echo $PATH /home/slynux/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/home/user/bin
Here we have added /home/user/bin
to PATH
.
Some of the well-known environment variables are HOME
, PWD
, USER
, UID
, SHELL
, and so on.
Let us see more tips associated with standard and environment variables.
Get the length of a variable value using the following command:
length=${#var}
For example:
$ var=12345678901234567890$ echo ${#var} 20
The length
parameter will bear the number of characters in the string.
To identify the shell which is currently being used, we can use the SHELL
variable, like so:
echo $SHELL
Or:
echo $0
For example:
$ echo $SHELL /bin/bash $ echo $0 /bin/bash
UID
is an important environment variable that can be used to check whether the current script has been run as a root user or regular user. For example:
If [ $UID -ne 0 ]; then echo Non root user. Please run as root. else echo Root user fi
When we open a terminal or run a shell, we see a prompt string such as user@hostname: /home/$
. Different GNU/Linux distributions have slightly different prompts and different colors. We can customize the prompt text using the PS1
environment variable. The default prompt text for the shell is set using a line in the ~/.bashrc
file.
We can list the line used to set the
PS1
variable as follows:$ cat ~/.bashrc | grep PS1 PS1='${debian_chroot:+($debian_chroot)}\u@\h:\w\$ '
To set a custom prompt string, enter the following command:
slynux@localhost: ~$ PS1="PROMPT>" PROMPT> Type commands here # Prompt string changed.
We can use colored text using the special escape sequences such as
\e[1;31
(refer to the Printing in the terminal recipe of this chapter).
There are also certain special characters that expand to system parameters. For example, \u
expands to username, \h
expands to hostname, and \w
expands to the current working directory.
Environment variables are often used to store a list of paths of where to search for executables, libraries, and so on. Examples are $PATH
, $LD_LIBRARY_PATH
, which will typically look like this:
PATH=/usr/bin;/bin LD_LIBRARY_PATH=/usr/lib;/lib
This essentially means that whenever the shell has to execute binaries, it will first look into /usr/bin
followed by /bin
.
A very common task that one has to do when building a program from source and installing to a custom path is to add its bin
directory to the PATH
environment variable. Let's say in this case we install myapp to /opt/myapp
, which has binaries in a directory called bin
and libraries in lib
.
A way to do this is to say it as follows:
export PATH=/opt/myapp/bin:$PATH export LD_LIBRARY_PATH=/opt/myapp/lib;$LD_LIBRARY_PATH
PATH
and LD_LIBRARY_PATH
should now look something like this:
PATH=/opt/myapp/bin:/usr/bin:/bin LD_LIBRARY_PATH=/opt/myapp/lib:/usr/lib;/lib
However, we can make this easier by adding this function in .bashrc-
:
prepend() { [ -d "$2" ] && eval $1=\"$2':'\$$1\" && export $1; }
This can be used in the following way:
prepend PATH /opt/myapp/bin prepend LD_LIBRARY_PATH /opt/myapp/lib
We define a function called prepend()
, which first checks if the directory specified by the second parameter to the function exists. If it does, the eval
expression sets the variable with the name in the first parameter equal to the second parameter string followed by :
(the path separator) and then the original value for the variable.
However, there is one caveat, if the variable is empty when we try to prepend, there will be a trailing :
at the end. To fix this, we can modify the function to look like this:
prepend() { [ -d "$2" ] && eval $1=\"$2\$\{$1:+':'\$$1\}\" && export $1 ; }
Note
In this form of the function, we introduce a shell parameter expansion of the form:
${parameter:+expression}
This expands to expression
if parameter is set and is not null.
With this change, we take care to try to append :
and the old value if, and only if, the old value existed when trying to prepend.
Arithmetic operations are an essential requirement for every programming language. In this recipe, we will explore various methods for performing arithmetic operations in shell.
The Bash shell environment can perform basic arithmetic operations using the commands let
, (( ))
, and []
. The two utilities expr
and bc
are also very helpful in performing advanced operations.
A numeric value can be assigned as a regular variable assignment, which is stored as a string. However, we use methods to manipulate as numbers:
#!/bin/bash no1=4; no2=5;
The
let
command can be used to perform basic operations directly. While usinglet
, we use variable names without the$
prefix, for example:let result=no1+no2 echo $result
Increment operation:
$ let no1++
Decrement operation:
$ let no1--
Shorthands:
let no+=6 let no-=6
These are equal to
let no=no+6
andlet no=no-6
respectively.Alternate methods:
The
[]
operator can be used in the same way as thelet
command as follows:result=$[ no1 + no2 ]
Using the
$
prefix inside[]
operators are legal, for example:result=$[ $no1 + 5 ]
(( ))
can also be used.$
prefixed with a variable name is used when(( ))
operator is used, as follows:result=$(( no1 + 50 ))
expr
result=`expr 3 + 4` result=$(expr $no1 + 5)
All of the preceding methods do not support floating point numbers, and operate on integers only.
bc
, the precision calculator is an advanced utility for mathematical operations. It has a wide range of options. We can perform floating point operations and use advanced functions as follows:echo "4 * 0.56" | bc 2.24 no=54; result=`echo "$no * 1.5" | bc` echo $result 81.0
Additional parameters can be passed to
bc
with prefixes to the operation with semicolon as delimiters throughstdin
.Decimal places scale with bc: In the following example the
scale=2
parameter sets the number of decimal places to2
. Hence, the output ofbc
will contain a number with two decimal places:echo "scale=2;3/8" | bc 0.37
Base conversion with bc: We can convert from one base number system to another one. Let us convert from decimal to binary, and binary to octal:
#!/bin/bash Desc: Number conversion no=100 echo "obase=2;$no" | bc 1100100 no=1100100 echo "obase=10;ibase=2;$no" | bc 100
Calculating squares and square roots can be done as follows:
echo "sqrt(100)" | bc #Square root echo "10^10" | bc #Square
File descriptors are integers that are associated with file input and output. They keep track of opened files. The best-known file descriptors are stdin
, stdout
, and stderr
. We even can redirect the contents of one file descriptor to another. This recipe shows examples on how to manipulate and redirect with file descriptors.
While writing scripts we use standard input (stdin
), standard output (stdout
), and standard error (stderr
) frequently. Redirection of an output to a file by filtering the contents is one of the essential things we need to perform. While a command outputs some text, it can be either an error or an output (nonerror) message. We cannot distinguish whether it is output text or an error text by just looking at it. However, we can handle them with file descriptors. We can extract text that is attached to a specific descriptor.
File descriptors are integers associated with an opened file or data stream. File descriptors 0, 1, and 2 are reserved as follows:
0:
stdin
(standard input)1:
stdout
(standard output)2:
stderr
(standard error)
Redirecting or saving output text to a file can be done as follows:
$ echo "This is a sample text 1" > temp.txt
This would store the echoed text in
temp.txt
by truncating the file, the contents will be emptied before writing.To append text to a file, consider the following example:
$ echo "This is sample text 2" >> temp.txt
You can view the contents of the file as follows:
$ cat temp.txt This is sample text 1 This is sample text 2
Let us see what a standard error is and how you can redirect it.
stderr
messages are printed when commands output an error message. Consider the following example:$ ls + ls: cannot access +: No such file or directory
Here
+
is an invalid argument and hence an error is returned.Tip
Successful and unsuccessful commands
When a command returns after an error, it returns a nonzero exit status. The command returns zero when it terminates after successful completion. The return status can be read from special variable
$?
(runecho $?
immediately after the command execution statement to print the exit status).The following command prints the
stderr
text to the screen rather than to a file (and because there is nostdout
output,out.txt
will be empty):$ ls + > out.txt ls: cannot access +: No such file or directory
In the following command, we redirect
stderr
toout.txt
:$ ls + 2> out.txt # works
You can redirect
stderr
exclusively to a file andstdout
to another file as follows:$ cmd 2>stderr.txt 1>stdout.txt
It is also possible to redirect
stderr
andstdout
to a single file by convertingstderr
tostdout
using this preferred method:$ cmd 2>&1 output.txt
Or the alternate approach:
$ cmd &> output.txt
Sometimes, the output may contain unnecessary information (such as debug messages). If you don't want the output terminal burdened with the
stderr
details then you should redirect thestderr
output to/dev/null
, which removes it completely. For example, consider that we have three filesa1
,a2
, anda3
. However,a1
does not have the read-write-execute permission for the user. When you need to print the contents of files starting witha
, we use thecat
command. Set up the test files as follows:$ echo a1 > a1 $ cp a1 a2 ; cp a2 a3; $ chmod 000 a1 #Deny all permissions
While displaying contents of the files using wildcards (
a*
), it will show an error message for filea1
as it does not have the proper read permission:$ cat a* cat: a1: Permission denied a1 a1
Here,
cat: a1: Permission denied
belongs to thestderr
data. We can redirect thestderr
data into a file, whereasstdout
remains printed in the terminal. Consider the following code:$ cat a* 2> err.txt #stderr is redirected to err.txt a1 a1 $ cat err.txt cat: a1: Permission denied
Take a look at the following code:
$ cmd 2>/dev/null
When redirection is performed for
stderr
orstdout
, the redirected text flows into a file. As the text has already been redirected and has gone into the file, no text remains to flow to the next command through pipe (|
), and it appears to the next set of command sequences throughstdin
.However, there is a way to redirect data to a file, as well as provide a copy of redirected data as
stdin
for the next set of commands. This can be done using thetee
command. For example, to printstdout
in the terminal as well as redirectstdout
into a file, the syntax fortee
is as follows:command | tee FILE1 FILE2
In the following code, the
stdin
data is received by thetee
command. It writes a copy ofstdout
to theout.txt
file and sends another copy asstdin
for the next command. Thecat -n
command puts a line number for each line received fromstdin
and writes it intostdout
:$ cat a* | tee out.txt | cat -n cat: a1: Permission denied 1a1 2a1
Examine the contents of
out.txt
as follows:$ cat out.txt a1 a1
Note that
cat: a1: Permission denied
does not appear because it belongs tostderr
. Thetee
command can read fromstdin
only.By default, the
tee
command overwrites the file, but it can be used with appended options by providing the-a
option, for example,$ cat a* | tee -a out.txt | cat -n
.Commands appear with arguments in the format:
command FILE1 FILE2 …
or simplycommand FILE
.We can use
stdin
as a command argument. It can be done by using-
as the filename argument for the command as follows:$ cmd1 | cmd2 | cmd -
For example:
$ echo who is this | tee - who is this who is this
Alternately, we can use
/dev/stdin
as the output filename to usestdin
.Similarly, use
/dev/stderr
for standard error and/dev/stdout
for standard output. These are special device files that correspond tostdin
,stderr
, andstdout
.
For output redirection, >
and >>
operators are different. Both of them redirect text to a file, but the first one empties the file and then writes to it, whereas the later one adds the output to the end of the existing file.
When we use a redirection operator, the output won't print in the terminal but it is directed to a file. When redirection operators are used, by default, they operate on standard output. To explicitly take a specific file descriptor, you must prefix the descriptor number to the operator.
>
is equivalent to 1>
and similarly it applies for >>
(equivalent to 1>>
).
When working with errors, the stderr
output is dumped to the /dev/null
file. ./dev/null
is a special device file where any data received by the file is discarded. The null device is often known as a black hole as all the data that goes into it is lost forever.
A command that reads stdin
for input can receive data in multiple ways. Also, it is possible to specify file descriptors of our own using cat
and pipes, for example:
$ cat file | cmd $ cmd1 | cmd
Sometimes we need to redirect a block of text (multiple lines of text) as standard input. Consider a particular case where the source text is placed within the shell script. A practical usage example is writing a logfile header data. It can be performed as follows:
#!/bin/bash cat<<EOF>log.txt LOG FILE HEADER This is a test log file Function: System statistics EOF
The lines that appear between cat <<EOF >log.txt
and the next EOF
line will appear as the stdin
data. Print the contents of log.txt
as follows:
$ cat log.txt LOG FILE HEADER This is a test log file Function: System statistics
A file descriptor is an abstract indicator for accessing a file. Each file access is associated with a special number called a file descriptor. 0, 1, and 2 are reserved descriptor numbers for stdin
, stdout
, and stderr
.
We can create our own custom file descriptors using the exec
command. If you are already familiar with file programming with any other programming language, you might have noticed modes for opening files. Usually, the following three modes are used:
Read mode
Write with truncate mode
Write with append mode
<
is an operator used to read from the file to stdin
. >
is the operator used to write to a file with truncation (data is written to the target file after truncating the contents). >>
is an operator used to write to a file by appending (data is appended to the existing file contents and the contents of the target file will not be lost). File descriptors can be created with one of the three modes.
Create a file descriptor for reading a file, as follows:
$ exec 3<input.txt # open for reading with descriptor number 3
We could use it in the following way:
$ echo this is a test line > input.txt $ exec 3<input.txt
Now you can use file descriptor 3
with commands. For example, we will use cat <&3
as follows:
$ cat<&3 this is a test line
If a second read is required, we cannot re-use the file descriptor 3
. It is required that we reassign the file descriptor 3
for read using exec
for making a second read.
Create a file descriptor for writing (truncate mode) as follows:
$ exec 4>output.txt # open for writing
For example:
$ exec 4>output.txt $ echo newline >&4 $ cat output.txt newline
Create a file descriptor for writing (append mode) as follows:
$ exec 5>>input.txt
For example:
$ exec 5>>input.txt $ echo appended line >&5 $ cat input.txt newline appended line
Arrays are a very important component for storing a collection of data as separate entities using indexes. Regular arrays can use only integers as their array index. On the other hand, Bash also supports associative arrays that can take a string as their array index. Associative arrays are very useful in many types of manipulations where having a string index makes more sense. In this recipe, we will see how to use both of these.
An array can be defined in many ways. Define an array using a list of values in a line as follows:
array_var=(1 2 3 4 5 6) #Values will be stored in consecutive locations starting from index 0.
Alternately, define an array as a set of index-value pairs as follows:
array_var[0]="test1" array_var[1]="test2" array_var[2]="test3" array_var[3]="test4" array_var[4]="test5" array_var[5]="test6"
Print the contents of an array at a given index using the following commands:
echo ${array_var[0]} test1 index=5 echo ${array_var[$index]} test6
Print all of the values in an array as a list using the following commands:
$ echo ${array_var[*]} test1 test2 test3 test4 test5 test6
Alternately, you could use:
$ echo ${array_var[@]} test1 test2 test3 test4 test5 test6
Print the length of an array (the number of elements in an array) as follows:
$ echo ${#array_var[*]} 6
Associative arrays have been introduced to Bash from Version 4.0 and they are useful entities to solve many problems using the hashing technique. Let us go into more detail.
In an associative array, we can use any text data as an array index. Initially, a declaration statement is required to declare a variable name as an associative array. This can be done as follows:
$ declare -A ass_array
After the declaration, elements can be added to the associative array using two methods as follows:
By using inline index-value list method, we can provide a list of index-value pairs:
$ ass_array=([index1]=val1 [index2]=val2)
Alternately, you could use separate index-value assignments:
$ ass_array[index1]=val1 $ ass_array'index2]=val2
For example, consider the assignment of price for fruits using an associative array:
$ declare -A fruits_value $ fruits_value=([apple]='100dollars' [orange]='150 dollars')
Display the content of an array as follows:
$ echo "Apple costs ${fruits_value[apple]}" Apple costs 100 dollars
Arrays have indexes for indexing each of the elements. Ordinary and associative arrays differ in terms of index type. We can obtain the list of indexes in an array as follows:
$ echo ${!array_var[*]}
Or, we can also use:
$ echo ${!array_var[@]
In the previous fruits_value
array example, consider the following command:
$ echo ${!fruits_value[*]} orange apple
This will work for ordinary arrays too.
An
alias
is basically a shortcut that takes the place of typing a long-command sequence. In this recipe, we will see how to create aliases using the alias
command.
There are various operations you can perform on aliases, these are as follows:
An alias can be created as follows:
$ alias new_command='command sequence'
Giving a shortcut to the install command,
apt-get install
, can be done as follows:$ alias install='sudo apt-get install'
Therefore, we can use
install pidgin
instead ofsudo apt-get install pidgin
.The
alias
command is temporary; aliasing exists until we close the current terminal only. To keep these shortcuts permanent, add this statement to the~/.bashrc
file. Commands in~/.bashrc
are always executed when a new shell process is spawned:$ echo 'alias cmd="command seq"' >> ~/.bashrc
To remove an alias, remove its entry from
~/.bashrc
(if any) or use theunalias
command. Alternatively,alias example=
should unset the alias namedexample
.As an example, we can create an alias for
rm
so that it will delete the original and keep a copy in a backup directory:alias rm='cp $@ ~/backup && rm $@'
There are situations when aliasing can also be a security breach. See how to identify them.
The alias
command can be used to alias any important command, and you may not always want to run the command using the alias. We can ignore any aliases currently defined by escaping the command we want to run. For example:
$ \command
The \
character escapes the command, running it without any aliased changes. While running privileged commands on an untrusted environment, it is always good security practice to ignore aliases by prefixing the command with \
. The attacker might have aliased the privileged command with his/her own custom command to steal the critical information that is provided by the user to the command.
While writing command-line shell scripts, we will often need to heavily manipulate information about the current terminal, such as the number of columns, rows, cursor positions, masked password fields, and so on. This recipe helps in collecting and manipulating terminal settings.
tput
and stty
are utilities that can be used for terminal manipulations. Let us see how to use them to perform different tasks.
There are specific information you can gather about the terminal as shown in the following list:
Get the number of columns and rows in a terminal by using the following commands:
tput cols tput lines
To print the current terminal name, use the following command:
tput longname
To move the cursor to a 100,100 position, you can enter:
tput cup 100 100
Set the background color for the terminal using the following command:
tputsetb n
n
can be a value in the range of 0 to 7.Set the foreground color for text by using the following command:
tputsetf n
n
can be a value in the range of 0 to 7.To make text bold use this:
tput bold
To start and end underlining use this:
tput smul tput rmul
To delete from the cursor to the end of the line use the following command:
tputed
While typing a password, we should not display the characters typed. In the following example, we will see how to do it using
stty
:#!/bin/sh #Filename: password.sh echo -e "Enter password: " stty -echo read password stty echo echo echo Password read.
Many applications require printing dates in different formats, setting date and time, and performing manipulations based on date and time. Delays are commonly used to provide a wait time (such as 1 second) during the program execution. Scripting contexts, such as monitoring a task every 5 seconds, demands the understanding of writing delays in a program. This recipe will show you how to work with dates and time delays.
Dates can be printed in variety of formats. We can also set dates from the command line. In Unix-like systems, dates are stored as an integer, which denotes the number of seconds since 1970-01-01 00:00:00 UTC. This is called epoch or Unix time . Let us see how to read dates and set them.
It is possible to read the dates in different formats and also to set the date. This can be accomplished with these steps:
You can read the date as follows:
$ date Thu May 20 23:09:04 IST 2010
The epoch time can be printed as follows:
$ date +%s 1290047248
We can find out epoch from a given formatted date string. You can use dates in multiple date formats as input. Usually, you don't need to bother about the date string format that you use if you are collecting the date from a system log or any standard application generated output. Convert the date string into epoch as follows:
$ date --date "Thu Nov 18 08:07:21 IST 2010" +%s 1290047841
The
--date
option is used to provide a date string as input. However, we can use any date formatting options to print the output. Feeding the input date from a string can be used to find out the weekday, given the date.For example:
$ date --date "Jan 20 2001" +%A Saturday
The date format strings are listed in the table mentioned in the How it works… section:
Use a combination of format strings prefixed with
+
as an argument for thedate
command to print the date in the format of your choice. For example:$ date "+%d %B %Y" 20 May 2010
We can set the date and time as follows:
# date -s "Formatted date string"
For example:
# date -s "21 June 2009 11:01:22"
Sometimes we need to check the time taken by a set of commands. We can display it using the following code:
#!/bin/bash #Filename: time_take.sh start=$(date +%s) commands; statements; end=$(date +%s) difference=$(( end - start)) echo Time taken to execute commands is $difference seconds.
While considering dates and time, epoch is defined as the number of seconds that have elapsed since midnight proleptic Coordinated Universal Time (UTC) of January 1, 1970, not counting leap seconds. Epoch time is very useful when you need to calculate the difference between two dates or time. You may find out the epoch times for two given timestamps and take the difference between the epoch values. Therefore, you can find out the total number of seconds between two dates.
To write a date format to get the output as required, use the following table:
Date component |
Format |
---|---|
Weekday |
|
Month |
|
Day |
|
Date in format (mm/dd/yy) |
|
Year |
|
Hour |
|
Minute |
|
Second |
|
Nano second |
|
Epoch Unix time in seconds |
|
Producing time intervals is very essential when writing monitoring scripts that execute in a loop. Let us see how to generate time delays.
To delay execution in a script for a particular period of time, use sleep:$ sleepno_of_seconds
. For example, the following script counts from 0 to 40 by using tput
and sleep
:
#!/bin/bash #Filename: sleep.sh echo -n Count: tput sc count=0; while true; do if [ $count -lt 40 ]; then let count++; sleep 1; tput rc tput ed echo -n $count; else exit 0; fi done
In the preceding example, a variable count is initialized to 0 and is incremented on every loop execution. The echo
statement prints the text. We use tput sc
to store the cursor position. On every loop execution we write the new count in the terminal by restoring the cursor position for the number. The cursor position is restored using tput rc
. This clears text from the current cursor position to the end of the line, so that the older number can be cleared and the count can be written. A delay of 1 second is provided in the loop by using the sleep
command.
Debugging is one of the critical features that every programming language should implement to produce race-back information when something unexpected happens. Debugging information can be used to read and understand what caused the program to crash or to act in an unexpected fashion. Bash provides certain debugging options that every sysadmin should know. This recipe shows how to use these.
We can either use Bash's inbuilt debugging tools or write our scripts in such a manner that they become easy to debug, here's how:
Add the
-x
option to enable debug tracing of a shell script as follows:$ bash -x script.sh
Running the script with the
-x
flag will print each source line with the current status. Note that you can also usesh -x script
.Debug only portions of the script using
set -x
andset +x
. For example:#!/bin/bash #Filename: debug.sh for i in {1..6}; do set -x echo $i set +x done echo "Script executed"
In the preceding script, the debug information for
echo $i
will only be printed, as debugging is restricted to that section using-x
and+x
.The aforementioned debugging methods are provided by Bash built-ins. But they always produce debugging information in a fixed format. In many cases, we need debugging information in our own format. We can set up such a debugging style by passing the
_DEBUG
environment variable.Look at the following example code:
#!/bin/bash function DEBUG() { [ "$_DEBUG" == "on" ] && $@ || : } for i in {1..10} do DEBUG echo $i done
We can run the above script with debugging set to "on" as follows:
$ _DEBUG=on ./script.sh
We prefix
DEBUG
before every statement where debug information is to be printed. If_DEBUG=on
is not passed to the script, debug information will not be printed. In Bash, the command:
tells the shell to do nothing.
The -x
flag outputs every line of script as it is executed to stdout
. However, we may require only some portions of the source lines to be observed such that commands and arguments are to be printed at certain portions. In such conditions we can use set builtin
to enable and disable debug printing within the script.
set -x
: This displays arguments and commands upon their executionset +x
: This disables debuggingset -v
: This displays input when they are readset +v
: This disables printing input
Like any other scripting languages, Bash also supports functions. Let us see how to define and use functions.
We can create functions to perform tasks and we can also create functions that take parameters (also called arguments) as you can see in the following steps:
A function can be defined as follows:
function fname() { statements; } Or alternately, fname() { statements; }
A function can be invoked just by using its name:
$ fname ; # executes function
Arguments can be passed to functions and can be accessed by our script:
fname arg1 arg2 ; # passing args
Following is the definition of the function
fname
. In thefname
function, we have included various ways of accessing the function arguments.fname() { echo $1, $2; #Accessing arg1 and arg2 echo "$@"; # Printing all arguments as list at once echo "$*"; # Similar to $@, but arguments taken as single entity return 0; # Return value }
Similarly, arguments can be passed to scripts and can be accessed by
script:$0
(the name of the script):$1
is the first argument$2
is the second argument$n
is the nth argument"$@"
expands as"$1" "$2" "$3"
and so on"$*"
expands as"$1c$2c$3"
, wherec
is the first character of IFS"$@"
is used more often than"$*"
since the former provides all arguments as a single string
Let us explore through more tips on Bash functions.
Functions in Bash also support recursion (the function that can call itself). For example, F() { echo $1; F hello; sleep 1; }
.
Tip
Fork bomb
We can write a recursive function, which is basically a function that calls itself:
:(){ :|:& };:
It infinitely spawns processes and ends up in a denial-of-service attack. &
is postfixed with the function call to bring the subprocess into the background. This is a dangerous code as it forks processes and, therefore, it is called a fork bomb.
You may find it difficult to interpret the preceding code. See the Wikipedia page http://en.wikipedia.org/wiki/Fork_bomb for more details and interpretation of the fork bomb.
It can be prevented by restricting the maximum number of processes that can be spawned from the config
file at /etc/security/limits.conf
.
A function can be exported—like environment variables—using export
, such that the scope of the function can be extended to subprocesses, as follows:
export -f fname
We can get the return value of a command or function in the following way:
cmd; echo $?;
$?
will give the return value of the command cmd
.
The return value is called exit status . It can be used to analyze whether a command completed its execution successfully or unsuccessfully. If the command exits successfully, the exit status will be zero, otherwise it will be a nonzero value.
We can check whether a command terminated successfully or not by using the following script:
#!/bin/bash #Filename: success_test.sh CMD="command" #Substitute with command for which you need to test the exit status $CMD if [ $? -eq 0 ]; then echo "$CMD executed successfully" else echo "$CMD terminated unsuccessfully" fi
Arguments to commands can be passed in different formats. Suppose -p
and-v
are the options available and -k N
is another option that takes a number. Also, the command takes a filename as argument. It can be executed in multiple ways as shown:
$ command -p -v -k 1 file
$ command -pv -k 1 file
$ command -vpk 1 file
$ command file -pvk 1
One of the best-designed features of shell scripting is the ease of combining many commands or utilities to produce output. The output of one command can appear as the input of another, which passes its output to another command, and so on. The output of this combination can be read in a variable. This recipe illustrates how to combine multiple commands and how its output can be read.
Input is usually fed into a command through stdin
or arguments. Output appears as stderr
or stdout
. While we combine multiple commands, we usually use stdin
to give input and stdout
to provide an output.
In this context, the commands are called filters
. We connect each filter using pipes, the piping operator being |
. An example is as follows:
$ cmd1 | cmd2 | cmd3
Here we combine three commands. The output of cmd1
goes to cmd2
and output of cmd2
goes to cmd3
and the final output (which comes out of cmd3
) will be printed, or it can be directed to a file.
We typically use pipes and use them with the subshell method for combining outputs of multiple files. Here's how:
Let us start with combining two commands:
$ ls | cat -n > out.txt
Here the output of
ls
(the listing of the current directory) is passed tocat –n
, which in turn puts line numbers to the input received throughstdin
. Therefore, its output is redirected to theout.txt
file.We can read the output of a sequence of commands combined by pipes as follows:
cmd_output=$(COMMANDS)
This is called subshell method . For example:
cmd_output=$(ls | cat -n) echo $cmd_output
Another method, called back quotes (some people also refer to it as back tick ) can also be used to store the command output as follows:
cmd_output=`COMMANDS`
For example:
cmd_output=`ls | cat -n` echo $cmd_output
Back quote is different from the single-quote character. It is the character on the ~ button in the keyboard.
There are multiple ways of grouping commands. Let us go through a few of them.
Subshells are separate processes. A subshell can be defined using the ( )
operators as follows:
pwd; (cd /bin; ls); pwd;
When some commands are executed in a subshell, none of the changes occur in the current shell; changes are restricted to the subshell. For example, when the current directory in a subshell is changed using the cd
command, the directory change is not reflected in the main shell environment.
The pwd
command prints the path of the working directory.
The cd
command changes the current directory to the given directory path.
Suppose we are reading the output of a command to a variable using a subshell or the back quotes method. We always quote them in double quotes to preserve the spacing and newline character (\n
). For example:
$ cat text.txt 1 2 3 $ out=$(cat text.txt) $ echo $out 1 2 3 # Lost \n spacing in 1,2,3 $ out="$(cat tex.txt)" $ echo$out 1 2 3
read
is an important Bash command to read text from the keyboard or standard input. We can use read
to interactively read an input from the user, but read
is capable of much more. Most of the input libraries in any programming language read the input from the keyboard; but string input termination is done when return is pressed. There are certain critical situations when return cannot be pressed, but the termination is done based on a number of characters or a single character. For example, in a game, a ball is moved upward when +
is pressed. Pressing +
and then pressing return every time to acknowledge the +
press is not efficient. In this recipe we will use the read
command that provides a way to accomplish this task without having to press return.
You can use various options of the read
command to obtain different results as shown in the following steps:
The following statement will read n characters from input into the
variable_name
variable:read -n number_of_chars variable_name
For example:
$ read -n 2 var $ echo $var
Read a password in the nonechoed mode as follows:
read -s var
Display a message with
read
using:read -p "Enter input:" var
Read the input after a timeout as follows:
read -t timeout var
For example:
$ read -t 2 var #Read the string that is typed within 2 seconds into variable var.
Use a delimiter character to end the input line as follows:
read -d delim_char var
For example:
$ read -d ":" var hello:#var is set to hello
When using your shell for everyday tasks, there will be cases where a command might succeed only after some conditions are met, or the operation depends on an external event (such as a file being available to download). In such cases, one might want to run a command repeatedly until it succeeds.
Define a function in the following way:
repeat() { while true do $@ && return done }
Or, add this to your shell's rc
file for ease of use:
repeat() { while true; do $@ && return; done }
We create a function called repeat
that has an infinite while
loop, which attempts to run the command passed as a parameter (accessed by $@
) to the function. It then returns if the command was successful, thereby exiting the loop.
We saw a basic way to run commands until they succeed. Let us see what we can do to make things more efficient.
On most modern systems, true is implemented as a binary in /bin
. This means that each time the aforementioned while
loop runs, the shell has to spawn a process. To avoid this, we can use the :
shell built-in, which always returns an exit code 0:
repeat() { while :; do $@ && return; done }
Though not as readable, this is certainly faster than the first approach.
Let's say you are using repeat()
to download a file from the Internet which is not available right now, but will be after some time. An example would be:
repeat wget -c http://www.example.com/software-0.1.tar.gz
In the current form, we will be sending too much traffic to the web server at www.example.com
, which causes problems to the server (and maybe even to you, if say the server blacklists your IP for spam). To solve this, we can modify the function and add a small delay as follows:
repeat() { while :; do $@ && return; sleep 30; done }
This will cause the command to run every 30 seconds.
The internal field separator (IFS) is an important concept in shell scripting. It is very useful while manipulating text data. We will now discuss delimiters that separate different data elements from single data stream. An internal field separator is a delimiter for a special purpose. An internal field separator is an environment variable that stores delimiting characters. It is the default delimiter string used by a running shell environment.
Consider the case where we need to iterate through words in a string or comma separated values (CSV). In the first case we will use IFS=" "
and in the second, IFS=","
. Let us see how to do it.
Consider the case of CSV data:
data="name,sex,rollno,location" To read each of the item in a variable, we can use IFS. oldIFS=$IFS IFS=, now, for item in $data; do echo Item: $item done IFS=$oldIFS
The output is as follows:
Item: name Item: sex Item: rollno Item: location
The default value of IFS is a space component (newline, tab, or a space character).
When IFS is set as ,
the shell interprets the comma as a delimiter character, therefore, the $item
variable takes substrings separated by a comma as its value during the iteration.
If IFS is not set as ,
then it would print the entire data as a single string.
Let us go through another example usage of IFS by taking the /etc/passwd
file into consideration. In the /etc/passwd
file, every line contains items delimited by ":"
. Each line in the file corresponds to an attribute related to a user.
Consider the input: root:x:0:0:root:/root:/bin/bash
. The last entry on each line specifies the default shell for the user. To print users and their default shells, we can use the IFS hack as follows:
#!/bin/bash #Desc: Illustration of IFS line="root:x:0:0:root:/root:/bin/bash" oldIFS=$IFS; IFS=":" count=0 for item in $line; do [ $count -eq 0 ] && user=$item; [ $count -eq 6 ] && shell=$item; let count++ done; IFS=$oldIFS echo $user\'s shell is $shell;
The output will be:
root's shell is /bin/bash
Loops are very useful in iterating through a sequence of values. Bash provides many types of loops. Let us see how to use them:
Using a
for
loop:for var in list; do commands; # use $var done list can be a string, or a sequence.
We can generate different sequences easily.
echo {1..50}
can generate a list of numbers from 1 to 50.echo {a..z}
or{A..Z}
or{a..h}
can generate lists of alphabets. Also, by combining these we can concatenate data.In the following code, in each iteration, the variable
i
will hold a character in the rangea
toz
:for i in {a..z}; do actions; done;
The
for
loop can also take the format of thefor
loop in C. For example:for((i=0;i<10;i++)) { commands; # Use $i }
while condition do commands; done
For an infinite loop, use
true
as the condition.A special loop called
until
is available with Bash. This executes the loop until the given condition becomes true. For example:x=0; until [ $x -eq 9 ]; # [ $x -eq 9 ] is the condition do let x++; echo $x; done
Flow control in a program is handled by comparison and test statements. Bash also comes with several options to perform tests that are compatible with the Unix system-level features. We can use if
, if else
, and logical operators to perform tests and certain comparison operators to compare data items. There is also a command called test
available to perform tests. Let us see how to use these.
We will have a look at all the different methods used for comparisons and performing tests:
Using an
if
condition:if condition; then commands; fi
Using
else if
andelse
:if condition; then commands; else if condition; then commands; else commands; fi
Note
Nesting is also possible with
if
andelse
. Theif
conditions can be lengthy, to make them shorter we can use logical operators as follows:[ condition ] && action; # action
executes if the condition is true[ condition ] || action; # action
executes if the condition is false
&&
is the logical AND operation and||
is the logical OR operation. This is a very helpful trick while writing Bash scripts.Performing mathematical comparisons: Usually conditions are enclosed in square brackets
[]
. Note that there is a space between[
or]
and operands. It will show an error if no space is provided. An example is as follows:[$var -eq 0 ] or [ $var -eq 0]
Performing mathematical conditions over variables or values can be done as follows:
[ $var -eq 0 ] # It returns true when $var equal to 0. [ $var -ne 0 ] # It returns true when $var is not equal to 0
Other important operators are as follows:
-gt
: Greater than-lt
: Less than-ge
: Greater than or equal to-le
: Less than or equal to
Multiple test conditions can be combined as follows:
[ $var1 -ne 0 -a $var2 -gt 2 ] # using and -a [ $var1 -ne 0 -o var2 -gt 2 ] # OR -o
Filesystem related tests: We can test different filesystem-related attributes using different condition flags as follows:
[ -f $file_var ]
: This returns true if the given variable holds a regular file path or filename[ -x $var ]
: This returns true if the given variable holds a file path or filename that is executable[ -d $var ]
: This returns true if the given variable holds a directory path or directory name[ -e $var ]
: This returns true if the given variable holds an existing file[ -c $var ]
: This returns true if the given variable holds the path of a character device file[ -b $var ]
: This returns true if the given variable holds the path of a block device file[ -w $var ]
: This returns true if the given variable holds the path of a file that is writable[ -r $var ]
: This returns true if the given variable holds the path of a file that is readable[ -L $var ]
: This returns true if the given variable holds the path of a symlink
An example of the usage is as follows:
fpath="/etc/passwd" if [ -e $fpath ]; then echo File exists; else echo Does not exist; fi
String comparisons: While using string comparison, it is best to use double square brackets, since the use of single brackets can sometimes lead to errors.
Two strings can be compared to check whether they are the same in the following manner:
[[ $str1 = $str2 ]]
: This returns true whenstr1
equalsstr2
, that is, the text contents ofstr1
andstr2
are the same[[ $str1 == $str2 ]]
: It is an alternative method for string equality check
We can check whether two strings are not the same as follows:
[[ $str1 != $str2 ]]
: This returns true whenstr1
andstr2
mismatch
We can find out the alphabetically smaller or larger string as follows:
[[ $str1 > $str2 ]]
: This returns true whenstr1
is alphabetically greater thanstr2
[[ $str1 < $str2 ]]
: This returns true whenstr1
is alphabetically lesser thanstr2
[[ -z $str1 ]]
: This returns true ifstr1
holds an empty string[[ -n $str1 ]]
: This returns true ifstr1
holds a nonempty string
It is easier to combine multiple conditions using logical operators such as
&&
and||
in the following code:if [[ -n $str1 ]] && [[ -z $str2 ]] ; then commands; fi
For example:
str1="Not empty " str2="" if [[ -n $str1 ]] && [[ -z $str2 ]]; then echo str1 is nonempty and str2 is empty string. fi
Output:
str1 is nonempty and str2 is empty string.
The test command can be used for performing condition checks. It helps to avoid usage of many braces. The same set of test conditions enclosed within []
can be used for the test command.
For example:
if [ $var -eq 0 ]; then echo "True"; fi can be written as if test $var -eq 0 ; then echo "True"; fi