Using Text Processing and Filters in Your Scripts

In the last chapter, you studied basic process management. You learned about the ps command. You also studied job management by using commands such as jobs, fg, bg, kill, and pkill, as well as various other tools, such as top, iostat, and vmstat. In this chapter, we will cover the following topics:

• Using more, less, head, and tail
• Using diff, cut, paste, comm, and uniq
• Working with grep
• Understanding standard input, standard output, and standard error
• Understanding various meta-characters and their usage

Text filtering tools

Normally, shell scripting involves report generation, which will include processing various text files and filtering their output to finally produce the desired results. Let's start discussing the two Linux commands, namely more and less:

• more: Sometimes we get a very large output on the screen for certain commands, which cannot be viewed completely in one screen. In such cases, we can use the more command to view the output text one page at a time. Add | more after the command, as follows:

    $ ll /dev | more

The | is called a pipe. You will learn more about it in the next chapters. In this command, pressing the spacebar will move the output on the screen one page at a time, or pressing Enter will move the screen one line at a time.

• less: Instead of more, if you use less, it will show a screen containing the full text all at once. We can move forward...

IO redirection

You will learn the very useful concept of I/O redirection in this section.

File descriptors

All I/O–including files, pipes, and sockets - are handled by the kernel via a mechanism called the file descriptor. A file descriptor is a small, unsigned integer, which is an index into a file-descriptor table maintained by the kernel, and used by the kernel to reference open files and I/O streams. Each process inherits its own file-descriptor table from its parent. The first three file descriptors are 0, 1, and 2. File descriptor 0 is standard input (stdin), 1 is standard output (stdout), and 2 is standard error (stderr). When you open a file, the next available descriptor is 3, and it will be assigned to the...

Pattern matching with the vi editor

To learn about pattern matching, we will ensure that the pattern that we will search for is highlighted when the pattern searched for is found. The configuration file for vi is /etc/vimrc.In the vi editor, use the following commands for various options:

Sr.	Commands	Description
1	:set hlsearch	Highlights the search pattern
2	:se[t] showmode	Shows when you are in insert mode
3	:se[t] ic	Ignores case when searching
4	:set noic	Shows a case-sensitive search

The user should open the file in vi, press the Esc button so that it enters command mode, and then type colon, followed by these commands. The following are the commands for pattern search and replace:

Sr.	Commands	Description
1	/pat	This searches for the pattern pat and places the cursor where the pattern occurs
2	/	This repeats the last search
3	:%s/old/new/g	Globally...

Pattern searching using grep

The command g/RE/p stands for globally search for the regular expression (RE) and print the line. The return statuses are 0 for success, 1 for pattern not found, and 2 for file not found:

    $ ps -ef | grep root

The preceding command will show all processes running currently whose user ID is root.

    $ ll /proc | grep "cpuinfo"

The preceding command will show the file with the name cpuinfo from the /proc directory.

    $ grep -lir "text" *          // show only file names containing text //   
    $ grep -ir "text" dir_name    // show lines of files //

We will try the following commands on file love.txt:

Meta-character	Function	Example	Description
^	Beginning-of-line anchor	'^mango'	Will display all lines beginning with mango
$	End-of-line anchor	'mango'$'	Will display all lines ending...

Summary

In this chapter, you have learned about using the more, less, head, and tail commands, and text processing tools such as cut, paste, comm, and uniq. You also have learned what standard input, standard output, and standard errors are. Finally, you have learned about meta-characters, and pattern matching using vi and grep. In the next chapter, you will learn about analysing the shell interpretation of commands, and you will learn about working with command substitution, command separators, and pipes.