Troubleshooting CentOS

During the course of this book, it is assumed that you will already have access to the basic tools associated with troubleshooting your server. Some of the more obscure tools will be mentioned and instructions will be given; however, for those who may or may not have access to the basic toolbox, as the root user you may want to begin by running the following command:

# yum groupinstall "Base" "Development Libraries" "Development Tools"

This action, if and when confirmed, will begin to download and install the common development tools, libraries, and base components of a CentOS server system. It also contains the relevant utilities required by RPM, additional text editors, and packages required to compile custom packages.

So, having prepared the system with the necessary tools and utilities, we shall begin in earnest by taking a closer look at the hardware. To do this, it is recommended that you continue with root access to the system in question.

As a matter of principle, most people will tend to suggest that all system information can be categorized as either hardware-or-software based. This approach certainly serves to simplify things, but throughout the course of this chapter I will go some way to infer that there are instances in which the interplay of both (hardware and software) can be the reason for the issues at hand.

So, before you begin troubleshooting a system, always consider that the need gathering information about a system is the recommended approach to gaining additional insight and familiarity. Look at it this way: the practice of gathering hardware information is not necessarily required, but an investigation of this type may assist you in the search for an eventual diagnosis.

To begin, we will start by running a simple CPU-based hardware report with the following command:

# cat /proc/cpuinfo

As you will see, the purpose of this command is to output all information related to the CPU model, family, architecture, the cache, and much more. The /proc approach is always a good tradition, but using the following command is generally considered to be a better practice and far easier to use:

# lscpu

This command will query the system and output all relevant information associated with the CPU in the following manner:

Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                2
On-line CPU(s) list:   0,1
Thread(s) per core:    1
Core(s) per socket:    2
Socket(s):             1
NUMA node(s):          1
Vendor ID:             GenuineIntel
CPU family:            6
...

On the other hand, rather than querying absolutely everything, you can specify criteria by using grep (a subject that we will return to a little later in this chapter) in order to obtain any pertinent information, like this:

# lscpu | grep op-mode

So, having done this and recorded the results for future reference, we will now continue our investigation by running a simple hardware report with the lspci command in the following way:

# lspci

The result of this command may output something similar to the following information:

00:00.0 Host bridge: Intel Corporation 82P965/G965 Memory Controller Hub (rev 02)
00:01.0 PCI bridge: Intel Corporation 82G35 Express PCI Express Root Port (rev 02)
00:05.0 Ethernet controller: Red Hat, Inc Virtio network device
00:0a.0 PCI bridge: Digital Equipment Corporation DECchip 21150
00:0e.0 RAM memory: Red Hat, Inc Virtio memory balloon
00:1d.0 USB controller: Intel Corporation 82801FB/FBM/FR/FW/FRW (ICH6 Family) USB UHCI #1 (rev 02)
00:1d.7 USB controller: Intel Corporation 82801FB/FBM/FR/FW/FRW (ICH6 Family) USB2 EHCI Controller (rev 02)
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev f2)
00:1f.0 ISA bridge: Intel Corporation 82801HB/HR (ICH8/R) LPC Interface Controller (rev 02)

The lspci command provides all the relevant information concerning the PCI devices of your server, which in turn, can be expanded by employing either the -v option or the alternative -vv / -vvv option(s), depending on the level of detail you require:

# lspci -v
# lspci -vv
# lspci -vvv

By default, the above commands will provide all the information required by you to confirm whether a device is supported by any of the modules currently installed on your system or not. It is expected that you should only need to do this when hardware upgrades have been implemented, when the system has just been installed, or if you are attempting to familiarize yourself with a new environment. However, in order to simplify this exercise even further, you will be glad to know that a "tree view mode" is also available. The purpose of this facility is to output the associated device ID and show how these values are associated with the relevant bus.

To do this, type the following command:

# lspci -t

As a troubleshooter, you will be aware that every device must maintain a unique identifier as CentOS, like all other operating systems, will use that identifier to bind a driver to that device. The lspci command works by scanning the /sys tree for all connected devices, which can also include the connection port, the device type, and class, to name but a few. Having done this, the lspci command will then consult /usr/share/hwdata/pci.ids to provide the human-readable entries it displays.

For example, you can display the kernel drivers/modules by typing the following lspci command with the -k option like this:

# lspci -k

Naturally, during any hardware-based troubleshooting investigation you will want to review the system logs for additional clues, but as we have seen, both the lscpu and lspci commands are particularly useful when attempting to discover more about the necessary hardware information present on your system.

You can learn more about these commands by reviewing the respective on-board manuals at any time:

$ man lscpu
$ man lspci

Meanwhile, if you want to practice more, a simple test would be to insert a USB thumb drive and to analyze the findings yourself by paying close attention to the enumeration found within /var/log/messages.

On the other hand, in the same way that we can use grep with lscpu, if you are already feeling comfortable with this type of investigation, then you may like to know that you can also use grep with the lspci command to discover more about your RAID controller in the following way:

# lspci | grep -i raid

Now, I am sure you will not be surprised to learn that there are many more commands associated with obtaining hardware information. This includes (but is not limited to) lsmod, dmidecode hdparm, df -h, or even lsblk and the many others that will be mentioned throughout the course of this book. All of them are useful, but for those who do not want to commit them to memory, a significant amount of information can be found by simply reading the files found within the /proc and /sys directories like this:

# find /proc | less
# find /sys | less

Consequently, and before we move on, you should now be aware that when you are dealing with hardware analysis, perfecting your skills is about practice and exposure to a server over the long term. My reason for stating this is based on the notion that a simple installation procedure can serve to identify these problems almost immediately, but without that luxury, and as time goes by, it is possible that the hardware will need replacing or servicing. RAID Battery packs will fail, memory modules will fail, and, on some occasions, it could be that a particular driver has not fully loaded during the most recent reboot. In this situation, you may find that the kernel is flooding the system with random messages to such an extent that it suggests an entirely different issue is causing the problem. So yes, hardware troubleshooting requires a good measure of patience and observation, and it is for this reason that a quick review of both the lscpu and lspci commands has formed our introduction to troubleshooting CentOS 7.

Before we dive into the subject of log files, I would like to begin by spending a few moments to discuss the importance of the dmesg command.

The dmesg command is used to record messages from the kernel that are specifically related to the process of hardware detection and configuration. I will not go in too much technical detail at this point, but it is important to realize that these messages are derived from the kernel ring buffer; a condition that can not only prove to be of great assistance because it relates back to the subject of hardware troubleshooting, but one that provides evidence as to why an understanding of the system hardware can reflect in a possible software diagnosis and vice versa.

The dmesg file is located in the /var/log/ directory, but unlike other files that reside in that directory, the basic syntax to view the contents of the dmesg file is as follows:

# dmesg | less

You can page through the results in the usual way, but if you would like to make the timestamp a little easier to read, you may want to invoke the -T option like this:

# dmesg -T | less

These commands will now provide us with information related to all the hardware drivers loaded into the kernel during the boot sequence. This information will include their status (success or failure), and if a failure is recorded, it will even provide an error message describing why a failure took place. However, as this file can be quite overwhelming, you should use grep to query dmesg in order to streamline this information and simplify the output.

To do this, simply customize the following syntax to suit your needs:

# dmesg -T | grep -i memory

This command will now display all relevant information regarding the total memory available and shared memory details associated with the server. Of course, similar approaches can be made to read the specific information for USB devices, direct memory access (DMA), or even tty.

For example, you can query dmesg to display hardware information related to any Ethernet ports in the following way:

# dmesg –T | grep -i eth0

Depending on your system configuration, the output will look similar to this:

[Sun Apr 19 04:56:57 2015] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready

To extend this approach, you can then modify the previous command in order to discover whether the kernel has detected a specific hard disk. To do this, type:

# dmesg –T | grep sda

Alternatively, you can then use the -i option to ignore the effects of case sensitivity when searching for tty references:

# dmesg | grep -i tty

As you will see, the output of the dmesg file is verbose and the information contained within it can be used to troubleshoot almost anything from network cards to storage issues. The demsg file may not give you the answer you are looking for straightaway, but it does provide you with another piece of the puzzle when it is used in combination with the information found in some of the more common log files associated with the CentOS operating system.

By default, all CentOS system log files can be found in /var/log and a full inventory on your current server can be obtained by typing the following command:

# find /var/log

With that said, every system is different, and for overall simplicity, you will find that some of the more common log files (associated with a minimal installation of CentOS 7) will include:

Now, before we continue, I want to draw your attention towards the importance of these files as it is often a good idea to store /var/log in a separate partition to / (root).

A perfect system would maintain a separate partition for /tmp, /usr, and others, but yes, there may be situations where storing your log files on the same partition as / (root) is unavoidable. So remember, if and when the opportunity does arise, you may want to consider storing these directories on a separate filesystem and a separate physical volume (if possible), as this is considered to be good practice with regard to maintaining the overall security, integrity, and performance of the system in question.

However, and having said that, it is also important to recognize that many other packages will create and store logs in other locations. You may even be required to specify these locations yourself, and for this reason, it should be remembered that not all logs are located in /var/log.

For example, if the server in question is hosting one or more websites and storing all the relevant Apache VirtualHost information in a specific /home directory, then the associated log files may be found in a location like this:

/path/to/virtualhost/domain1/log/access_log
/path/to/virtualhost/domain1/log/error_log

The same can be said of many other packages, and this issue arises because the packages may not have the required privileges to write to that directory, while others are designed to maintain all logging activity within their own installation directory. Therefore, and depending on the nature of your system, you may need to spend a few moments analyzing your server's installation structure in order to locate the appropriate log file(s).

Viewing or reading a log file is very easy and depending on your personal preferences, the basic syntax to view any of these files can be expressed in any of the following formats:

# less /var/log/filename
# more /var/log/filename
# cat /var/log/filename
# cat /var/log/filename | less
# cat /var/log/filename | more

Log files can vary between applications and services, but the general purpose of these files is to record the time and date of an event and the security level, and to provide a message or general description. Most messages will be general notices or warnings of one type or another, but on certain occasions, errors will also be trapped.

For example, you may see something like this:

Dec  4 12:49:05 localhost postfix/postfix-script[1909]: starting the Postfix mail system

Messages like this are quite ordinary and merely explain what is happening and when it happened. Yes, you can safely ignore them, but due to the number of messages you see, some may remark or feel that the system is acting a little oversensitive to the extent that a log file is being flooded with low-level information. This information may serve no real purpose to many, but in some circumstances, you may consider that the information supplied isn't sensitive enough, and more information is needed. In the end, only you can decide what best suits your needs. So, in order to take a case in point, let's increase log sensitivity for the purpose of troubleshooting the system.

To do this, we will begin by running the following command:

# cat /proc/sys/kernel/printk

The output of the preceding command enables you to view the current settings for the kernel, which, on a typical system, will look like this:

4       4       1       7

There is a relationship at work here, and it is important that you understand that printk maintains four numeric values that control a number of settings related to the logging of error messages, while every error message in turn maintains its very own log level in order to define the importance of that message.

The log level values can be summarized in the following way:

So, based on the above information, the log level values of 4, 4, 1, and 7 tell us that the following is now apparent:

Consequently, you are now in a position to consider making changes to the log level through a configuration file found at /etc/sysctl.conf. This file enables you to make fine adjustments to default settings, and it can be accessed with your favorite text editor in the following manner:

# nano /etc/sysctl.conf

To make the required change use the following syntax:

kernel.printk = X X X X

Here, the actual value of X is a log level setting taken from the options described earlier.

For example, you can change the number of messages by adding the following line:

kernel.printk = 5 4 1 7

Of course, such a modification implies a change to the kernel, and for this reason a reboot would be warranted. So, having done this, you will find that the output of running cat /proc/sys/kernel/printk should now reflect the new values. However, and as a supplementary note of caution, having considered doing this (and yes, you can easily reverse any changes made), it is important to realize that there are many questions based on the validity of changing these settings. Look at it this way: it may not help you at all, so you should always read around the subject before making these changes in order to confirm that making this alteration will suit your general purposes.

To view the onboard manual, simply use the following command:

$ man sysctl

On the other hand, for the many other services and applications on your server, you will have additional avenues of investigation to consider and these are generally set by the service or application in question.

A common example of this is Apache. So, if you are debugging a web-based issue related to this service, you may be inclined to open the httpd configuration file like this:

# nano /etc/httpd/conf/httpd.conf

Look or search for the following line:

LogLevel warn

Then, replace the instruction with a more appropriate setting (before saving the file and restarting the service). In this case, you can use:

LogLevel debug

Fortunately, it is nice to know that most services and applications do support a form of debugging mode for an improved log output. This will make the log file much more descriptive and easier to work with when troubleshooting the server, but just before we leave this subject, here comes the small print…

When you are working with log files, you should be aware that the information contained within those log files will not always be enough to help you diagnose the issue at hand or discover the cause of a problem. Log files may not only lack the required information, but they can also contain unknown errors and misleading messages. After all, log files only contain a series of (mainly) predefined messages or break points in a package, and these messages have been designed by programmers to make a remark concerning a known event that could have, or has taken place.

Based on these circumstances, there are no hard and fast rules because we also know that log files have limitations. So, in the end you will rely on a keen eye for detail and a great deal of patience, and for these reasons alone, you must always learn to "listen to the server" as a whole.

Let's put it this way: the answer is there, but it may not be in the log files. Perseverance and a calm (but firm) hand will win the day, and it is this point of contention that will be echoed throughout the pages of this book.

So, armed with the previous information and knowing that log files tend to describe events by specifying the time of occurrence, a level of severity, and a preordained message, the key to success in any troubleshooting scenario is based on an ability to work with these records and manipulate them in such a way that they provide us with the information we require to get the job done.

For the purpose of troubleshooting, one of the most useful commands you will use is known as tail. A command-line expression that can be used to read the last lines of a log file is as follows:

# tail -n 100 /var/log/maillog

Similarly, tail can also be used to obtain the most recently added lines like this:

# tail -f /var/log/maillog

Using this command not only gives you the most recent view of the log file in question, but also ensures that all updates are displayed immediately, which provides an instant way to read log files in a live environment. This approach can be described as the perfect way to troubleshoot Apache, Postfix, Nginx, MySQL, and the many other applications or services your server may be using.

For example, you can view the Apache access_log like this:

# tail -f /var/log/httpd/access_log

To take this feature one step further, let's assume that you wanted to get the last 3,000 lines from a log file knowing that it will not fit within your shell window. To account for this requirement, you can pipe the results with the less command like this:

# tail -n 3000 /var/log/messages | less

In this situation, you can now page the results as required, but having used this technique a few times, I think you would agree that this is far more flexible than using the generic cat command; unless of course, you wanted to do something very specific.

The cat command has been with us for a long time and, returning to our previous discussion relating to hardware and the contents of the /proc directory, you can use the cat command to view detailed information about your server's CPU:

# cat /proc/cpuinfo

If you wish to know more about the server's memory, you can use:

# cat /proc/meminfo

Then, there is always the chance to learn more about your devices by typing:

# cat /proc/devices

As useful as cat is, it is also known for providing a dump of the entire content on the screen, a condition that can seem a little unwieldy if the file is greater than 1,000 lines long. So, in these circumstances, the other option is to use the less and more commands in order to page through specific (static) files in the following way:

# less /var/log/messages
# more /var/log/messages

However, because more is relatively old, most will argue that less is far superior. The less command is similar to more, but less will allow you to navigate back and forth between paged results. So yes, it's an old joke, but from now on, and wherever possible, always know that less really does mean more.

For example, less allows you to search for a particular string. To do this, simply open the following file using less like this:

# less /var/log/messages

Now, in the lower left portion of the screen, type /, followed by a string value like this:

/error

The output will now be adjusted to highlight the search results, and if you are looking for a larger selection of options, simply hit the H key while less is open.

Now let's consider the need to search the server's log files for specific keywords.

In this situation, you would use the command known as grep, which also becomes a very helpful technique to learn when you would like to perform an advanced string-based search of almost any file on your server.

Let's say you wanted to search for a specific e-mail address in the mail log file. To do this, you would use grep in the following way:

# grep "user@domain.tld" /var/log/maillog

Taking this one step further, grep can also be used to search in a recursive pattern across one or more files at the same time.

For example, in order to search the log file directory for an IP address (XXX.XXX.XXX.XXX), you would use the grep command in combination with the -R option like this:

# grep -R "XXX.XXX.XXX.XXX" /var/log/

Similarly, you can add line numbers to the output with the -n option like this:

# grep -Rn "XXX.XXX.XXX.XXX" /var/log/

Moreover, you will also notice that, during a multi-file based search, the filename is made available for each search result, but by employing the -h option, this can be disabled in the following way:

# grep -Rh "XXX.XXX.XXX.XXX" /var/log/

You can ignore case with the -i option in the following way:

# grep -Ri "XXX.XXX.XXX.XXX" /var/log/

Moving beyond this, grep can be used to sort the content of a search result by simply calling the sort command. An alphabetical sort order (a to z) can be achieved by simply adding sort at the end of your original command like this:

# grep -R "XXX.XXX.XXX.XXX" /var/log/ | sort 

A reverse alphabetical sort order (z to a) can be achieved by adding the -r option like this:

# grep -R "XXX.XXX.XXX.XXX" /var/log/ | sort -

And finally, if you wish to search for more than one value, you can invoke the –E argument like this (but do not include unnecessary white spaces between the pipes):

# grep -E "term 1|term 2|term 3" /var/log/messages

Of course, grep can do so much more, but for the purpose of troubleshooting, I would now like to draw your attention to one final, but very useful command. Known as diff, this command can be very useful in determining the differences between two files.

The diff command is not necessarily considered to be a tool that is associated with log files unless you are comparing backups for a specific purpose. However, the diff command is very useful when comparing changes across an application.

For example, diff will enable you to compare the differences between two Apache configuration files, but by using the -u option, you will be able to include additional information such as the time and date:

# diff -u /etc/httpd/conf/httpd.conf /etc/httpd/conf/httpd.conf.backup

Now, depending on the size of the files in question and the speed of your server, it may take a few seconds (or even minutes) to complete the task, and yes, I do realize we were digressing from the context of log files, but in time, I think that you will find this command will prove to be very useful.

For example, you may want to compare the contents of two folders using the –rq option to make it recursive like this:

# diff –rq /path/to/folder1 /path/to/folder2

To learn more about the diff command, simply review the manual by typing:

$ man diff

So, having been shown how easy it is to work with log files, we should always be mindful that records like this do grow in size, and for this precise reason, they can become difficult to work with as time passes. In fact, you should be aware, oversized log files can impact the system's performance. With this in mind, it is a good idea to monitor any log rotation process and adjust it (on a regular basis) according to need.

Moreover, where log rotation can be critical for a medium- to high-load environment, I would suggest that you manage this solution effectively. However, in situations where the effect of this process proves negligible, the following fail-safe technique will enable you to scrub a log file clean by typing either one of the following commands:

# cat /dev/null > /path/to/file

Or more appropriately, you can simply use the truncate command like this:

# truncate --size 0 /path/to/file

This process is known as truncation, and as mentioned, this should remain something of a last resort, as the preceding command will remove all the data contained within the file in question. So remember, if the file contains important information that you may need to review at some time in the future, back it up before you use truncate.

This chapter was intended to provide an introduction to the subject of troubleshooting CentOS 7 without the intention of burdening you with yet another list of rules, instructions, or procedures that would ill-suit your circumstances or immediate needs. As we know, troubleshooting is a journey, and where the first chapter has served to introduce you to a selection of concepts and methods, every page that follows will ensure that you are one step closer to being at ease with the server you are about to diagnose and repair. So yes, the journey has just begun, and we will now approach the subject of troubleshooting active processes.