Chapter 16: Troubleshooting Guidelines for the KVM Platform

If you've followed this book all the way from Chapter 1, Understanding Linux Virtualization, then you'll know we went through a lot together in this book – hundreds and hundreds of pages of concepts and practical aspects, including configuration examples, files and commands – everything. 700 or so pages of it. So far, we've almost completely ignored troubleshooting as part of that journey. We didn't do this on the premise that everything just works in Linux and that we didn't have any issues at all and that we achieved a state of nirvana while going through this book cover to cover.

It was a journey riddled with various types of issues. Some of them aren't worth mentioning as they were our own mistakes. Mistakes like the ones we made (and you will surely make more too) mostly come from the fact that we mistyped something (in a command or configuration file). Basically, humans...

Verifying the KVM service status

We're starting off with the simplest of all examples – verifying the KVM service status and some of its normal influence on host configuration.

In Chapter 3, Installing a KVM Hypervisor, libvirt, and ovirt, we did a basic installation of the overall KVM stack by installing virt module and using the dnf command to deploy various packages. There are a couple of reasons why this might not end up being a good idea:

• A lot of servers, desktops, workstations, and laptops come pre-configured with virtualization turned off in BIOS. If you're using an Intel-based CPU, make sure that you find all the VT-based options and enable them (VT, VT-d, VT I/O). If you're using an AMD-based CPU, make sure that you turn on AMD-V. There's a simple test that you can do to check if virtualization is enabled. If you boot any Linux live distribution, go to the shell and type in the following command:

  cat /proc/cpuinfo | egrep "vmx|svm...

KVM services logging

When discussing KVM services logging, there are a couple of locations that we need to be aware of:

• Let's say that you're logged in as root in the GUI and that you started virt-manager. This means that you have a virt-manager.log file located in the /root/.cache/virt-manager directory. It's really verbose, so be patient when reading through it.
• The /etc/libvirt/libvirtd.conf file is libvirtd's configuration file and contains a lot of interesting options, but some of the most important options are actually located almost at the end of the file and are related to auditing. You can select the commented-out options (audit_level and audit_logging) to suit your needs.
• The /var/log/libvirt/qemu directory contains logs and rotated logs for all of the virtual machines that were ever created on our KVM host.

Also, be sure to check out a command called auvirt. It's really handy as it tells you basic information about the virtual...

Enabling debug mode logging

There's another approach to logging in KVM: configuring debug logging. In the libvirtd configuration file that we just mentioned, there are additional settings you can use to configure this very option. So, if we scroll down to the Logging controls part, these are the settings that we can work with:

• log_level
• log_filters
• log_outputs

Let's explain them step by step. The first option – log_level – describes log verbosity. This option has been deprecated since libvirt version 4.4.0. In the Logging controls section of the file, there's additional documentation hardcoded into the file to make things easier. For this specific option, this is what the documentation says:

Figure 16.1 – Logging controls in libvirtd.conf

What people usually do is see the first part of this output (Logging level description), go to the last line (Iog_level), set it to 1, save, restart the libvirtd service...

Advanced troubleshooting tools

Depending on the subject – networking, hardware and software problems, or specific application problems – there are different tools that we can use to troubleshoot problems in our environments. Let's briefly go over some of these methods, with the chapters of this book in mind as we troubleshoot:

• oVirt problems
• Problems with snapshots and templates
• Virtual machine customization issues
• Ansible issues
• OpenStack problems
• Eucalyptus and AWS combo problems
• ELK stack issues

Interestingly enough, one thing that we usually don't have problems with when dealing with KVM virtualization is networking. It's really well documented – from KVM bridges all the way to open vSwitch – and it's just the matter of following the documentation. The only exception is related to firewall rules, which can be a handful, especially when dealing with oVirt and remote database connections while...

Best practices for troubleshooting KVM issues

There are some common-sense best practices when approaching troubleshooting KVM issues. Let's list some of them:

• Keep it simple, in configuration: What good does a situation in which you deployed 50 OpenStack hosts across three subnets in one site do? Just because you can subnet to an inch of an IP range's life doesn't mean you should. Just because you have eight available connections on your server doesn't mean that you should LACP all of them to access iSCSI storage. Think about end-to-end configuration (for example, Jumbo Frames configuration for iSCSI networks). Simple configuration almost always means simpler troubleshooting.
• Keep it simple, in troubleshooting: Don't go chasing the super-complex scenarios first. Start simple. Start with log files. Check what's written there. With time, use your gut feeling as it will develop and you'll be able to trust it.
• Use monitoring tools such...

Summary

In this chapter, we tried to describe some basic troubleshooting steps that can be applied generally and when troubleshooting KVM. We also discussed some of the problems that we had to deal with while working with various subjects of this book – Eucalyptus, OpenStack, the ELK stack, cloudbase-init, storage, and more. Most of these issues were caused by misconfiguration, but there were quite a few where documentation was severely lacking. Whatever happens, don't give up. Troubleshoot, make it work, and celebrate when you do.

Questions

1. What do we need to check before deploying the KVM stack?
2. What do we need to configure after deploying the KVM stack in terms of making sure that virtual machines are going to run after reboot?
3. How do we check KVM guest log files?
4. How can we turn on and configure KVM debug logging permanently?
5. How can we turn on and configure KVM debug logging at runtime?
6. What's the best way to solve oVirt's installation problems?
7. What's the best way to solve oVirt's minor and major version upgrade problems?
8. What's the best way to manage the oVirt Engine and host updates?
9. Why do we need to be careful with snapshots?
10. What are the common problems with templates and cloudbase-init?
11. What should be our first step when installing Eucalyptus?
12. What kind of advanced capabilities for monitoring and reporting can we use with the ELK stack?
13. What are some of the best practices when troubleshooting KVM-based environments?

...

Further reading

Please refer to the following links for more information regarding what was covered in this chapter:

• Working with KVM debug logging: https://wiki.libvirt.org/page/DebugLogs
• Firewall requirements for oVirt nodes and oVirt Engine: https://www.ovirt.org/documentation/installing_ovirt_as_a_standalone_manager_with_remote_databases/#dns-requirements_SM_remoteDB_deploy
• oVirt upgrade guide: https://www.ovirt.org/documentation/upgrade_guide/
• Common problems with NetApp and Openstack integration: https://netapp-openstack-dev.github.io/openstack-docs/stein/appendices/section_common-problems.html
• Integrating the IBM Storwize family and SVC driver in OpenStack: https://docs.openstack.org/cinder/queens/configuration/block-storage/drivers/ibm-storwize-svc-driver.html
• Integrating IBM Storwize and OpenStack: https://www.ibm.com/support/knowledgecenter/STHGUJ_8.2.1/com.ibm.storwize.v5100.821.doc/storwize_openstack_matrix.html
• HPE Reference Architecture...