Chapter 6. Troubleshooting OpenStack Block Storage

Cinder is the OpenStack service that provides persistent block storage for your Nova instances. Cinder block storage is like an external hard drive or USB drive for your instance, in that it can only be attached to one instance at a time. Cinder boasts a pluggable architecture, allowing OpenStack operators to leverage several backend storage plugins, for different types of storage. The supported plugins include Ceph, Sheepdog, NFS, Gluster, LVM, and others. While Cinder supports many storage drivers, we will focus on Linux Volume Manager (LVM). In this chapter, we will cover the following topics:

• Troubleshooting Cinder processes

• Exploring Cinder logging

• Cinder authentication issues

• Cinder and RabbitMQ

• Common Cinder errors

Cinder processes

A working Cinder installation will have several processes running. Say, for example, you run the following command:

ps –aux | grep cinder

After this, you should see an output similar to the output shown here:

You will note that there are several cinder-api processes, several cinder-volume processes, and a cinder-scheduler process. When troubleshooting Cinder, you want to make sure that these three processes are up and successfully running. If one of the processes isn't running for some reason, you can use the respective start commands given here on Ubuntu systems using upstart:

start cinder-api
start cinder-volume
start cinder-scheduler

If you have trouble starting any of the processes using the upstart scripts, you may want to try and start them manually, where you will be able to see any errors that are being thrown on startup. To start the process manually, use the following command:

sudo -u cinder cinder-api --config-file=/etc/cinder/cinder.conf --log-file=/var/log/cinder...

Cinder dependencies

Like most other OpenStack services, Cinder has dependencies on other OpenStack components. These components must be present and operate successfully in order for Cinder to do its job. In a troubleshooting scenario, you want to check these dependencies and confirm that they are up and running.

Keystone authentication problems

Cinder leverages Keystone to provide authentication and authorization services. If the Keystone service is not running, it shouldn't take long for you to see the result. If you run a cinder command, such as cinder list or cinder create without running Keystone, you will see an error like the one shown here:

In this case, you want to check and make sure that the Keystone service is up and running. Refer to Chapter 2, Troubleshooting OpenStack Identity, for details on how to make sure that Keystone is running successfully. Imagine a scenario where you have confirmed that Keystone is up and running and yet when you run a cinder command you receive an error...

Cinder errors

In this section, we will explore a few of the more common errors you might encounter with Cinder. While we highlight just a few of the errors you may come across, the troubleshooting techniques we illustrate should be helpful in various troubleshooting situations.

Missing the cinder-volumes volume group

When using LVM to back Cinder, OpenStack expects there to be a volume group named cinder-volumes. When running the vgdisplay command, you can check for the cinder-volumes volume group as shown here:

If you attempt to create cinder volume without having a volume group named cinder-volumes, it's very likely that your volume will end up in an error state as illustrated in the following screenshot with MyVol2:

You can further confirm that this is the issue by checking the scheduler log at /var/log/cinder/scheduler.log file. In this log file, you are looking for an error similar to the one shown here:

The last line in the log file may point to the fact that No valid host was found, but...

Summary

Cinder provides a persistent block storage service for OpenStack. The service is not overly complex, but there are a few things that can cause it to break. There will likely come a time when you need to troubleshoot Cinder. In this chapter, you learned how to troubleshoot the Cinder processes, check the log files, and confirm the authentication settings. We also looked at how to troubleshoot Cinder and RabbitMQ, and finally, we looked at some common Cinder errors. With the tools and tips presented in this chapter, you should be well on your way to solving the common Cinder issues that come your way. In the following chapter, we will work with Swift, OpenStack's object storage.