We have covered a lot of errors and problems that you need to troubleshoot in a typical OpenStack installation. In this final chapter, we want to cover some of the chronic issues that might be early signs of trouble. This chapter is more about prevention and aims to help you avoid emergency troubleshooting as much as possible.
We will be looking at the following topics in this chapter:
Databases
RabbitMQ
Services
Community resources
As we have seen throughout this book, many OpenStack services make heavy use of databases. Production deployments typically use MySQL or Postgres as a backend database server. As you have learned, a failing or misconfigured database will quickly lead to trouble in your OpenStack cluster. Database problems can also present more subtle concerns that may grow into huge problems if neglected.
The database server can become a single point of failure if your database server is not deployed in a highly available configuration. OpenStack does not require a high-availability installation of your database, and as a result, many installations may skip this step. However, production deployments of OpenStack should take care to ensure that their database can survive the failure of a single database server.
RabbitMQ is the default message broker used in OpenStack installations. However, if it is installed as is out the box, it can become a single point of failure. Administrators should consider clustering RabbitMQ and activating mirrored queues.
You can confirm whether or not your RabbitMQ installation is operating as a cluster by running the cluster status command:
rabbitmqctl cluster_status Cluster status of node 'rabbit@tc-ost1' ... [{nodes,[{disc,['rabbit@tc-ost1']}]}, {running_nodes,['rabbit@tc-ost1']}, {partitions,[]}] ...done.
In the preceding output, note that in the running_nodes section, there is only one node listed. This is an indication that we are not running clustered RabbitMQ. For details on how to cluster RabbitMQ, refer to the latest Rabbit documentation at https://www.rabbitmq.com/clustering.html.
As we saw throughout this book, OpenStack deployments run multiple services. Each service typically runs multiple Linux processes. If you find yourself troubleshooting OpenStack, you should always make sure that the expected services are up and running. This is the equivalent of making sure that everything is plugged in. Monitoring the Linux processes for each service may save you a ton of headaches when it comes to troubleshooting.
You should consider monitoring the key processes for each service. This could be as simple as configuring your monitoring service to run ps --aux on the service at regular intervals to ensure that the process is alive. Another option is to use a monitoring library that supports process monitoring and the ability to automatically restart services if they crash. Deploying methods that allow monitoring and automatic recovery of the OpenStack processes will save you time and headaches as you troubleshoot.
The OpenStack community is large, smart, and welcoming. If you find yourself stuck, the community will be willing and able to lend a hand. Don't hesitate to reach out for help if you find yourself hitting a dead end. The community provides a variety of tools and resources that can help you with troubleshooting.
The OpenStack software comes with an extensive suite of tests that are used during the development process to ensure a high-level of quality. This test suite can also be useful to the troubleshooter as a means to keep an eye on the overall reliability of your installation. This test suite can help you ensure that the software is running as intended. You can run the suite periodically against the various projects you are using and check for any errors. Not every error will need to be addressed, but this information could lead to early clues pointing to potential trouble in your stack.
OpenStack is leading open source software that is used to run private clouds. Its popularity has grown exponentially since it was founded by Rackspace and NASA. The output of this engaged community is staggering, resulting in plenty of new features finding their way into OpenStack with each release. The project is at a size now where no one can truly know the details of each service. When working with such a complex project, it is inevitable that you are going to run into problems, bugs, errors, issues, and plain old trouble. Our goal with this book was to give you the tools necessary to tackle those problems and find your path to a solution. We hope this book helps you get unstuck and start stacking.
© 2016 Packt Publishing Limited All Rights Reserved
