Chapter 5: Patch Management Strategies

In day-to-day operations of your Linux systems probably the most common task for an average system administrator is going to be patch management (aka patching.) Unlike the Windows and macOS worlds, it is standard for the system administrator to handle a broad variety of operating system and application patching tasks covering both primary and third-party ecosystems. It is also standard for there to be built in, and sometimes third party, application management ecosystems to assist with this potentially daunting task.

Patching, and of course system updates, are large parts of what we do and while it may feel mundane it is very important that it be something that we get right. And production Linux systems today have become much more complex and diverse than they were just ten years ago. And of course, patching has become more important than ever, something that we expect to only see increase over time as well.

We will start by understanding...

Binary, source, and script software deployments

Software can come in all shapes and sizes. So, software deployments are not a one size fits all affair. The standard means of deploying software are directly as a binary package, through source code that needs to be compiled into a binary package, or as a script. We will dig into each of these as it is necessary to understand what each is and when they might be appropriate.

Compiled and interpreted software

Many system administrators have never worked as developers and often are not aware of how software exists. There are two fundamental types of programming languages: compiled and interpreted.

Compiled languages are written in one language (source code) and run through a compiler to produce binary executable code that can run directly on an operating system. This can be an oversimplification, but we are not developers and need only be concerned with the original code being compiled into a binary format. For Linux, this format...

Patching theory and strategies

One might think that patching is pretty straightforward, and that there would be little to discuss. This is not the case. In fact, if you talk to several system administrators you are bound to get some pretty widely varying opinions. Some people patch daily, some weekly, some wait as long as they can, some do so only haphazardly, and some believe that you should never patch at all (hey, it if isn't broke, don't fix it!)

We should first establish why we patch our software. Patching, as opposed to updating or upgrading, implies that we are applying minor fixes to software to fix a known problem or bug but not to implement new features or functionality. Adding new features is generally considered to be an update.

Most software vendors and operating system vendors honor this system and maintain patching systems that only address security or stability issues in their software between releases. In the Linux ecosystem this is primarily tied...

Compilations for the administrator

It was not all that long ago when major system administrator guidelines included a requirement that any mid-level or senior administrator had to be well acquainted with the details of standard software compilation processes. Of course, all knowledge is good, and we would never say that it should not be learned at all. However, even at the time, this seemed like an odd amount of under-the-hood development knowledge and knowledge about the packaging of individual software solutions expected to be known by a person in a non-development role.

It would not be unlike if when ordering a new car from your favorite car company that it was expected to be delivered as parts and that every potential new car owner would be expected to assemble the car before driving it. It is important to note that this process was only ever possible in the open-source world and that the majority of software outside of the Linux space and even a significant portion within...

Linux deployment and redeployment

Today we have many potential methods for deploying, and just as importantly, redeploying, our servers (or our workstations, for that matter.) In my opinion, this topic was relatively unimportant in the past because most companies depended on slow deployment methods and their disaster recovery models depended on restoring, rather than redeploying, their systems. But there are so many more modern disaster recovery methods today that depend on the ability to rapidly deploy servers that we have to look at this topic with a new eye.

With modern deployment technologies and techniques, it is not uncommon to be able to deploy a new base operating system in a matter of seconds when in the past, even heavily automated systems would often take many minutes if not hours (not even considering the possibilities that would come with custom compiled systems!). Of course, computers are just faster today, and this plays a role in speeding deployments. Vendors have...

Rebooting servers

Ask your average system administrator, or even a non-technical but interested third party, and they will tell you the importance of long uptimes on servers and how they want to see those ultra-high time since reboots on them. It feels natural, and nearly everyone brags about it. My servers have not needed a reboot in three years!

There are two key problems with this, however.

The first problem is that time since reboot carries no business value, and business value determines IT value. So why should we care, let alone brag, about something that has no value? It might be interesting to know how long a system has managed to stay online, but an investor is not going to reap a reward from the fact that a computer system has gone an extended period of time without a reboot. We work for the good of the business, if we start to care about something other than resultant business value, we have lost our way. This happens when we focus on means instead of ends, server...

Summary

System patching, updates, and reboots may feel very pedantic. And in some ways, I suppose that they are. But sometimes really important things can also be kind of boring. And really, patching and basic system maintenance should be boring. It should be predictable, reliable, and scheduled. And if at all possible, it should be automated.

Patching should not become a challenge or a scary proposition. With proper planning, backups, testing and so forth, it is generally easy to have a reliable patching and even update processes that very rarely experience major issues of any kind. If we fail to make our patching and updates regular and reliable, we will begin to fear the process which will almost certainly lead us to avoid it more which will just exacerbate the problem.

In the modern world of computing, there is always someone looking to exploit our systems and while nothing can protect against every possible attack, we can heavily mitigate our exposure through rapid, regular...