Few things in our industry sound like they should be simpler to answer than this one, simple question: what is a system administrator? And yet, ask anyone and you'll get some widely differing opinions. Everyone seems to have their own take on what the title or role of System Administrator implies, including and possibly most varying in people who use this title for themselves or from the companies that hand it out!

Welcome to system administration and specifically Best Practices of Linux Administration. In this chapter we are going to dive into understanding the job, role, and functions of a real system administrator and try to understand how we, in that role, fit into an organization.

In tackling this book, it is necessary both for myself to have some semblance of a clear course in writing, but also for you to understand if this book is for you, or to grasp the scope that I am attempting to cover, for me to clearly define what a system administrator is to me.

Understanding exactly what is expected of a true system administrator will be the foundation for applying that definition of the role to the upcoming best practices that apply both to system administration generally and specifically to Linux administration.

In this chapter we are going to cover the following main topics:

• Where are system administrators in the real world
• Wearing the administrator and engineering hats
• Understanding systems in the business ecosystem
• Learning system administration
• Introducing the IT professional

I think that one of the most challenging things about attempting to understand what a system administrator is comes from the fact that the title of system administrator is often given out, willy nilly, by companies with little to no understanding of Information Technology (IT), systems, or administration and treat it like a general filler for IT roles that they do not understand or know how to name. It also has a strong tendency to be given out in lieu of pay raises or promotions to entice junior staff to remain in an otherwise unrewarding job in the hopes that an impressive title will help them later in their career, so much so that in the end, the number of people working as system administrators is a very small number of people compared to the number of people with the title. In fact, it is no small stretch to guess that the average person with the title of system administrator has never thought about the meaning of the title and may have little inkling of what someone in that role would be expected to do.

If we look solely by title, system administrators are everywhere. But they exist mostly at companies too small to have plausibly employed a system administrator at all. Systems administration as a dedicated job is nearly exclusive to large companies. Most companies need someone to do the tasks of system administration, but only as a part of, and often only a small part of, their overall duties. It is the nature of IT that in small and medium sized companies you typically have generalists who wear many hats and do every needed IT role while having little to no time to focus on any one specific function. Whereas in large enterprises you generally get focused roles, often grouped into focused departments, that do just a single IT role: such as system administrator. But even in some enterprises you find departments organized like separate, small businesses and still having generalists doing many different tasks rather than separating out duties to lots of different people.

There is nothing wrong with this, of course. It is totally expected. It's much like how, as a homeowner, you will often do a lot of work on the house yourself, or you might hire a handyman who can do pretty much whatever is needed. You might need some plumbing, painting, carpentry, wiring, or whatever done. Whether you do it yourself, or your handyman does, you do not refer to either of yourselves as plumbers, painters, carpenters, and others. You are just a handy person, or the person that you hired is. You still recognize that a dedicated, full time, focused plumber, painter, carpenter, or electrician is a specialized role. You might do all those tasks occasionally, you might even be good at it, but it's not the same as if that was your full-time career. If you decided to claim to be these things to your friends, they would quickly call you out on the fact that you are quite obviously not those things.

System administrators are like plumbers. Everyone who owns a house does at least a little plumbing. A handyman who does home maintenance full time might do a fair amount of plumbing. But neither is a plumber. A very large housing development, or a construction crew might have a dedicated plumber on staff. Maybe even more than one. And nearly every homeowner must engage one from time to time. If you are me, regularly. Most plumbers either work for large companies that have need of continuous plumbing services or they work for plumbing contracting firms and have the benefits of peers and mentors to help them advance in their knowledge.

Nearly every business no matter what size we are talking about needs system administration tasks done. For very small businesses it is not uncommon for these tasks to amount to no more than a few hours per year, and when needed the scheduling is often unpredictable with many hours needed all at once and large gaps of time in which nothing is needed. In large businesses, you might need tens of thousands of hours of system administration tasks per week and require entire departments of dedicated specialists. So just like plumbers, you find small businesses either hiring IT generalists (akin to the homeowner's handyman) or outsourcing system administration tasks to an outside firm like an Managed Service Provider (commonly referred to as an MSP) or keeping a consultant on retainer; and you will find large companies typically hiring full time specialist system administrators that do nothing else and work only for that firm.

System administration tasks exist in every business, in every industry and create the foundation of what I feel is one of the most rewarding roles within the IT field. With system administration skills you can chart your own course to work in a large firm, be a consultant, join a service provider, or enhance other skills to make yourself a better and more advanced specialist. Without a firm foundation in system administration a generalist will lack one of the most core skills and have little ability to advance even in the generalist ranks. And at the top of the generalist field, true CIO roles primarily pull from those with extensive system administration comprehension.

At this point we know what a system administrator is, where you will find them in businesses, and why you might want to pursue system administration either as your career focus or as an enhancement to a career as a generalist. Now we can go into real detail about what a system administrator really does!

In this section, we will explore two parts:

• How does administration and engineering differ.
• How to identify the role you are performing.

The name system administrator itself should clue us in as to what the role should entail. The title is not meant to be confusing or obfuscated. Yet many people believe that it is some kind of trick. If you spend enough time working in the small business arena you might even find that many people, people who are full time IT professionals, may not even believe that true system administrators exist!

System is a reference to operating system and designates the scope of the role: managing the platform on which applications run. This differentiates the system administrator role from, say, a database administrator (commonly called a DBA) who manages a database itself (which runs on top of an operating system), or an application administrator (who manages specific applications on top of an operating system), or a platform administrator (who manages the hypervisor on which the operating system runs), or a network administrator (who manages the network itself.) Being called a system administrator should imply that the focus primarily or nearly entirely of the person or role is centered around the care and feeding of operating systems. If your day isn't all about an operating system, you aren't really a system administrator. Maybe system administration is part of your duties but being a system administrator is not the right title for you.

Administrator tells us that this role is one that manages something. The direct alternative to an administrator is an Engineer. An engineer plans and designs something; an administrator runs and maintains something. I often refer to these roles as the A&E roles and often the titles are used loosely and meaninglessly based on how the speaker thinks that it will sound. But, when used accurately, they have very definite meanings and in each area within IT (systems, platforms, network, databases, applications, and others.) you have both working in concert with one another. Of course, it is exceptionally common to have one human acting as both an engineer and an administrator within an organization, the roles have extensive overlap in skills and knowledge and necessarily must work in great cooperation to be able to do what they do effectively.

The difference between the role of an administrator and the role of an engineer

There is a key difference between the two roles, however, that impacts organizations and practitioners in a very meaningful way, which is very important to discuss because otherwise, we are tempted to feel that separating the two roles is nothing more than pedantic or a game of semantics. This difference is in how we measure performance or success.

An engineering role is measured by throughput or the total quantity of work done. It's all about productivity – how many systems the engineering team can design or build in a given period of time.

The administrator role would make no sense in this same context. Administrators manage systems that are already running rather than implementing new ones. An administrator is measured on availability, rather than productivity. This may sound odd to hear, but this is mostly because the average organization has little to no understanding of administration and has never contemplated how to measure the effectiveness of an administration department.

Hats

I spend a lot of time talking about hats. Understanding hats is important.

By hats, I mean in the sense of the different job roles that we take on and understanding when we are performing the tasks of one role or another – referred to as wearing a hat. For example, if I work in a restaurant I might act as a waiter one day and we would say that I was wearing a waiter hat. The next day I might be working in the kitchen making salads and then I would be wearing the short order cook hat.

This may sound a bit silly, but it is important to understand. The term hats tends to allow us to understand the difference between being a thing and performing the actions associated with that thing better. We all know what is required to be a plumber, but most plumbers drive trucks to their job sites. Does this make them truck drivers? Is the skill of driving a truck actually part of the skill of plumbing? No, it is not. But it is a generally useful ancillary skill. But a large plumbing company could easily consider hiring professional drivers that drive their plumbers from job to job to keep everyone focused on what they are trained to do, lower insurance costs, increase plumbing billable hours, and even allow for hiring excellent plumbers who maybe do not even have a driver's license!

In IT, it is so common to be asked to perform the duties of myriad different roles and functions that we often forget that we are doing it. IT professionals are often seen as a jack of all trades, able to do a little of everything and often this ends up being true just from the nature of what makes people get into IT in the first place. But it really helps when we separate our intended roles, our specialties, our training, and what earns us our salaries from when we are just helping out by performing other duties outside of our strict arena because we happen to be skilled, flexible, or just otherwise helpful.

As we progress through our exploration of Linux System Administration, the idea of hats and really digging into job roles and functions will become more and more clear. Thinking of our functions as wearing a specific hat at a specific time is a powerful tool for understanding, and possibly more importantly, for communicating our roles, requirements, capabilities, expectations, and responsibilities to others and ultimately, to ourselves. In the real world, it is very rare to find a company or role where the engineering and administration aspects of systems can be truly separated. This has benefits, and it has consequences. But it is worth mentioning that most many larger companies do, in fact, keep these two roles apart – sometimes even to the extent of having them in different departments with unique management teams. There are a lot of benefits to these being separate, primarily because the soft skills needed for the two aspects of systems are generally opposing.

Engineers benefit from the strong soft skill of being good planners. Engineering is all about planning, almost exclusively. You get to take your time, think about how a system will be used, and design around that future need. Your job is to prevent emergencies rather than reacting to them.

Administrators benefit from the strong soft skill of being good perceivers – that is, responding to events in real time, rather than planning ahead. Administrators are tasked with managing live, running systems and that means that their primary challenges are presented to them in real time, and being able to triage, prioritize, and work under pressure is paramount.

Those who make good engineers rarely make good administrators and vice versa. While the technical skills are almost a total overlap, the human skills are highly opposing and very few people manage to be truly adept at both planning and triage. Those highly skilled at administration will also naturally deprioritize engineering work because they know that they can react to it effectively later, even if poorly planned for.

There is other language that can often guide us here as well. Engineers tend to work in a world of projects. There is a goal, an end point when their work is handed off to the administrators. Administration works in a world of steady, ongoing support. There is rarely a starting or ending point and the term project would make no sense to them. They run the systems that the company needs until those systems are replaced, generally with a new workload that the administrator runs in the same way. An administrator might need to give input to a project manager as a project might need to report what its long-term administration impact is going to be, or to get the blessings of administration that they are willing to take on the results of the project when it is handed over to them. But the project itself would always be done, by definition, by an engineering role.

Knowing when we are wearing an engineer's hat or when we are wearing an administrator's hat gives us the power to understand how we can function effectively as well as how to communicate our needs and capabilities to the rest of the organization. Most organizations are blind to these kinds of psychological and performance ramifications of the job roles and require that we provide this understanding. The better that we are at identifying our role and our needs means the better prepared we are to attempt to articulate those to management.

A common problem arising from requiring IT staff to function as both an engineer and an administrator at the same time is burnout. Being an administrator naturally leads to being very busy with requests and always being the first called when things go wrong: systems performing slowly, a computer crashing, a new bug discovered, and patches needing to be applied. Administrators tend to not only work long hours but also be on-call extensively. Getting downtime when they can get it is critical to their ability to remain in the role long term.

Engineers have no need to be on-call and are not the responsible role during an emergency or problem. Engineers, however, would not be expected to need special downtime to compensate for the extreme demands on their time and schedule. They do not get interrupted, they do not have to carry a beeper (although no one literally carries a beeper anymore), they do not miss their kids' school play or have to answer questions seconds before boarding a plane, or have to run out of a family dinner to walk someone through fixing a problem remotely. They get to live scheduled lives where they can rest like normal jobs do. The image of the IT worker running ragged and never getting a break, always being expected to drop everything for the company, and living off of promises that often never materialize of a chance to rest sometime in the future is one of system administrators in nearly all cases. No other role carries so much responsibility at the times when it is most critical.

Combining these roles together means that not only would a person wearing both hats risk being on-call and having to drop everything to respond to issues all day, every day, but then needing to fill every potential remaining moment with project work for their engineering role that expects them to have available capacity to do. Trying to stay productive on projects with deadlines while also trying to react to every question, ticket, or outage is a recipe for burnout or worse.

If we can convey to management the difference between the administration function and the engineering function, we can begin a dialogue on how to make our jobs tenable, which ultimately makes things better for both parties. Being pushed too hard leads to a loss of productivity, mistakes, oversights, and staff turnover.

The fifteen-minute rule

In the software development world, which is all engineering with no administration, the standard rule of thumb is that any interruption that requires an engineer's focus will cost that engineer the time of the interruption plus fifteen minutes additional time to get back to the point of productivity where they were before the interruption. It does not take much to see that a handful of interruptions throughout a day would reduce the effective time of an engineer to essentially zero. They might be attempting to work all day, sitting at their desk trying to focus on the task, burning through their available time, but if they keep getting interruption, they will just spin their wheels and feel more and more exhausted and disheartened as they do not get a chance to rest, but neither do they get to be productive and show something for their efforts.

This fifteen-minute rule is known as task switching overhead.

The administrator's role is one of nearly all interruptions. Monitoring systems to see what might be going wrong, being alert for new patches requiring their attention, responding to tickets or questions from other teams, and so forth. Administrators are, of course, impacted by the fifteen-minute rule just the same as an engineer is. But unlike the engineer, an administrator is normally resolving a problem, or a request and the worst is a small, atomic chunk that is completed before the next interruption takes focus. When a task is completed, or an outage resolved the next task to be tackled is a different one whether it is already lined up and ready to go or doesn't arrive for some time. The administrator must go through task switching between each issue regardless, so the overhead that this incurs is a given that cannot be avoided.

Not only does task switching overhead help us to explain and understand why administrators and engineers need to either be different people or be given extreme accommodation, but it also helps to explain the much broader need for quiet, effective work environments for all staff. Interruptions don't just come from system emergencies but from anywhere like meetings, water cooler banter, office workers who just stop by, general office noise, fire drills, you name it.

Tools to measure skills for an administrator or engineer

If we are in a larger organization where these roles can be kept separate, we can show how measuring each by their unique benefit can make each department better. If we are in a smaller organization and are forced to transition from wearing one hat to another, we need to be able to effectively work with the organization to demonstrate our needs and work with management to produce working schedules or extra resources can make us more effective.

One of the best tools that I've seen used to understand the soft skills for administrator versus engineer is the Myers-Briggs Type Indicator of Judging and Perceiving. Myers-Briggs is an industry standard psychological exam that, when handled well, I feel can be highly beneficial for helping us to understand our own natural strengths and weaknesses.

The following shows an overview of the different personality types and their respective cognitive functions in the Myers Briggs model:

Figure 1.1 – Myers Briggs Cognitive functions of the personality types

Engineers typically require a strong leaning towards the judging profile, while administrators need a strong leaning towards the perceiving profile. Judging effectively equates to planning, in this case, and perceiving equates roughly to the ability to react or perform triage. Almost no one is good at both planning and responding. Knowing your strengths lets you focus on what will make you happy and what will let you excel. While knowing your weaknesses allows you (and your organization) to adapt accordingly.

The wonderous variety of the role

One of the more challenging aspects of a book like this is that the role of system administrator, even when we agree on the definition of what a system administrator is, varies so wildly that when performing the role at one company to performing the same role at another company our position can look like a completely different career path! It is easy to find system administrators who spend nearly their entire day manually deploying software and never, ever do performance tuning; another system administrator might do little other than performance tuning; another might focus on managing system applications or databases; one will manage lots of printers, but most admins will never manage even a single printer; and another managing users and userland storage concerns, while again, most, will never manage users at all. All of these system administrators may never meet another system administrator who does a similar workload to what they do, and few will ever get a second job within their field that has them do the same tasks again. Each system administration position is, effectively, unique. Almost absurdly so.

Even more extreme is that so many people think of system administration purely in terms of servers, and certainly the title tends to only be applied there, but there are careers in desktop, appliance, and even IoT realms as well. Linux, in all its forms, may remain a rather small player in the desktop and laptop space, but companies of all sizes do deploy it in production and need system administrators who understand the aspects of the operating system as they pertain to end user devices. Do not casually dismiss the desktop administration subset of systems as being all that different. While desktop administration is often less stressful or critical than server administration it is often more varied and complex and often presents some unique and extreme technical challenges.

In this book I am going to do my best to look at system administration from a high level and provide insight and guidance that applies to essentially everyone - whether you are a full-time working system administrator, a student hoping to become one someday, or an IT generalist for whom system administration is part of your regular tasks. I feel that too many guides to system administration take a myopic approach and try to look at the field as being only one or two of the myriads of aspects out there and totally forget that most of the field will never encounter the need for most of what they are taught.

Here's an image to entertain you with the jack of all trades and the master of one:

Figure 1.2 – Generalist Vs Specialist

There are three essential types of books on system administration. They are as follows:

• The first type takes a high level and attempts to present the field of administration.
• The second type digs in to detailed and highly specific tasks.
• And the third focuses on teaching skills required by a certification exam.

All three of these styles of books or resources are very valuable. In this book, we are tackling the first and we will leave detailed commands, syntax, tools, and other in the weeds implementation details up to other excellent books that already focus on those things today. I will also attempt to keep my advice as distribution agnostic as is reasonably possible. Linux is all but unique in that it is an enormous and varied ecosystem rather than a single product. When talking about best practices we are given a natural pass on needing to dig in too deeply to a specific operating system built off Linux, but it is still tempting to lean overly heavily on a certain one or two popular implementations.

It is also important to make a distinction between what is intrinsic to Linux, to an operating system built from Linux (more on that later), or intrinsic to system administration versus what is simply convention or assumption. In approaching a book like this, we can go in so many potential directions and I will, whenever possible, attempt to clarify when something is simply convention versus when something is truly how it must be done.

Great, so now we know what aptitude and psychological skills we will need to succeed at system administration, and we know about how awesome this role can be. We are getting a clear picture of what the purpose of the system administrator is. Next, we need to see how that role is going to fit into the IT department and the business itself.

Systems, that is the operating system layer of IT infrastructure, plays one of the most critical roles within IT and the business. In most businesses, most of the most critical aspects of IT functionality tend to fall to the systems roles to oversee. The system administrator is often saddled with tackling security, performance, data integrity, storage, planning, access control, backups, innovation, design, consulting, and so much more. No other role commonly must address all, if even any, of those functions.

For many reasons, system administrators often form the backbone of an IT departmental infrastructure. The operating system, because of its deep roots into storage and networking, and its close association with data and applications, sits in the position with the greatest control and visibility throughout the IT organization. System administrators may have little direct contact with end users but tend to be the nexus around which nearly all the infrastructure and support departments focus and rely.

System administration is generally seen as forming the largest group of focused IT professionals, as well, at least within the scope of purely technical roles. End user touching positions such as helpdesk or deskside support may involve larger headcount, but much of those roles often involves customer service or end user training that accounts for much of their time spent during a day. Within the technical realms of networking, storage, applications, databases, and so forth, it's most likely that systems will represent the largest portion of your team and cover the broadest range of skills.

Being so central, core, and often large, you can think of systems as often functioning as the glue that holds the IT department together; or you can think of systems as being the hub onto which all other IT disciplines will tend to attach. The following diagram demonstrates this:

Figure 1.3 - Systems at the Core of the IT Ecosystem

Unlike networking who might know little or nothing of systems, for example, systems professionals cannot be unfamiliar with networking concepts, even extremely detailed ones. Or unlike a database administrator who can generally just assume that someone else will handle backups properly, the system administrator is often the comprehensive point of communications that must understand how the database is talking to storage, how storage is quiescing data, and how a backup is being decoupled from the system. Just as the operating system is essentially the communications hub of a workload, the system administrator naturally becomes the point of holistic understanding and management of a workload.

In my experience, systems administration often is expected to work as a full consulting peer to other teams. I've seen applications, database, and networking teams all turn to systems departments when they are looking for broader experience within their own realms. The nature of system administrators to have to deeply understand not only their own realm, but all adjacent ones, leads to a department that can often act as a consultant to others, much as how IT often does so to the business at large.

Naturally, system administration gains the greatest overall knowledge of the overall workings of a business from the IT perspective which because it touches nearly every aspect of a business, is often one of the most important views that there can be of an organization. This will lend itself to the system administrator also being in a key role to report on and potentially advise business leaders within the organization as to issues and opportunities. Of all specialized roles within IT the system administrator is the most likely to interact heavily with the business on a large scale.

Now we have a good feel for how our role is going to fit into the IT department and the business. Next, we will learn about to tackle learning this role in the first place.

If you are not already working as a system administrator, you might feel that getting the necessary experience and training to become one will prove to be difficult. This is not true. In fact, moreso than nearly any other area of IT you can experiment with and practice nearly every aspect of system administration for very low cost in a home lab. System administration may have a lot of facets and favor those with deep knowledge, but it is also ultimately accessible to those willing to put in the effort.

In the following sections, I will describe some ways that you can get started on learning to become a System Administrator.

Build a home lab

Few people are as big of a proponent of building a home lab as I am. My own home lab experience has been among the most valuable of my career. And in my role as a hiring manager, no matter how much office experience someone has, I always want to know what they do at home on their own time. Home labs show dedication, interest, passion, and provide unique opportunities to really learn a process from end to end. In my decades in system administration, I have found that it is actually a rare administrator who has gained any real experience running a system from cradle to grave and understanding what a system and workload lifecycle are really like. Having that experience, even if only at home, can be a major point of differentiation in an interview or on the job. It can easily be the stand out factor that makes all of the difference.

Virtualization has brought the cost and complexity of a home lab from something that was onerous in the late 1990s to something that is almost an afterthought in the 2020s. The resource needs of a lab are often incredibly small, while the spare resources of most computers, by comparison, are large. And this is before we consider cloud computing and the ability to do lab work using on demand cloud resources today. While not free, cloud computing is often extremely affordable as another resource that a home lab can deploy expanding the student's experience even further. Leveraging both gives you even more to put on a resume and to discuss at the interview table.

Of course, to make this experience valuable, you must really dedicate a lot of effort - every bit as much, or more, than you would if it was a production system at a paying job. Take the time to automate, secure, monitor, update, and maintain real workloads even if you just use them around the house. A web server, a house VoIP phone PBX, email, instant messaging, a database, a media server, a remote access server, a DNS filter, you name it. There are many things that you can run in your own lab that are similar, or even identical, or heck even superior, to workloads that you would see in a common business environment. Do not be afraid to use your lab to make interesting and fun projects for family and friends, too.

Getting family and friends involved

Building a home lab that is only for yourself can be challenging because you never get to see your environment being used. When possible, I highly recommend recruiting friends and family to get involved as well. This lets you get end user feedback and experience systems more like they are meant to be used. Running systems only for yourself is far more difficult to get the experience that would match what happens in a business.

When I was first running my own home labs for system administration, I ran a family website, an email server, and probably the most interesting a full PBX. A VoIP PBX (or Voice over IP Private Branch Exchange) for home can be great because it tends to be complex compared to most home style workloads, can be used completely realistically as a valuable part of your family's communications platform and few experienced IT professionals have worked with them themselves.

In the PBX example, you can approach it with having extensions in multiple rooms and offering extensions to family members outside of your physical home. Using a PBX to build a free calling mechanism for family to stay in contact with each other easily can be truly utilitarian beyond the IT experience that it provides. Building something that provides real value and gets really used will not only make your projects more enjoyable but also makes the educational experience more like the real world.

Part of the secret to leveraging the home lab experience is to not only do everything right as you would in the best of businesses, but also to be prepared to discuss and explain in detail what you have done and how that meets and possibly exceeds the experience that someone is going to have gained from having worked in more typical work environments. Highlighting best practices, holistic management, experimentation, and the chance to have worked on the latest technologies can make all the difference between an interviewer dismissing home experience as inconsequential to them feeling that their own staff may be ill prepared to interview you due to your broader experience base and more up to date knowledgebase!

Start as a generalist and progress onto a specialist in the System Administrator field

Beyond your home lab there are other ways to gain experience before attempting to jump straight into the big business world where system administrators are most likely to be found. Of course, a common path taken is to work as a generalist in a small company where systems administration is part of the job. This approach does work but does not work often or as well as is generally expected. Making the leap from small business generalist to big business specialist can be almost as challenging as going directly from student to big business specialist in a single leap. So, we need to look at some ways to either get there directly or ways that we can assist in the transition.

Volunteer for non-profits or non-business organizations

Another option is volunteering. Non-profits and other non-business organizations often need resources that they have no means to find, attract, or afford. Volunteering in these types of organizations can be a great way to gain production experience and to demonstrate how dedicated you are to your craft. They can rarely provide any sort of mentoring or guidance, which is a significant negative, but they also generally provide little oversight or structure and will appreciate the effort that you put in, giving you more latitude to choose approaches that benefit your experience as much as benefiting them. As a volunteer, you will easily find that you can spend more time focusing on real solutions for the organization rather than needing to play politics or attempt to please a manager since you are working for free and have no job to defend or promotion to angle toward.

One of my better career decisions early on in my system administration years was to use my free time for a couple of years to volunteer as the sole system administrator of an all-Linux environment at a K-12 school. The school had almost no existing computers at all, and certainly no network or Internet access, and I was able to put on my engineering hat and design their entire environment including desktops, servers, telephones, networking, printing, and others. And once implemented, I was able to switch to my administrator's hat and manage the entire desktop, server, telephone, and storage environment from cradle to grave built entirely from Linux. I was able to do everything and do it all myself. I paid the price as an administrator for my oversights as an engineer. I saw the initial cost and design considerations and how they played out with end users. I deal with a workload from procuring the initial hardware, installing the software, configuring, deploying, training, and supporting. That experience didn't just catapult me forward in my career in the short term but is so significant that it is a valid talking point even in senior enterprise CIO level interviews much later in my career.

Presenting this experience in a Fortune 100 interview was very easy. Showing that I understood the processes and ramifications of decisions was great. Being able to demonstrate the ability to research, plan, and action every step from racking a server to production support of a system was unique. In the enterprise those skills rarely exist because typically different teams often handle each different portion of the process. Being able to show holistic oversight of the entire process brough much to the table. This range and type of experience was something other candidates did not have and made a huge different in my career trajectory.

Do not overlook doing small time support for friends and family. Almost everyone knows someone who could use some technical support at home and mostly what people need at home is system administration support! It goes without saying that this will be desktop support rather than support of servers. Real world support of desktops is certainly better than no experience at all. Putting together many different types of experience is best to show the most well-rounded total portfolio.

Self-study

Learning system administration has the benefit of there being so many great resources available on the market from books and certifications to online communities and videos. Both structured and unstructured materials abound, and thousands of passionate professionals are always online in communities just waiting to answer questions and help others in the field.

It is tempting to seek out classes or the newer trend of boot camps to learn IT skills in general, and system administration in specific, but in most cases I will advise against that approach. Boot camps especially are designed to teach you only very specific skills necessary to get up and running and through an interview process on a specific process and cannot take the necessary time to deeply teach topics. This is dangerous and often leads to long term career failure. More formal traditional class style education can be better, and some people learn especially well that way, but there are caveats.

Typically, classroom learning happens at a very slow pace and is not flexible to your schedule. This is only so much of a problem, but you lose the opportunity to move as fast as possible while focusing on what matters to you most. Very rarely can you find classroom education that is current and relevant. While possible, it is almost unheard of and there are seldom checks and balances to verify that what is being taught is useful or even accurate. Most importantly is that classroom learning is not a sustainable approach. Once you are working in the field you will not have the time to continuously take classes to keep your knowledge up to date, nor will you be able to find classes that provide for the immediate needs of the next project or challenge. Classroom learning is essentially a special case that is only available to people before they enter their careers. It is not a useful approach in an ongoing way. Because of this, learning through a classroom setting does not demonstrate a practical skill for the workplace.

Instead, teaching yourself through books, articles, experimentation, online communities, and other self-starting means that show that you are able to learn without someone else's assistance are crucial both because you will almost never have someone available to know everything that you need to know before you do and be prepared to teach it to you, and because you will be spending your entire career constantly needing to stay current and learn new technologies and techniques. Any serious profession involves lifelong learning, but very few demands it to the degree that is required in IT. Showing the ability to teach yourself whatever is needed is a very big deal to a potential employer.

Learning to teach yourself gives you a broader range of potential learning opportunities, more flexibility in how and when you learn, the potential to start learning at any time, and the ability to move at the fastest possible pace. Of course, as a reader of this book, you have already taken at least this step in seeking out formal, self-paced learning.

Age does not matter

You might be wondering when you should consider tackling the beginning of studies into system administration. Is this something you begin at university? Do you need several years in the field working in networking or perhaps in helpdesk before you can move over to systems? Do you start running desktop support before you can switch to servers?

Great questions, I am glad that you asked. The answer is actually: You can start at any age! For real, I mean it. While landing a big corporate job might be out of reach for almost any high schooler getting hands on experience and education in systems is anything but. Between books, online videos, articles, home labs, volunteer opportunities, and sometimes even actual paying work with smaller companies a high school student has so many potential avenues to build a resume, get an interview, and get their foot in the proverbial door that the best time for anyone to start studying to be a system administrator is always today.

A big advantage of IT compared to most other fields is that we have a lot of industry-backed certifications that can do quite a lot for building a resume early on in your career and almost none of them require previous industry experience or that you be of any specific age or have a degree. In fact, many high school trade programs specifically target assisting students get their first industry certifications during high school. A motivated student could go far beyond what any normal school program could provide and quite quickly building a resume of both experience and certifications.

Getting a great job might require turning eighteen before getting a chance to really apply, but there is every possibility that a high school student could use their school years to build up an impressive level of study and walk almost immediately into a great starting role and move up very quickly. If a student started studying towards an IT career in middle school and stuck with it for four to six years, it would not be unreasonable to have gained more hands-on experience than many mid-career working adults. Not every business will be keen to hire someone so young regardless of what they can bring to the business, but we don't want every job, we want that one job and a business that is really serious about their IT is going to potentially view that candidate as being the most impressive.

Internships

In traditional trade situations the standard means of gaining experience was through an internship. Today this is much less common, yet it still exists. Most internships will be unpaid. Internships can be extremely valuable, especially for younger learners generally of high school or university age, because they can provide access to traditional businesses, rather than non-profits or charities, and should provide a mentor to guide you while learning. Even paid internships are assumed to be quite poorly paid making them quite difficult for someone beyond student ages to typically consider.

Be wary when approaching a company to discuss an internship. Most companies offering internships do so believing that they are getting low-cost labor when the purpose of an internship is supposed to be to give back to the industry and community. In doing this we often see no mentoring or guidance. The intern is just used to do jobs that require no skill and teach them nothing. In companies that do at least attempt to treat an internship properly it is quite common for the company to lack the skill or resources to mentor and simply not be able to do so. Potentially the worst result is being mentored by someone who is not well trained themselves and the intern being taught incorrectly and coming out with fewer skills than when going in!

Do not let this dissuade you from investigating the option of finding an internship. Internships look great on a resume and even poor experiences can be used to make good talking points in an interview. If you were taught everything incorrectly, perhaps you can present a what not to do that was taught to you by example! Just be wary and do not think that all internships should be bad or involve doing menial labour or not being mentored. A true internship is about providing you an education via a mentor who is able to show you how the job is done in the real world. It is not an excuse to get cheap labor. That does not mean that you will never do any work in an internship. You very well may and getting your hands dirty in a real environment is part of the goal. But a true intern is there to learn and not to work per se.

Now we have a good idea of how to approach positioning ourselves to moving into a system administration role as well as how to tackle the problem of continuous life-long learning. Now we can look at the IT professional in the context of the business as a whole.

I would be remiss to go this far talking about systems administration within IT if I did not step back for a moment and talk about the role of the IT professional in the broader sense, as well. It's all too easy to assume that you've picked up this book because you are an experienced IT professional looking to hone their craft, tweak their skills, or maybe make some adjustments to get yourself into the Linux Administration path and probably many of you are doing just that. But some of you might be new to the field and wondering if Linux Administration specifically, or systems administration more generally, are the area in which you are wanting to focus your attentions.

First, I have to say, that after well more than thirty years in the field very little else is as generally rewarding as working in IT. IT isn't just an enormous field with countless opportunities, but it is one that gives you geographic opportunity, the chance to explore any business market (finance, manufacturing, insurance, healthcare, veterinary, hospitality, research, government, military, journalist, media, software, tourism, marketing, and so on), and myriad different roles within the field. Everyone's IT journey is a unique one, and the chances for your career to be rewarding and exciting are higher than in nearly any other field. IT is uniquely positioned as not only a technical field, but also as a customer service field, but most importantly as a core business function. IT builds and maintains the business infrastructure. As such we are a key player in every aspect of the internals of an individual business, even before we consider the fact that we work in all businesses!

As we alluded to previously, system administrators are, in some ways, the IT of IT, the most core and broadly reaching role within the IT department acting much like a meta-IT role often combining or connecting all of the other roles.

One area of IT, though, that I think is worth special mention is the topic of the general titles of Information Technology and IT Professional. Let's look at both closely:

• First, Information Technology. Certainly, this is what IT stands for, but truly IT is not about technology. It's about the information, communication, storage, security, and the efficiency of the business - the infrastructure of the business. As such, technology naturally is assumed, but sometimes IT can be about whiteboards, notepads, and just bringing good decision making and common sense to the organization. I often compare IT to legal and accounting departments: each has a focus, but each also is just part of the business itself.
• Second, we call ourselves professionals because, well to be honest, because it sounds great. Everyone wants to be called a professional. This aligns us with doctors, lawyers, civil engineers, and so forth. But the truth is, none of those are good analogues to what we do in IT. All those fields start with stringent certifications, follow exacting rules, and would be just following the script if their approaches were applied in IT. There is a reason that IT certifications are almost exclusively about products rather than roles, and that is because the concept of certifying someone for an IT role conceptually doesn't really make sense. Why is that?

IT roles cannot really be certified in a meaningful way because if you could codify IT, you could automate IT, but you can't. IT is primarily creative and works as a business function more like the CEO than to any other department. IT's job is to maximize the profits of the business through improvements and good decision making in business infrastructure, which is an extremely broad mandate. The CEO has the same mandate, just without the limitation of being focused on the infrastructure. You would never certify someone as a CEO Professional, that's crazy. CEOs are totally creative, wild, unique. IT is the same way or should be. We would never accept a CEO that just did what everyone else did, there would be no value to them. Business professionals like the CEO or the IT department are there to take massive amounts of information and training; add common sense, experience, and creativity; and then apply all of that to a unique business in relationship to its customers, market, regulations, and competition. Almost nothing in these roles is repeatable on a large scale.

At the end of the day, the IT department, whether just one person or one hundred thousand people, has the singular function of helping the company to maximize profits. To do that we have to understand the business inside and out, business as a general concept, technology, decision making, risk and reward, and so on. You can't make those kinds of business decisions, take those kinds of profit risks, if you are tied to the confines associated with professionals.

A doctor, as a prime example, has so many strict rules to follow and everything centers around their certification process and their overall focus is all about avoiding mistakes. A doctor will prioritize any number of lives lost through inaction over being the direct cause of a death themselves. The professional approach is to avoid mistakes at the tactical level, while eschewing the strategic level.

IT, like any business function, is the opposite. We must look at the overall risk assessment, calculate potential reward, and make decisions that, mathematically, make sense for business profits. As IT professionals, it's not just okay to risk losing a patient from time to time, but if we never lose one, chances are pretty good that we are too risk averse to make good decisions. IT should never be about avoiding failure at all costs (or even just at irrational costs), but about choosing the level of risk that is dictated as wise based on the math and logic of the situation. For this reason, I often refer to those in our field as IT Practitioners because this better reflects the proper mindset that we should have when properly representing our field in a company that takes the role of the IT department seriously (and, by extension, takes itself seriously.)

The fallacy of success at any cost

Something that I have heard from businesses, which should instantly have set off alarms in the minds of every executive and manager there, is concepts like we cannot have server downtime at any cost. This comes up from the normal course of risk assessment and discovery. System engineers ask what the value of a system is so that they can gauge the needs of risk mitigation only to be told something like downtime is not an option or we have to be up one hundred percent of the time, at any cost.

Of course, if we really take time to think about it, we know that they are just avoiding the question by stating something absurd. We are left with nothing to work with, no way to know how to approach system design. No system has zero chance of failure, that is not possible. Saying that we must protect against all possible failures at any cost means that to even attempt to fulfill that demand that we just consume literally all resources that the organization can provide purely for risk mitigation. Any IT department attempting to follow that directive with any sincerity would bankrupt the firm. It should be obvious that no workload, in any scenario, is worth that. Yet, it is surprisingly often that company management expects IT to work with little other direction in determining which workloads get what degree of protection.

Knowing business, finance, and ITs place in the business are necessary when IT has to force the business to act rationally and effectively.

Don't be surprised to find that in your role as an IT practitioner, especially one in system administration, that you will be playing an instrumental role in guiding and advising the company in many different business capacities. While we would hope that other roles, such as CEO and CFO were more broadly trained in business practices, the harsh reality of business is that it is far easier to become one of those roles with little or no training than to be an effective system administrator without that business training. IT roles at a decision making or influence level require so much business knowledge and practical business thinking to do the job in with any semblance of success that often we must act as advisors to every level of the business as often there is little business experience elsewhere in the organization.

Hopefully at this point you have some solid understanding of what we mean by the concepts of System Administrator and Linux Administrator and how that role will potentially fit into an organization. We will treat the rest of this book as addressing both administration and engineering within the systems realm and with an eye towards the unique needs of Linux as our system family of choice.

We have looked at understanding what systems are, and how they fit into the IT departmental offerings. We have looked at the engineering and administration aspects of roles. We have broken down how a role can be dedicated or just one of many hats worn by a specialist. We have even broken into understanding more about IT as a general field and how we could go about acquiring an education to let us make the move into an IT career.

At this point, I think that we are ready to start to tackle the specific best practices of Linux Administration!

In our next chapter we are going to move on from examining what it means to be a System Administrator and focus in the same way on what it means for an operating system to be a Linux Operating System and look at how Linux fits into the technology ecosystem, who the key players are, and how we select Linux for our workloads.