DevOps with Windows Server 2016

Organizations work within the realms of economic, political, and social backdrops, and they have to constantly adapt themselves to a continuously changing environment. Economic changes might introduce an increase in competition in terms of price, quality of products and services, changing marketing strategies, and mergers and acquisitions. The political environment introduces changes in legislation, which has an impact on the rules and regulation for enterprise. The tax system and international trade policies are also examples of areas in which change can have an impact. Society decides which products and services are acceptable or preferred and which are discarded. Customers demand change on a constant basis. Their needs and requirements change often and this manifests in the systems they are using. Organizations not adept at handling changes in their delivery processes and who resist making changes to their products and features eventually find themselves outdated and irrelevant. These organizations are not responsive to change. In short, the environment is ever changing and organizations perish if they do not change along with it.

Software organizations with a traditional mindset release their products and services on a yearly or multi-year basis. Their software development life cycle is long and their operations do not have many changes to deploy and maintain. Customers demand more but they wait till the next release from the company. The organization is either not interested or does not have the capability to release changes faster. Meanwhile, if the competitor is able to provide more and better features faster, customers will soon shift their loyalty and start using them. The first organization will start losing customers, have reduced revenues, and fade away.

Generally, there are multiple teams behind any system or service provided to the customer. Typically, there is a development team and an operations team. The development team is responsible for developing and testing the system, while the operations team is responsible for managing and maintaining the system on production. The operations team provides post-deployment services to the customer. These two teams have different skills, experience, mindset, and working culture. The charter of the development team is to develop newer features and upgrade existing ones. They constantly produce code and want to see it in production. However, the operations team is not comfortable with frequent changes. The stability of the existing environment is more important to them. There is a constant conflict between these two teams.

There is little or no collaboration and communication between these two teams. The development team often provides code artifacts to the operations team for deployment on production without helping them to understand the change. The operations team is not comfortable deploying the new changes since they are neither aware of the kind of changes coming in as part of a new release nor have confidence deploying the software. There is no proper hand-off between the development and operations teams. Often, the deployments fail on production and the operations team has to spend sleepless nights ensuring that the current deployment is either fixed or rolled back to a previous working release. Both the development and operations teams are working in silos. The development team does not treat the operations team as equivalent to itself. The operations team has no role to play in the software development life cycle, while the development team has no role to play in operations.

Development goes on for multiple months before testing begins. The flow is linear and the approach is Waterfall, where the next stage in software development life cycle happens only when the prior stage is completed or nearing completion. Deployment is one giant exercise in deploying multiple artifacts on multiple servers based on documented procedures. Such practices have many inherent problems. There are a lot of features and configuration steps for large applications and everything needs to be done, in order, on multiple servers. Deploying a huge application is risky and fails when a small step is missed during deployment. It generally takes weeks to deploy a system such as this in production.

Software development enterprises often do not employ proper automation in their application lifecycle management. Developers tend to check-in code only after a week, the testing is manual, configuration of the environment and system is manual, and documentation is either missing or very dense, comprising hundreds of pages. The operations team follows the provided documentation to deploy the system manually on production. Often this results in a lot of downtime on production because smaller steps have been missed in deployment. Eventually, customers become dissatisfied with the services provided by the company. Also, this introduces human dependencies within the organization. If a person leaves the organization, their knowledge leaves with them and a new person has to struggle significantly to gain the same level of expertise and knowledge.

Organizations starts losing out to competition when they are not flexible to meet customer expectation with newer and upgraded products and services. The result is falling revenues and profits, eventually making them nonexistent in the marketplace. Organizations that do not innovate newer products and services consistently nor update them cannot provide exponential customer satisfaction.

One of the prime tenets of DevOps is collaboration. Collaboration means that different teams come together to achieve a common objective. It defines clear roles and responsibilities, overall ownership, accountability, and responsibility for the team. The team comprises both development and operations people. Together they are responsible for delivering rapid high-quality releases to the end customer.

Both teams are part of the end-to-end application life cycle process. The operations team contributes to the planning process for features, providing their feedback on overall operational readiness and issues regarding business application and services. Concurrently, the development team must play a role in operational activities. They must assist in deploying the release to production and provide support in terms of fixing any production issues that arise. This kind of environment and ecosystem fosters continuous feedback and innovation. There is a shared vision, where everyone in the team are working toward common goals.

Agility refers to the flexibility and adaptability of people, processes, and technology. People should have a mindset open to accepting change, playing different roles, and taking ownership and accountability. Processes would generally refer to the following:

Application lifecycle management

Wikipedia defines application lifecycle management as follows:

Application lifecycle management (ALM) is the product lifecycle management (governance, development, and maintenance) of computer programs. It encompasses requirements management, software architecture, computer programming, software testing, software maintenance, change management, continuous integration, project management, and release management.

Application lifecycle management (ALM) refers to the management of planning, gathering requirements, building and hosting code, testing code in terms of code coverage, unit tests, versioning of code, releasing code to multiple environments, tracking and reporting, functional tests, environment provisioning, deployment to production, and operations for business applications and services. The operational aspects include monitoring, reporting, and feedback activities. Overall, ALM is a huge area and comprises multiple activities, tools, and processes. Special attention should be given to crafting appropriate application lifecycle steps to induce confidence in the final deployed system. For example, processes can be implemented which mandate that code cannot be checked in the source code repository if unit tests do not pass completely. ALM comprises multiple stages such as planning, development, testing, deployment, and operations.

In short, ALM defines a process to manage an application from conception to delivery and integrates multiple teams together to achieve a common objective. The phases of a typical application lifecycle management process is shown in Figure 1. ALM is a continuous process that starts with the planning of an iteration, building and testing the iteration, deploying it on a production environment, and providing post-deployment services to the customer.

Feedback from customers and operations is passed on to the planning team, which eventually incorporates them into subsequent iterations, and this process loop continues.

Figure 1: Application lifecycle management phases

Software design should implement architectural principles that foster modularity, decomposition of large functionality into smaller features, reliability, high availability, scalability, audit capabilities, and monitoring, to name a few.

Automation plays an important role in achieving overall DevOps goals. Without automation, DevOps cannot achieve its end objectives. Automation should be implemented for the entire application lifecycle management, from building the application, to delivery and deployment to the production environment. Automation brings trust and a high level of confidence in the output from each phase of the software development life cycle.

The probability that deliverables are of high quality, robust, and relatively risk-free is quite high. Automation also helps in the rapid delivery of a business application to multiple environments because it is capable of running multiple build processes, executing thousands of unit tests, figuring out code coverage comprising millions of lines of code, provisioning environments, deploying applications, and configuring them at the desired level.

At first glance, it seems weird to talk about failure in a DevOps book that is supposed to assist with the successful delivery of software. Trust me, it is not! Failing fast and early refers to the process of finding issues and risks as early as possible within application lifecycle development. Not knowing the issues that arise toward the end of the ALM cycle is an expensive affair because a lot of work has already been done on it. Such issues might involve making design and architectural changes, which can jeopardize the viability of the entire release. If the issues can be found at the beginning of the cycle, they can be resolved without much impact to the release. Automation plays a big part in identifying the issues early and fast.

DevOps fosters a culture of innovation and continuous learning. There is a constant feedback flow regarding the good and bad, and what's working and what's not working in various environments. The feedback is used to try out different things, either to fix existing issues or find better alternatives. Through this exercise, there is a constant information flow about how to make things better and that in turn provides the impetus to find alternative solutions. Eventually, there are breakthrough findings and innovation, which can be further developed and brought to production.

Software applications and services need a physical or virtual environment on which they can be deployed. Typically, the environment is an infrastructure comprising both hardware and operating system on which software can be deployed. Software applications are decomposed into multiple services running on different servers, either on-premises or in the cloud. Each service has its own application and infrastructure configuration requirement. In short, both infrastructure and application are needed to deliver software systems to customers, and each has their own configuration. If the configuration drifts, the application might not work as expected, leading to downtime and failure. Modern ALM dictates the use of multiple stages and environments on which an application should be deployed with different configurations. For example, the application will be deployed to a development environment for developers to see the result of their work. It will also be deployed to multiple test environments, with different configurations, for executing different types of tests . It would also be deployed to a preproduction environment to conduct user acceptance tests, and finally, it will be deployed on a production environment. It is important to ensure that the application can be deployed to multiple environments without undertaking any manual changes to its configuration.

Configuration management provides a set of processes and tools which help ensure that each environment and application gets its own configuration. Configuration management tracks configuration items, and anything that changes from environment to environment should be treated as a configuration item. Configuration management also defines the relationships between configuration items and how changes in one configuration item will impact another.

Configuration management helps in the following ways:

The configuration settings applied to environments and application should also be monitored. Records for expected and desired configuration along with the differences should be maintained. Any drift from this expected and desired configuration can make the application unavailable and unreliable. Configuration management is capable of finding the drift and reconfiguring the application and environment to their desired state.

With automated configuration management in place, the team does have to manually deploy and configure the environments and applications. The operations team is not dependent on the development team for deployment activities.

Another aspect of configuration management is source code control. Software comprises code, data, and configuration. Generally, team members working on an application change the same files simultaneously. The source code should be up to date at any point in time and should only be accessible by authenticated team members. The code and other artifacts themselves are configuration. Source code control helps in increased collaboration and communication within the team, since each team member is aware of other team members activities. This ensures that conflicts are resolved at an early stage.

Multiple developers write code stored and maintained in a common repository. The code is normally checked-in or pushed to the repository when a developer has finished developing a feature. This can happen in a day, or it might take days or weeks. Developers might be working together on the same feature and they might also follow the same practices of pushing/checking-in code in days or weeks. This can cause issues with code quality. One of the tenets of DevOps is to fail fast. Developers should check-in/push their code to the repository often and as soon as it makes sense to check-in. The code should be compiled frequently to check that developers have not introduced any bugs inadvertently and the complete code base can be compiled at any point of time. If a developer does not follow such practices, then there is possibility of each developer having stale code in their local workstation not integrated with other developer's code. Eventually, when such stale and large codebase is integrated from all developers, it starts failing and becomes difficult and time-consuming to fix issues arising from it.

Continuous integration solves these kinds of challenges. Continuous integration helps with the compilation and validation of any code pushed/checked-in by a developer by taking it through a series of validation steps. Continuous integration creates a process flow consisting of multiple steps and is comprised of continuous automated build and continuous automated tests. Normally, the first step is the compilation of the code. After successful compilation, each step is responsible for validating the code from a specific perspective. For example, when unit tests are executed on the compiled code, code coverage can be measured to check which code paths are covered. This could reveal whether comprehensive unit tests have been written or whether there is scope to add further unit tests. The result of continuous integration is deployment packages that can be used by continuous deployment for deployment to multiple environments.

Developers are encouraged to check-in their code multiple times a day instead of after multiple days or weeks. Continuous integration initiates the execution of the build pipeline automatically as soon as the code is checked-in or pushed. When all activities comprising the build execute successfully without any errors, the build-generated artifacts are deployed to multiple environments. Although every system demands its own configuration of continuous integration, a typical example is shown in Figure 3.

Continuous integration increases the productivity of developers. They do not have to manually compile their code, run multiple types of tests one after another, and then create packages out of it. It also reduces the risk of introducing bugs into the code. It also provides early feedback to the developers about the quality of their code. Overall, the quality of deliverables is high and deliverables are delivered faster by adopting a continuous integration practice:

Figure 3: Sample continuous integration process

By the time the process reaches the stage of deployment, continuous integration has ensured that there is a functional application that can now be deployed to multiple environment for further quality checks and testing. Continuous Deployment refers to the capability to deploy applications and services to preproduction and production environments through automation. For example, Continuous Deployment could provision and configure an environment, deploy and configure an application on top of it. After conducting multiple validations, such as functional tests and performance tests, on a preproduction environment, the production environment is provisioned and configured, and the application is deployed to production environments through automation. There are no manual steps in the deployment process. Every deployment task is automated.

Continuous deployment should provision new environments or update existing environments. It should then deploy applications with newer configuration on top of it.

All the environments are provisioned through automation using principle of Infrastructure as Code. This will ensure that all environments, be it development, test, preproduction, production, or any other environment, are similar. Similarly, the application is deployed through automation, ensuring that it is also deployed uniformly across all environments. The configuration across these environments could be different depending the application.

Continuous deployment is generally integrated with continuous integration. When continuous integration has done its work by generating the final deployable packages, continuous deployment kicks in and start its own pipeline. This pipeline is called the release pipeline. The release pipeline consists of multiple environments, each consisting of tasks responsible for the provision of the environment, configuration of the environment, deploying applications, configuring applications, executing operational validation on environments, and testing the application on multiple environments. We will look at the release pipeline in greater detail in the next chapter and also in Chapter 10, Continuous Delivery and Deployment.

Employing continuous deployment provides immense benefits. There is a high degree of confidence in the overall deployment process, which helps ensure faster, risk-free releases on production. The chance of anything going wrong is drastically reduced. The team will have lower stress levels and rollback to a previous working environment is possible if there are issues with the current release:

Figure 4: Sample continuous deployment/release pipeline process

Although every system demands its own configuration of a release pipeline, a typical example of is shown in Figure 4. It is important to note that, generally, provisioning and configuring multiple environments is part of the release pipeline and approval should be sought before moving to the next environment. The approval process might be manual or automated, depending on the maturity of the organization.

Continuous delivery and continuous deployment might sound similar to many readers; however, they are not the same. While continuous deployment talks about deployment to multiple environments and finally to a production environment through automation. Continuous delivery is the ability to generate application packages in a way that they are readily deployable in any environment. To generate artifacts that are readily deployable, continuous integration should be used to generate the application artifacts. A new or existing environment should be used to deploy these artifacts, conduct functional tests, performance tests, and user acceptance tests, through automation. Once these activities are successfully executed with no errors, the application package is referred to as readily deployable. It helps get feedback faster from both operations and the end user. This feedback can then be implemented in subsequent iterations.

With all the previously mentioned DevOps practices, it is possible to create stable, robust, reliable, performant business applications and deploy them automatically to a production environment. However, the benefits of DevOps will not last for long if a continuous improvement and feedback principle is not in place. It is of utmost important that real-time feedback about the application's behavior is passed on as feedback to the development team from both end users and the operations team.

Feedback should be passed to the teams, providing relevant information about what is going well and, importantly, what is not going well.

Applications should be built with monitoring, auditing, and telemetry in mind. The architecture and design should support these. The operations team should collect telemetry information from the production environment, capture any bugs and issues, and pass this information on to the development team such that they can be fixed in subsequent releases. This process is shown in Figure 5.

Continuous learning helps make the application robust and resilient to failures. It also helps make sure that the application is meeting consumer requirements:

Figure 5: Sample continuous learning process