Continuous Delivery and DevOps: A Quickstart guide

5 (3 reviews total)
By Paul Swartout
  • Instant online access to over 7,500+ books and videos
  • Constantly updated with 100+ new titles each month
  • Breadth and depth in over 1,000+ technologies
  1. Evolution of a Software House

About this book

For a while now, there has been a buzz around the IT industry regarding continuous delivery and DevOps. This book will provide you with some background information into these two new kids on the block and how they can help you to optimize, streamline and improve the way you work and ultimately how you ship quality software.

"Continuous Delivery and DevOps: A Quickstart guide" will provide you with a clear and concise insight into what continuous delivery and DevOps are all about, how to go about preparing for and implementing them and what quantifiable business value they bring. Included within are some tricks and trips based upon real world experiences which may help you reduce the time and effort needed if you were to go it alone.

In this book, you will be taken through a journey of discovery starting with real world successes, how you should prepare, plan for and implement CD and DevOps and what the pitfalls are along the way

We will start looking at the evolution of a typical software house from fledgling start-up through the growing pains that comes with global success to a best of both worlds. We’ll delve into the many aspects of what they did to complete this evolution covering topics such as how they realized there was a problem to solve, how they set about preparing for and implementing continuous delivery and DevOps and what tools, techniques and approaches they used along the way – some technical and some not so. If you work within an organization that delivers software, you will be able to plot where you are on the evolutionary scale, understand where you need to do to be more effective, cherry pick the tools, techniques and approaches that work for you and realize the best of both worlds.

"Continuous Delivery and DevOps: A Quickstart guide" will provide you with the background and information you need to realize the benefits within your own business

Publication date:
November 2012
Publisher
Packt
Pages
154
ISBN
9781849693684

 

Chapter 1. Evolution of a Software House

Before we get into the meat of what continuous delivery (CD) and DevOps are all about, I would like to introduce you to ACME systems. This fictional web-based software business started out—as these things sometimes do—in a back bedroom of one of the founders. The founders were visionaries with big ambitions, good ideas, dreams, and a little bit of cash.

After a few years of hard work, determination, and much blood, sweat and tears, the dream of the founders was realized. The business was recognized as a leader in their field and was acquired by a multinational corporate. This acquisition brought with it the funding and resources needed to allow the business to grow and expand to become a global player. That is all well and good, but the story doesn't end there. With corporate owners comes corporate responsibilities, corporate rules, corporate bureaucracy, and corporate processes.

The ACME systems team—now embellished with the corporate owner's name and now operating under corporate governance—started to find it increasingly difficult and painful to deliver quality software. They adopted and adhered to the parent company's processes to improve quality and reduce risk but this made the simple task of delivering software, laborious and extremely complex.

They came to an evolutionary crossroad and had to make a decision—live with the corporate baggage that they had inherited, try and get back to the good old days and good old ways, which had reaped benefits previously, or try to take the best of both worlds.

It so happens that they made the right choice to implement CD and DevOps and did indeed reap the best of both worlds.

The following diagram gives an overview of the evolution of ACME systems:

Overview of ACME systems evolution

Over the next few pages, we'll go through this evolution in more detail. As we do, you may recognize some familiar traits and challenges.

Note

The name ACME is purely fictional and based upon the ACME Corporation first used in Road Runner Cartoons in the 1950s—just in case you were wondering.

We'll start with the initial incarnation of the ACME systems business, which for want of a better name, will be called ACME systems Version 1.0.

 

ACME systems Version 1.0


Some of you have most probably worked for (or currently work for) a small software business. There are many such businesses scattered around the globe and they all have one thing in common: they need to move fast to survive and they need to entice and retain customers at all costs. They do this by delivering what the customer wants just before the customer needs it. Deliver too soon and you may have wasted money on building solutions that the customer decides they no longer need, as their priorities or simply their minds have changed. Deliver too late and someone else may well have taken your customer—and more importantly your revenue—away from you. The important keyword here is deliver.

As previously mentioned, ACME systems started out in humble beginnings; the founders had the big vision and could see a gap in the market for a web-based solution. They had entrepreneurial spirit and managed to obtain backing from a few parties who injected the life blood of all small businesses—cash.

They then went about sourcing some local, keen, and talented engineers and set about building the web-based solution that bridged the gap, which they had seen before anyone else could.

At first, it was slow going and hard work; lots of pre-sales prototypes needed to be built in a hurry—most of which never saw the light of day—some went straight into production. After many long days, long nights, long weeks, long weekends, and long months, things started to come together. The customer base started to grow and the orders started rolling in; so did the revenue. Soon the number of employees were in double figures and the founders had become directors.

This is all well and good but what has this got to do with CD or DevOps? Well everything really. The culture, default behaviors, and engineering practices of a small software house are what would be classed as pretty good in terms of CD and DevOps. For example:

  • There are next to no barriers between developers and the Operations teams—in fact, they are generally one and the same

  • Developers normally have full access to the production environment and can closely monitor their software

  • All areas of the business are focused on the same thing, that is, getting software into the production environment as quickly as possible and delighting customers

  • Speed of delivery is of the essence

  • When things break, everyone swarms around to help fix the problem

  • The software evolves quickly and features are added in incremental chunks

  • The ways of working are normally very agile

There is a reason for stating that the culture, default behaviors, and engineering practices of a small software house would be classed as pretty good rather than ideal. This is because there are many flaws in the way a small software house has to operate to stay alive:

  • Corners will be cut to hit deadlines, which will compromise software design and elegance

  • Engineering best practices will be compromised to hit deadlines

  • Testing will not be in the forefront of the developer's mind and even if it were, there may not be enough time to work in a test-driven development way

  • Source and version control is not used religiously

  • With unrestricted access to the production environment, tweaks and changes can be made to the infrastructure

  • Software releasing will be mainly manual and most of the time an afterthought of the overall system design

  • At times a rough and ready prototype may well become production code without the opportunity for refactoring

To emphasize this, let's have a look at a selection of typical conversations between the Development and Operations teams and the management team of ACME systems.

Developer/Ops

Management

We would like to invest some time in implementing a source control system.

Is it free? We don't have time.

We would like to invest time in developing a fully automated test suite.

Is it free? We don't have time.

We would like to invest time in implementing automated server provisioning.

Is it free? We don't have time.

The production environment has crashed.

You're not going home until it's fixed. I'll get the pizza!

This prototype is rough and ready and needs to be rewritten before we hand it over to our customers.

Don't worry, you'll get time to rewrite it.

The prototype we are now using in production keeps crashing.

You're not going home until it's fixed. I'll get the pizza!

We want to work in a test-driven development mode.

That will only slow things down and we don't have the time.

I can manually hack the production server to improve performance and stability to overcome the issues we're having.

I fully trust your judgment on this, just get it done quickly.

The manual hack I did last week has caused the disks to fill up and the production server has crashed.

You're not going home until it's fixed. I'll get the pizza!

We'll now have a look at the software delivery process for ACME systems Version 1.0, which, to be honest, shouldn't take too long.

Software delivery process flow Version 1.0

The following diagram gives an overview of the simple process used by ACME systems to deliver software. It's simple, elegant (in a rough and ready kind of way), and easy to communicate and understand.

Overview of ACME Version 1.0 software delivery process

Let's move forward a few years and see how ACME systems is doing and gain some insight into the benefits and pitfalls of being the leader of the field.

 

ACME systems Version 2.0


The business has grown in size, numbers, and turnover. The customer base spans a number of countries and the ACME software platform is being used by millions of customers on a daily basis. ACME systems is well established, well renowned, and recognized as being at the forefront in their area of expertise.

So much so, that the board of ACME systems is approached by a well-known and established multinational corporation and discussions are entered into regarding an acquisition. The discussions go very well and an agreement is reached; ACME systems is acquired outright. The board members are extremely happy with this and the business as a whole sees this as a positive recognition that they have at last reached the big time.

At first everything is good—everything is great in fact. The ACME systems team now has the backing they need to invest in the business and be able to scale out and obtain a truly global reach. They can also focus on the important things such as building quality software, re-architecting the software platform to be scalable and performant, investing in new technologies, tools, and R&D. The drier side of business—administration, program, and project management, sales, marketing and so on—can be passed to the new parent company that has all of this in place already.

The ACME systems team moves forward in excited expectation. The level of investment is such that the software engineering team doubles in size in a number of months. The R&D team—as they're now called—introduce new tools and processes to enable speedy delivery of quality software. Scrum is adopted across the R&D team and the opportunity to fully exploit engineering best practices is realized. The original ACME platform starts to creak and is showing its age, so further investment is provided to completely re-architect and rewrite the platform using the latest technologies. In short, the R&D team feels that it's all starting to come together and they have the opportunity to do it right.

In parallel to this, the Operations team is augmented into the parent's global Operations organization. On the face of it this seems a very good idea there are data centers filled with cutting edge kit, global network capabilities, and scalable infrastructure. Everything that is needed to host and run the ACME platform is there. Like the R&D team, the Operations team has more than they could have dreamed of. In addition to the tin and string, the Operations team also has resource available to help maintain quality, control change to the platform, and ensure the platform is stable and available 24 x 7.

Sitting above all of this, the parent company also has well-established program and project management functions to control and coordinate the overall end-to-end product delivery schedule and process.

On the face of it everything seems rosy and the teams are working more effectively than ever before. At first this is true but very soon things start to take a downward turn. Under the surface, things are not all that rosy at all.

We'll shift forward another year or so and see how things are:

  • It is getting increasingly difficult to ship software—what took days now takes weeks or even months

  • Releases are getting more and more complex as the new platform grows and more integrated features are added

  • Developers are now far removed from the production environment and as such are ignorant as to how the software they are writing performs, once it eventually goes live

  • Focus is being lost as new people come into the organization with no prior knowledge of the platform or the business

  • There is a greater need to provide proof that software changes are of the highest quality and performance before they can go anywhere near the production servers

  • The software quality is starting to suffer as last minute changes and frantic bug fixes are being applied to fit into release cycles

  • Scope is being cut at the last minute as features don't fit into the release cycles, which is leaving lots of redundant code lying around

  • More and more Development resource is being applied to assisting releases, which is impacting development of new features

  • Deployments are causing system downtime—planned and unplanned

  • Deadlines are being missed, stakeholders are being let down, and trust is being eroded

  • The once glowing reputation is being tarnished

The main problem here, however, is that this attrition has been happening very slowly over a number of months and not everyone has noticed—they're all too busy trying to deliver.

Let's now revisit the process flow for delivering software and see what's changed—it's not a pretty picture.

Software delivery process flow Version 2.0

As you can see from the following diagram, things have become very complicated for the ACME team. What was simple and elegant has become complex, convoluted, and highly inefficient. The number of steps and barriers have increased, making it extremely difficult to get software delivered. In fact, it's increasingly difficult to get anything done.

Overview of ACME Version 2.0 software delivery process

Not only has the process become very inefficient—and to all intents and purposes broken—but the dialogue and the quality of the communication have also broken down. Let's again review a typical discussion, this time between the R&D and Operations teams (who you'll remember were pretty much one and the same in the early days of ACME systems).

R&D

Operations

We need to urgently fix a performance bug and need to check some of the system configuration values for one of the production servers.

Have you obtained permission to see this information?

We have now obtained permission to see the system configuration values for one of the production servers. Can you please supply it?

Which server?

We've no idea—the one running the secure web login service.

You'll have to be more specific, we've got hundreds of servers.

We've checked through an old project plan and it's listed as DC02MM03DB16.

That sounds like a database server, it shouldn't be running code. Application servers normally have AP rather than DB.

Maybe it's DC02MM03AP16?

That sounds more like it. We've found it. What information do you need?

The heap size for the JVM.

It's 16 GB. However, the server's only got 8 GB of RAM.

That's not good, no wonder there are performance issues. Can we increase it up to 16 GB? That's the minimum space the application needs.

You'll have to raise a change ticket.

Okay—for now I'll set the deployment parameters to use 8 GB, can you update the system configuration?

That's an infrastructure change. You'll have to raise an infrastructure change ticket.

What about spinning up a couple of new servers so we can spread the burden?

That's a DC infrastructure change. You'll have to raise a DC infrastructure change ticket.

We now have all the tickets raised and signed off. We're now ready to deploy this fix.

As the heap size has changed, we need to see the results from integration, performance, and functional tests as this deployment could have an adverse impact on the production platform.

Arrggghhh! I give up!!

 

Okay, so this may be a little over the top but it just goes to highlight the massive disjoin between the R&D and Operations team(s). This communication is now normally via e-mail.

As was previously stated, not everyone noticed the attrition within the organization—luckily a few brave souls did.

A few brave men and women

A small number of the ACME team can see the issues within the process as clear as day and become determined to expose them and more importantly sort them out—it is just a question of how to do this while everyone is going at 100 MPH to get software delivered at all costs.

At first, they form a small virtual team, start breaking down the issues, and try to implement and/or build tooling to ease the pain:

  • Tools to automate software builds

  • Automated testing suites

  • Continuous integration (CI) systems

  • Automated no downtime deployment tools

  • Clones of the production environment for functional and performance testing using virtualization technologies

This goes some way to address the issues but there is still one fundamental problem that tooling cannot address—the organization itself and the many disjointed silos within it. It becomes obvious that all the tools and tea in China will not bring pain relief; something more drastic is needed.

The team refocuses and works to highlight this now obvious fact to as many people as they can up and down the organization while at the same time obtaining backing to address it.

We're now going on to the third stage of the evolution where things start to come back together and the team regains their ability to deliver quality software when it is needed.

 

ACME systems Version 3.0


The CD team—as they are now called—gets official backing and becomes dedicated to addressing the culture and behavior and developing ways to overcome and/or remove the barriers. They are no longer simply a Development team; they are a catalyst for change.

The remit is clear—do whatever is needed to streamline the process of software delivery and make it seamless and repeatable. In essence, implement CD and DevOps.

The first thing they do is to go out and simply talk with as many people as possible, across the business, who are involved in the process of getting software from conception to consumer. This not only gathers useful information but also gives the team the opportunity to evangelize and form a wider circle of like-minded individuals.

They have a vision, purpose, and they passionately believe in what needs to be done.

Over the next few months they embark on (amongst other things):

  • Running various in-depth sessions to understand and map out the end to end product delivery process

  • Refining and simplifying the tooling based upon continuous feedback from those using it

  • Addressing the complexity of managing dependencies and order of deployment

  • Engaging experts in the field of CD to independently assess the progress being made

  • Arranging CD and DevOps training and ensuring that both R&D and Ops team members attend the training together (it's amazing how much DevOps collaboration stems from a chat in the hotel bar)

  • Reducing the many handover and decision making points throughout the software release process

  • Removing the barriers to allow developers to safely deploy their own software to the production platform

  • Working with other business functions to gain trust and help them to refine and streamline their processes

  • Working with R&D and Operations teams to implement other agile methodologies such as Kanban

  • Openly and transparently sharing information and data around deliveries and progress being made across all areas of the business

  • Replacing the need for complex performance testing with the ability for developers to closely monitor their own software running in the production environment

  • Evangelizing across all areas of the business to share and sell the overall vision and value of CD and DevOps

These initiatives are not easy to implement and it takes time to produce results but after some nine months or so the process of building and delivering software has transformed to the extent that a code change could be built, fully tested, and deployed to the production platform in under 18 minutes many times per day—all at the press of a button and initiated and monitored by the developer who made the change.

Let's have a final look at the software delivery process flow to see what results have been realized.

Software delivery process flow Version 3.0

As you can see from the diagram the process looks much healthier. It's not as simple as Version 1.0 but it is efficient, reliable, and repeatable. Some much needed checks and balances have been retained from Version 2.0 and optimized to enhance rather than impede the overall process.

Overview of ACME 3.0 software delivery process

This highly efficient process has freed up valuable Development and Operations resources so that they can get back to their day jobs—developing and delivering new software features and ensuring that the production platform is healthy and customers are again delighted.

The ACME systems team has got back its mojo and is moving forward with a new found confidence and drive. They now have the best of both worlds and there's nothing stopping them.

 

Summary


The ACME systems evolution story is not typical of the many software businesses out there today. As stated previously, you may recognize some of the traits and challenges and should be able to plot where you, your business, or your employer currently sit within the three stages detailed.

Whatever point in the evolution a software-based business sits, there are some very simple and fundamental truths that are universal:

  • Developers enjoy solving problems by writing code and delivering software for consumers to use

  • System operators enjoy solving problems by implementing technical solutions and running stable platforms

  • Businesses who rely on software and hardware to generate revenue want both to be optimal and work in harmony

Being in a position to fully realize and accommodate all of the above will reap rewards.

About the Author

  • Paul Swartout

    Paul Swartout has spent over 2 decades working in the IT industry. He has worked across several different industries and sectors and within organizations of various sizes, from start-ups to multinational corporates. He is and always has been passionate about quality software and how it is delivered. Since first encountering “Agile” he has been committed to the adoption and implementation of Agile techniques and approaches to improve the efficiency, output, and lives of everyone involved in software development. He strongly believes that CD and DevOps add massive value to the way software is delivered, and he wants to ensure as many people realize this as possible. Paul lives in a small seaside town in the southwest of the UK.

    Browse publications by this author

Latest Reviews

(3 reviews total)
Easy to find, relevant and met my expectations.
Excellent
Excellent
Book Title
Access this book, plus 7,500 other titles for FREE
Access now