Hands-On Security in DevOps

5 (1 reviews total)
By Tony Hsu
  • Instant online access to over 7,500+ books and videos
  • Constantly updated with 100+ new titles each month
  • Breadth and depth in over 1,000+ technologies
  1. DevSecOps Drivers and Challenges

About this book

DevOps has provided speed and quality benefits with continuous development and deployment methods, but it does not guarantee the security of an entire organization. Hands-On Security in DevOps shows you how to adopt DevOps techniques to continuously improve your organization’s security at every level, rather than just focusing on protecting your infrastructure.

This guide combines DevOps and security to help you to protect cloud services, and teaches you how to use techniques to integrate security directly in your product. You will learn how to implement security at every layer, such as for the web application, cloud infrastructure, communication, and the delivery pipeline layers. With the help of practical examples, you’ll explore the core security aspects, such as blocking attacks, fraud detection, cloud forensics, and incident response. In the concluding chapters, you will cover topics on extending DevOps security, such as risk assessment, threat modeling, and continuous security.

By the end of this book, you will be well-versed in implementing security in all layers of your organization and be confident in monitoring and blocking attacks throughout your cloud services.

Publication date:
July 2018
Publisher
Packt
Pages
356
ISBN
9781788995504

 

Chapter 1. DevSecOps Drivers and Challenges

Due to the rapid release of cloud services, law enforcement, security incidents, and tenants' data protection, the security is indispensable to cloud/internet services. Moving security activities from right to left during the development lifecycle and having built-in security practices in the continuous integration pipeline are the goals of DevSecOps.

The business environment, culture, law compliance, and external market drive relate to how the DevSecOps security assurance program rolls out in an organization. The DevSecOps or security assurance program management involved with the whole organization across all business units and the key success to DevSecOps will require all stakeholders to agree with the goal and approaches.

We will cover the following topics in this chapter:

  • Security compliance (ISO 2700x, FIPS, CSA-CCM)
  • Legal/law compliance—General Data Protection Regulation (GDPR)
  • New technology (third-party, cloud, containers, and virtualization)
  • Cloud service hacks/abuse
  • Rapid release

As shown in the following diagram, this is how external drivers and challenges impact on a team when delivering secure cloud services:

 

Security compliance


For cloud services, it's very important to have security compliance-ready. Security compliance not only shows how the security controls of the cloud service meet security standards but also demonstrates security trustworthiness for customers and partners. Security compliance provides an overview of a security assurance program, but it won't specifically tell us which security technical approach it should apply. For frequent cloud service releases, constantly monitoring and auditing to meet security compliance can be a big challenge.

Although most cloud service providers are security compliance ready (ISO, PCI, FedRAMP, SOC, and so on), it's still the cloud service customer's responsibility to secure data and manage their own application compliance assessment. Both cloud service customers and providers need to maintain system or application audit logs, configuration lists, and change histories for compliance assessment. The compliance assessment should be considered a continuous activity—not a one-time audit check.

In this chapter, we will introduce key cloud services security compliance as a reference to building a security assurance program, and how these security compliance standards relate to DevSecOps.

ISO 27001

ISO 27001 is an information security management system (ISMS). It provides an overview of organization-level security assurance programs. ISO 27001 won't specify a technical security approach, but it provides a complete set of a security management programs. As the diagram shows, the segments in the upper parts may be more directly related to DevOps security practices, such as compliance, business continuity, operation security, access control, software development, cryptography, incident management, and communication. This will serve as a guideline to further developing our own DevOps security program:

We won't introduce ISO 27001 details, but the following table summarizes how ISO 27001 relates to each role and the DevOps team:

Role

Company/organization security policy

Operation or DevOps team

Development team

ISO 27001 chapters

5  Information security policies

6 Organization of information security

7 Human resource security

8 Assess management

15 Supplier relationships 

11 Physical and environmental security 

 

9 Access Control

10 Cryptography

12 Operation security

13 Communication security

17 Information security aspects of business continuity management

16 Information security incident management

18 Compliance; with internal requirements, such as policies, and external requirements, such as laws

19 Cloud services control

14 System development

10 Cryptography

9 Access control

ISO 27017 and ISO 27018

ISO 27018 is mainly for the protection of personally identifiable information (PII) in the cloud. It's an extended security compliance based on ISO 27001 and ISO 27002. On top of ISO 27001/27002, ISO 27018 additionally defines PII protection security requirements

ISO 27017 provides both service providers and cloud service consumers with the ability to implement security controls for cloud services. ISO 27017 is an extension to ISO 27002 to address cloud-specific security issues.

Cloud Security Alliance (CSA)

As there are many cloud security compliance methods out there, we may get frustrated trying to follow each of them. The CSA (Cloud Security Alliance)  Cloud Controls Matrix (CCM) consolidated most security compliance methods into one matrix called CCM. Take application and interface application security as an example—CCM includes all security compliance controls such as ISO, FedRAMP, and NIST 800-53 related to this area, and defines the control ID. The key benefit of referring to CCM is that we can simply focus on CCM and know all other security compliance regulations will be met as well. 

In addition, CSA provides a Consensus Assessments Initiative Questionnaire (CAIQ). It's a yes/no questionnaire for cloud consumers or cloud provides to do security self-assessment and to understand the requirements of security controls. Google Vendor Security Assessment Questionnaires (VSAQ) also provide a security assessment questionnaire in terms of Web Application Security, Security and Privacy Program, Infrastructure Security and Physical and Datacenter Security.

Furthermore, if you are looking for the top cloud threats and security control mitigations, Cloud Security Alliance (CSA) cloud top threats provide guidelines. At the time of writing, it defines the top 12 cloud threats, mappings to threat analysis, CCM/Control ID, and the domains of CSA Security Guidance reference. The following table shows related CSA security guides and how to apply security practices in your organization:

CSA security guides

What it is?

When to apply?

CSA Security Guidance reference

Cloud security white paper

If your organization needs a cloud service security guideline or white paper, this can be a good reference.

Cloud top threats

Top 12 cloud threats and mappings to threat analysis, CCM/Control ID, and domains of CSA Security Guidance reference

It can be the basis for cloud threat modeling.

CAIQ

Yes/no questionnaire

A list of yes/no questions for self-assessment to understand existing security control requirements.

CSA CCM

One consolidated worldwide security standard mapping

It's a great consolidated reference and includes most security compliance standards (ISO 27001, PCI, NIST, and so on). It's the only matrix you need to review security standards compliance.

Federal Information Processing Standards (FIPS)

The FIPS mainly defines minimum security requirements for the use of cryptographic modules. Every organization that is not going to get a FIPS certificate should also refer to it. It's highly recommended that you refer to Security Requirements for Cryptographic Modules to understand what cryptographic modules may be considered safe, legacy, or weak. 

For developers who would like to learn how to implement cryptographic modules correctly, the following resources are recommended.

  • OWASP Cryptographic Storage Cheat Sheet. 
  • OWASP Guide to Cryptography
  • OWASP Key Management Cheat Sheet

Here is a summary of the minimum security requirements for each cryptography algorithm and its usage:

Usage scenario

Unsafe cryptography algorithm

(key length)

Legacy Systems Only

Recommended cryptography algorithm

Symmetric encryption

Blowfish, DES, Skipjack, RC4

3 DES only when

(key 1 != key 2 != key 3)

AES > 128 bits

Asymmetric encryption

RSA (< 1024 bits)

RSA (1024 bits)

RSA (> 1024 bits)

Hash

MD5

SHA1 (1024 bits)

SHA256

Digital signature

RSA (< 1024 bits)

DSA (< 1024 bits)

ECDSA (<= 160 bits)

DSA (1024 bits)

RSA (1024 bits)

RSA (>=2048 bits)

DSA (>=2048 bits)

ECDSA (>=256 bits)

Hellman key exchange (DH)

DH ( < 1024 bits)

DH (1024-2047 bits)

DH (>=2048 bits)

ECDH(>-256 bits)

Center for Internet Security (CIS) and OpenSCAP – securing your infrastructure

The CIS defines security benchmarks and the National Checklist Program (NCP), defined by the NIST SP 800-70, provides guidance on the security configurations of the operating system, database, virtualization, framework, and applications.

The IT and operation team are primarily responsible for ensuring the security of the infrastructure. However, the development team may also share some responsibilities for securing the infrastructure. For example, the development team may decide to deliver the application package in the form of a container or to apply Infrastructure as Code frameworks, such as Puppet or Chef. These technologies allow development teams to define a secure configuration, even in the development stage, and the operation team just needs to apply the secure configuration definition during application deployment.

In addition, it's also the development team's job to provide a list of configuration changes for every release's deployment. This will allow the operation team to review if the configuration changes are secure and appropriate. Due to the complexity and the amount of configuration that needs to be reviewed, the adoption of scanning tools to check if all the configurations are secure and comply with industry best practices is necessary. Cloud service providers may provide such scanning services or tools. Here, we recommend open source tools such as CIS-CAT Lite provided by CIS and OpenSCAP.

The journey to secure the infrastructure and platform can be completed in three stages. The first stage is to define a secure configuration baseline by referring to industry practices such as CIS or NIST NCP. Then, we may apply tools such as Chef or Puppet to ensure every deployment includes a secure configuration as well. The final stage is to do constant monitoring of frequent configuration changes and security compliance assessment.  

Typical infrastructure components are listed in the following table. CIS provides secure configuration suggestions to each system component and also tools to do the scanning against the security best practice baseline. 

CIS provides the CIS Benchmark, which defines the secure configuration of operating systems, server software, cloud services, networking devices, and so on. It helps operation teams to understand how to secure and configure an infrastructure and platform.

Infrastructure layers

System

Web services

Apache, Nginx, IIS

Database

MS SQL, MySQL, Oracle, MongoDB

Virtualization/container

VMware, Docker, Kubernetes

Networking

Cisco devices

Operating systems

Windows, Linux, Ubuntu, CentOS, SUSE

 

In addition to CIS Benchmark documents, CIS also provides tools to infrastructure or operation teams for secure configuration scanning. The CIS Security website provides related security configuration scanning tools to download. 

Source: https://www.cisecurity.org/cybersecurity-tools/

National Checklist Program (NCP) repository

The NCP repository provides secure configuration for specific software components. For example, if you are looking for Apache security configuration or the CIS of Apache, you may use the NCP to do the search.  The screenshot is from the NIST NCP (National Checklist Program).

Source: https://nvd.nist.gov/ncp/repository

OpenSCAP tools

OpenSCAP is similar to CIS security benchmarks; it also provides a secure configuration baseline. In addition, OpenSCAP also provides different kinds of tool for operation or infrastructure teams to do secure configuration evaluation and scanning. Depending on the requirements, there are four kinds of tool provided, as shown in the following screenshot:

Source: https://www.open-scap.org/tools/

 

Legal and security compliance


The EU GDPR, which came into force in May 2018, protects all EU citizens from privacy and data breaches. According to the GDPR FAQ:

"The GDPR not only applies to organizations located within the EU but it also applies to all companies processing and holding the personal data of data subjects residing in the European Union, regardless of the company's location."

In other words, if a company is providing services to customers in the European Union, its data handling will need to comply entirely with GDPR. From a DevSecOps point of view, it's related to data collection, handling, storage, backup, modification, transport, and removal—in a secure manner. According to GDPR Article 5, there are six privacy principles:

  • Lawfulness, fairness, and transparency
  • Purpose limitations
  • Data minimization
  • Accuracy
  • Storage limitations
  • Integrity and confidentiality

GDPR, like other security compliance policies, doesn't define the technical approach to achieve it. GDPR can still be too high-level for an engineering team. It needs to translate into software security requirements, design, threat modeling, tools, and so on. The following table summarizes typical security practices for the engineering team:

Stage

Common security practices for privacy or sensitive info handing

Design

Privacy Impact Assessment (PIA)

Coding

  • Data masking library
  • Anonymous toolbox
  • RAPPOR—privacy-preserving reporting algorithms
  • Encryption storage (RSA, ASE)
  • Secure erasure
  • Secure communication protocol (such as TLS v1.2, SSH v2, SFTP, SNMP v3)
  • Cookie consent
  • Data Vault
  • Key management

Testing

OWASP testing for weak cryptography, testing for error handling, testing for configuration, and so on

Deployment

  • OWASP configuration and deployment management testing
  • CIS secure environment configuration
  • Sensitive information in Git

Monitoring

  • ELK for log analysis
  • Integrity monitoring (IDS/IPS) to monitor any unauthorized changes
  • CIS secure configuration monitoring
  • Sensitive information leakage in Git
 

New technology (third-party, cloud, containers, and virtualization)


New technologies such as virtualization, Docker, and microservices introduce new methods of software delivery and speeds up application deployment, but also brings new security threats and risks. We will briefly discuss how these new technologies change the practices of security and DevOps.

Virtualization

It's very common to install application services on top of a virtualized OS. Virtualization technology helps to make the most physical machine resources such as the CPU, memory, and disks. However, virtualization is a shared OS technology. It also introduces security risks such as VM escape, information leakage, and denial-of-service for applications running on top of virtualization.

Security practices in guest OS virtualization are normally involved with both OS and virtualization hardening. Here are some key security configurations related to virtualization. Refer to CIS Benchmarks for details:

  • Limit informative messages from the VM to the VMX file
  • Limit sharing console connections
  • Disconnect unauthorized devices (USB, DVD, serial devices, and so on)
  • Disable BIOS Boot Specification (BBS)
  • Disable guest-host interaction protocol handler
  • Disable host guest filesystem server
  • Disable VM console paste operations
  • Disable virtual disk shrinking
  • Do not send host information to guests

The following diagram shows the adoption of virtualization. Virtualization adds a hypervisor layer on top of the physical server so that the virtualized guest OS can run on top of it:

In addition to the secure configuration of virtualization, applying a security patch to virtualization is also a must for operation or IT teams. 

In addition, the following resources may help you to search for Common Vulnerabilities and Exposures (CVE) in vulnerability databases:

To search for the vulnerabilities of a specific product or vendor, refer to the URL search for VMware as following:

Dockers

The introduction of Docker provides software package delivery and installation with new choices and can be one of the best ways to isolate different applications without using a whole separate OS virtual machine. Software can be packaged into a container by Docker. A container, like a VM image, includes everything needed to run application services such as runtime, system libraries, and settings. The key difference between a virtual machine image and a container is that the container doesn't actually include the whole OS. The container only includes key necessary system libraries and every container shares the same OS kernel during runtime. Therefore, Docker containers can boot up within seconds and use much less memory or far fewer disks than virtualization images.

The use of Docker can also greatly help operation teams to do deployment and secure configuration since a Docker container includes every configuration and the settings you need to run. To understand Docker security practices, check out the CIS Docker Benchmark and Docker security in the Further reading section. 

Key secure practices and configurations of Docker are listed here:

  • Separate partition for containers
  • Updated Linux kernel
  • Only allow trusted users to control the Docker daemon
  • Audit the Docker daemon, files, and directories
  • Restrict network traffic between containers
  • TLS authentication for the Docker daemon
  • Do not bind Docker to another IP/port or a Unix socket
  • Docker daemon configuration files permissions
  • Container runtime (Linux Kernel capabilities, SSH, ports, memory, CPU, IPC)

The following diagram shows the key difference between virtualization and Docker. Virtualization is a guest OS level while Docker is actually an application-level isolation and shares the same guest OS:

Here is a summary of the known security vulnerabilities identified in Docker.

CVE ID

Related CWE ID

Description

CVE-2014-5282

20

Docker before 1.3 does not properly validate image IDs, which allows remote attackers to redirect to another image through the loading of untrusted images via Docker load.

 

CVE-2017-14992

20

Lack of content verification in Docker-CE (also known as Moby), and earlier allows a remote attacker to launch a Denial of Service attack via a crafted image layer payload; a.k.a Gzip bombing.

CVE-2017-7297

264

Rancher Labs rancher server 1.2.0+ is vulnerable to authenticated users disabling access control via an API call. This is fixed in versions rancher/server:v1.2.4, rancher/server:v1.3.5, rancher/server:v1.4.3, and rancher/server:v1.5.3.

CVE-2016-9962

362

RunC allowed additional container processes via runc exec to be ptraced by the pid 1 of the container. This allows the main processes of the container, if running as root, to gain access to file-descriptors of these new processes during initialization and can lead to container escapes or modification of runC state before the process is fully placed inside the container.

CVE-2014-0047

n/a

Docker before 1.5 allows local users to have an unspecified impact via vectors involving unsafe /tmp usage.

 

Here is a tip to query a specific vulnerability.  Take 'CVE-2014-0047' as an example; just replace the CVE ID number at the end of the following URL.

Infrastructure as Code (IaC)

Puppet, Chef, Ansible, and SaltStack are tools to apply IaC. The key advantage of using these tools is that any system configuration can be defined in a text file at the design stage and changes can be managed by versions. This will help both operation or development teams to build security configuration baselines such as file permissions, firewall rules, web configurations, or MySQL connections. Once the security configuration baseline is defined, the operation team can monitor any unauthorized changes or roll back the configuration to previous specific versions.

For example, we may have baseline security firewall rules for a web services environment to only allow ports 80 and 443. All an operation team needs to do is to define the firewall rules by using one of the tools (Puppet, Chef, Ansible, SaltStack), and the framework will apply the rules, audit, and even correct changes if other ports are opened by mistake or by other service deployments.

The DevSec Hardening Framework project available at https://github.com/dev-sec provides Ansible, Chef, and Puppet secure configuration baseline template scripts.

The following diagram shows how IaC (for example, Puppet) works:

 

Cloud services hacks/abuse


A CSA survey on the top cloud security concerns has identified the following 12 issues:

  • Data breaches
  • Weak identity, credentials, and access management
  • Insecure APIs
  • System and application vulnerabilities
  • Account hijacking
  • Malicious insiders
  • Advanced Persistent Threats (APTs)
  • Data loss
  • Insufficient due diligence
  • Abuse and nefarious use of cloud services
  • Denial of service
  • Shared technology issues

In addition, service abuse has also become a headache for most e-commerce or shopping sites. Let's take one example to understand how hackers or misconduct users can benefit from it.

Case study – products on sale

Assume that one online shopping store is going to have a 50% discount on one new model phone for only the first 100 customers; it will be available at 12:00 on February 1.

What do hackers do?

For this kind of sale with 50 % profit is a great attraction for malicious users to do something. What underground users typically may do involves the massive registration of user accounts. There can be more than 10,000 users accounts registered in a short period of time just before the sales. At the moment of the sale, they will use automated scripts to trigger purchase behaviors and finish the orders within seconds. Once they have ordered or occupied all the phones, they may either sell them at higher prices or even not pay for the orders.

Is this illegal? These behaviors follow the business rules for registration and purchases. Although the behavior may not be against the law, it may be considered misconduct or service abuse. Therefore, this kind of on-sale activity may require additional business rules and regulations. After all, it's not purely hacking behavior. We will discuss this in later chapters. Here, we provide an overview of alleviating measures, which can be by means of business rules or technical approaches:

  • The sale is only limited to those customers with a certain period of purchase history
  • Apply CAPTCHA to distinguish humans from machines
  • Two-factor authentication and registration via phone SMS
  • Detection and correlation of IP, phone number, email, account ID, physical address, and GeoIP location
  • Unusual page browsing behavior such as skipping products and jumping to the purchase directly
  • Unusual massive logins or registration from the same IP or devices
 

Rapid release


Rapid, frequent, and iterative releases are very common for cloud services. This normally drives the need for DevOps practices. This can be both an opportunity and a challenge to security. The challenge is that a short period of frequent releases may not include enough time to do a full cycle of security testing. There are three maturity levels of DevOps practices:

Maturity level

Achieved

Technology adoption

Continuous integration

  •  Source code repository and version control
  • CI workflow with a daily build and unit testing
  • Jenkins
  • Git
  • Unit testing

Continuous delivery

  • Automated deploy to the staging environment
  • Integration testing on the staging environment
  • Deployment to production is done manually
  • IaC(Puppet)
  • Docker

Continuous deployment

  • Automated deployment and acceptance testing on production
  • Production changes or configuration management
  • IaC (puppet)
  • Docker
  • Automated acceptance testing
  • Configuration monitoring

 

The adoption of DevOps practices means more collaboration between development, QA, IT, and operation teams, and more in-progress adoption of continuous integration or continuous delivery tools. This provides a good foundation to move to DevSecOps. Depending on the maturity level of the existing CI/CD, security practices or tools can be added on top of the existing CI/CD framework. It's the most effective and least learning curve to introduce security is don't change existing development, QA, IT, operation team the way they work. Building security tools around the existing CI/CD is still the best approach. We will explore this more in upcoming chapters.

The diagram below shows the security involved with development, QA, and operations through the whole CI/CD lifecycle.

 

Summary


In this chapter, we discussed external factors that drive the need for security such as security compliance, regulations, and the market. In addition, the adoption of new technologies also brings about new challenges such as Docker, virtualization, cloud services, and IaC. 

For security compliance, we briefly discussed ISO 27001 and some security best practices/tools introduced by CSA  such as CCM, cloud security guide, CAIQ, and Cloud top threats. FIPS was also discussed for the correct usage of cryptography. In terms of infrastructure security, CIS and OpenSCAP were introduced. Finally, the EU GDPR law regulates and drives the security requirements of data and privacy protection. 

Based on all these security challenges and compliance rules, we introduced one small case study for cloud services, which could be hacked and abused. Moreover, what security technologies may apply to DevOps practices. In upcoming chapters, we will further discuss how security goals, metrics, and security assurance programs apply to different kinds of organization and practices. 

 

Questions


  1. Does FIPS define the security requirements for cryptography?
  2. Which of the following defines the security compliance is primarily focused on personal data privacy? 
    1. ISO 27018
    2. FIPS 
    3. GDPR 
    4. CIS
  3. What can be considered cloud service abuse?          
    1. Account sharing   
    2. Brute-force logins   
    3. API abuse
    4. All of the above
  1. What is the CIS security benchmark used for?
    1. Anti-virus     
    2. Defining secure configuration of the OS, platform, databases, and so on
    3. Firewall   
    4. Integrity
  2. Which role is involved with security practices during the DevOps cycle?      
    1. QA   
    2. RD     
    3. Operations   
    4. All of the above 
  3. How does the technology Infrastructure as Code help security operation teams?     
    1. Virus detection   
    2. Secure configuration   
    3. Intrusion detection   
    4. Encryption
  4. Which of the following is not a privacy principle?               
    1. Spoofing        
    2. Purpose limitations         
    3. Storage limitations   
    4. Accuracy
 

Further reading


Read the following links for further readings:

About the Author

  • Tony Hsu

    Tony Hsu is a senior security architect with over 20 years of experience in security services technology. He has rich experience with Secure Software Development LifeCycle (SSDLC), is deeply involved with security activities such as security requirements planning, threat modeling, secure architecture and design review, secure code review, automated security testing, and cloud services security monitoring. He is also in-house SDL trainer.

    He is also a co contributor on OWASP projects such as OWASP testing guide, proactive control guide, and deserialization security cheatsheet.

    Browse publications by this author

Latest Reviews

(1 reviews total)
Great content, updated and good price.

Recommended For You

Practical Security Automation and Testing

Your one stop guide to automating infrastructure security using DevOps and DevSecOps

By Tony Hsiang-Chih Hsu
Mastering Kali Linux for Advanced Penetration Testing - Third Edition

A practical guide to testing your infrastructure security with Kali Linux, the preferred choice of pentesters and hackers

By Vijay Kumar Velu and 1 more
DevOps with Kubernetes - Second Edition

Leverage the power of Kubernetes to build an efficient software delivery pipeline.

By Hideto Saito and 2 more
Azure Networking Cookbook

Learn to distribute network traffic, optimize application delivery, and defend network-level threats.

By Mustafa Toroman