Security Automation with Ansible 2

5 (4 reviews total)
By Madhu Akula , Akash Mahajan
  • Instant online access to over 7,500+ books and videos
  • Constantly updated with 100+ new titles each month
  • Breadth and depth in over 1,000+ technologies
  1. Introduction to Ansible Playbooks and Roles

About this book

Security automation is one of the most interesting skills to have nowadays. Ansible allows you to write automation procedures once and use them across your entire infrastructure. This book will teach you the best way to use Ansible for seemingly complex tasks by using the various building blocks available and creating solutions that are easy to teach others, store for later, perform version control on, and repeat.

We’ll start by covering various popular modules and writing simple playbooks to showcase those modules. You’ll see how this can be applied over a variety of platforms and operating systems, whether they are Windows/Linux bare metal servers or containers on a cloud platform. Once the bare bones automation is in place, you’ll learn how to leverage tools such as Ansible Tower or even Jenkins to create scheduled repeatable processes around security patching, security hardening, compliance reports, monitoring of systems, and so on.

Moving on, you’ll delve into useful security automation techniques and approaches, and learn how to extend Ansible for enhanced security. While on the way, we will tackle topics like how to manage secrets, how to manage all the playbooks that we will create and how to enable collaboration using Ansible Galaxy. In the final stretch, we’ll tackle how to extend the modules of Ansible for our use, and do all the previous tasks in a programmatic manner to get even more powerful automation frameworks and rigs.

Publication date:
December 2017
Publisher
Packt
Pages
324
ISBN
9781788394512

 

Chapter 1. Introduction to Ansible Playbooks and Roles

According to Wikipedia, Ansible is an open source automation engine that automates software provisioning, configuration management, and application deployment. But you already knew that. This book is about taking the idea of IT automation software and applying it to the domain of Information Security Automation. 

The book will take you through the journey of security automation to show how Ansible is used in the real world. 

In this book, we will be automating security-related tasks in a structured, modular fashion using a simple human-readable format YAML. Most importantly, what you will learn to create will be repeatable. This means once it is done, you can focus on fine-tuning, expanding the scope, and so on. The tool ensures that we can build and tear down anything, from simple application stacks to simple, but extensive, multi-application frameworks working together. 

If you have been playing around with Ansible, and in this book we assume you have, you would have definitely come across some of the following terms:

  • Playbook
  • Ansible Modules 
  • YAML
  • Roles
  • Templates (Jinja2)

Don't worry, we will address all of the aforementioned terms in this chapter. Once you are comfortable with these topics, we will move on to covering scheduler tools, and then to building security automation playbooks. 

 

Ansible terms to keep in mind 


Like all new subjects or topics, it is a good idea to get familiar with the terminology of that subject or topic. We will go through some of the Ansible terms that we will be using throughout the book, and if at any point you are not able to follow, you might want to come back to this chapter and refresh your understanding for that particular term. 

Playbooks

A playbook, in the classic sense, is about offensive and defensive plays in football. The players keep a record of the plays (plan of action) in a book, usually in the form of a diagram.

In Ansible, a playbook is a series of ordered steps or instructions for an IT process. Think of a nicely-written instruction manual that can be read and understood by humans and computers alike. 

In the subsequent chapters, all the automation we will focus on regarding security will lead us toward building both simple and complex playbooks. 

This is what an Ansible playbook command looks like:

ansible-playbook -i inventory playbook.yml

Ignore the -i flag for now and notice the extension of the playbook file. 

As stated in http://docs.ansible.com/ansible/playbooks_intro.html:

"Playbooks are expressed in YAML format (see YAML syntax (http://docs.ansible.com/ansible/YAMLSyntax.html)) and have a minimum of syntax, which intentionally tries to not be a programming language or script, but rather a model of a configuration or a process."

Ansible modules

Ansible ships with a number of modules (called the module library) that can be executed directly on remote hosts or through playbooks.Tasks in playbooks call modules to do the work. 

Ansible has many modules, most of which are community contributed and maintained. Core modules are maintained by the Ansible core engineering team and will always ship with Ansible itself.

Note

Users can also write their own modules. These modules can control system resources, like services, packages, or files (anything really), or handle executing system commands. Here is the list of modules available by Ansible: http://docs.ansible.com/ansible/latest/modules_by_category.html#module-index. If you use Dash (https://kapeli.com/dash) or Zeal (https://zealdocs.org/), you can download the offline version for easy reference.

Modules can be executed via the command line as well. We will be using modules to write all the tasks inside our playbooks. All modules technically return JSON format data. 

Note

Modules should be idempotent and should avoid making any changes if they detect that the current state matches the desired final state. When using Ansible playbooks, these modules can trigger change events in the form of notifying handlers to run additional tasks.

Documentation for each module can be accessed from the command line with the ansible-doc tool:

$ ansible-doc apt

We can list all the modules available on our host:

$ ansible-doc -l

Start the Apache web server on all nodes grouped under webservers by executing the httpd module. Note the use of the -m flag:

$ ansible webservers -m service -a "name=httpd state=started"

This snippet shows the exact same command but inside a playbook in YAML syntax:

-name:restart webserverservice:name:httpdstate:started

Each module contains multiple parameters and options, get to know more about the features of the modules by looking at their documentation and examples.

YAML syntax for writing Ansible playbooks

Ansible playbooks are written in YAML, which stands for YAML Ain't Markup Language. 

According to the official document (http://yaml.org/spec/current.html):

YAML Ain’t Markup Language

 (abbreviated YAML) is a data serialization language designed to be human-friendly and work well with modern programming languages for everyday tasks.

Ansible uses YAML because it is easier for humans to read and write than other common data formats, such as XML or JSON. All YAML files (regardless of their association with Ansible or not) can optionally begin with --- and end with .... This is part of the YAML format and indicates the start and end of a document.

Note

YAML files should end with .yaml or .yml. YAML is case sensitive.You can also use linters, such as www.yamllint.com, or your text editor plugins for linting YAML syntax, which help you to troubleshoot any syntax errors and so on. 

Here is an example of a simple playbook to showcase YAML syntax from Ansible documentation (http://docs.ansible.com/ansible/playbooks_intro.html#playbook-language-example):

-hosts:webserversvars:http_port:80max_clients:200remote_user:root
tasks:-name: Ensure apache is at the latest versionyum:
      name: httpd
      state: latest-name: Write the apache config filetemplate:
      src: /srv/httpd.j2
      dest: /etc/httpd.conf
notify:-restart apache
-name: Ensure apache is running (and enable it at boot)service:
      name: httpd
      state: started
      enabled: yes
handlers:-name: Restart apacheservice:
        name: httpd
        state: restarted

Ansible roles

While playbooks offer a great way to execute plays in a pre-defined order, there is a brilliant feature on Ansible that takes the whole idea to a completely different level. Roles are a convenient way to bundle tasks, supporting assets such as files and templates, coupled with an automatic set of search paths.

By using a concept most programmers would be familiar with, of including files and folders and ascribing what is being included, a playbook becomes infinitely more readable and understandable. Roles are basically made up of tasks, handlers, and configurations, but by adding an additional layer to how a playbook is structured, we can easily get the big picture overview as well as the low-level details. 

This allows for reusable code and a division of work in a team tasked with writing playbooks. For example, the database guru writes a role (almost like a partial playbook) for setting up the database and the security guru writes one on hardening such a database.

Note

While it is possible to write a playbook in one very large file, eventually you want to reuse files and start to organize things. Large and complex playbooks are hard to maintain and it is very difficult to reuse sections of a large playbook. Breaking a playbook into roles allows very efficient code reuse and makes playbooks much easier to understand.

The benefits of using roles while building large playbooks include:

  • Collaborating on writing playbooks
  • Reusing existing roles
  • Roles can be updated, improved upon independently
  • Handling variables, templates, and files is easier

Note

LAMP usually stands for Linux, Apache, MySQL, PHP. A popular combination of software that is used to build applications for the web. Nowadays, another common combination in the PHP world is LEMP, which is Linux, NGINX, MySQL, PHP.

This is an example of what a possible LAMP stack site.yml can look like:

- name: LAMP stack setup on ubuntu 16.04
  hosts: all
  gather_facts: False
  remote_user: "{{remote_username}}"
  become: yes

 roles:
   - common
   - web
   - db
   - php

Note the list of roles. Just by reading the role names we can get an idea of the kind of tasks possibly under that role. 

Templates with Jinja2

Ansible uses Jinja2 templating to enable dynamic expressions and access to variables. Jinja2 variables and expressions within playbooks and tasks allow us to create roles that are very flexible. By passing variables to a role written this way, we can have the same role perform different tasks or configurations. Using a templating language, such as Jinja2, we are able to write playbooks that are succinct and easier to read.

By ensuring that all the templating takes place on the Ansible controller, Jinja2 is not required on the target machine. Only the required data is copied over, which reduces the data that needs to be transferred. As we know, less data transfer usually results in faster execution and feedback. 

Jinja templating examples

A mark of a good templating language is the ability to allow control of the content without appearing to be a fully-fledged programming language. Jinja2 excels in that by providing us with the ability to do conditional output, such as iterations using loops, among other things. 

Let's look at some basic examples (obviously Ansible playbook-related) to see what that looks like. 

Conditional example

Execute only when the operating system family is Debian:

tasks:-name:"shutdownDebianflavoredsystems"command:/sbin/shutdown -t nowwhen:ansible_os_family == "Debian"
Loops example

The following task adds users using the Jinja2 templating. This allows for dynamic functionality in playbooks. We can use variables to store data when required, we just need to update the variables rather than the entire playbook:

-name:add several usersuser:name:"{{item.name}}"state:presentgroups:"{{item.groups}}"with_items:-{name:'testuser1',groups:'wheel'}-{name:'testuser2',groups:'root'}

LAMP stack playbook example – combining all the concepts

We will look at how to write a LAMP stack playbook using the skills we have learned so far. Here is the high-level hierarchy structure of the entire playbook:

inventory               # inventory file
group_vars/             #
   all.yml              # variables
site.yml                # master playbook (contains list of roles)
roles/                  #
    common/             # common role
        tasks/          #
            main.yml    # installing basic tasks
    web/                # apache2 role
        tasks/          #
            main.yml    # install apache
        templates/      #
            web.conf.j2 # apache2 custom configuration
        vars/           # 
            main.yml    # variables for web role 
        handlers/       #
            main.yml    # start apache2
    php/                # php role
        tasks/          # 
            main.yml    # installing php and restart apache2
    db/                 # db role
        tasks/          #
            main.yml    # install mysql and include harden.yml
            harden.yml  # security hardening for mysql
        handlers/       #
            main.yml    # start db and restart apache2
        vars/           #
            main.yml    # variables for db role

Let's start with creating an inventory file. The following inventory file is created using static manual entry. Here is a very basic static inventory file where we will define a since host and set the IP address used to connect to it.

Configure the following inventory file as required:

[lamp]
lampstack    ansible_host=192.168.56.10

The following file is group_vars/lamp.yml, which has the configuration of all the global variables:

remote_username: "hodor"

The following file is the site.yml, which is the main playbook file to start:

- name: LAMP stack setup on Ubuntu 16.04
 hosts: lamp
 gather_facts: False
 remote_user: "{{ remote_username }}"
 become: True

 roles:
   - common
   - web
   - db
   - php

The following is the roles/common/tasks/main.yml file, which will install python2, curl, and git:

# In ubuntu 16.04 by default there is no python2
- name: install python 2
  raw: test -e /usr/bin/python || (apt -y update && apt install -y python-minimal)

- name: install curl and git
  apt:
    name: "{{ item }}"
    state: present
    update_cache: yes

  with_items:
    - curl
    - git

The following task, roles/web/tasks/main.yml, performs multiple operations, such as installation and configuration of apache2. It also adds the service to the startup process:

- name: install apache2 server
  apt:
    name: apache2
    state: present

- name: update the apache2 server configuration
  template: 
    src: web.conf.j2
    dest: /etc/apache2/sites-available/000-default.conf
    owner: root
    group: root
    mode: 0644

- name: enable apache2 on startup
  systemd:
    name: apache2
    enabled: yes
  notify:
    - start apache2

The notify parameter will trigger the handlers found in roles/web/handlers/main.yml:

- name: start apache2
  systemd:
    state: started
    name: apache2

- name: stop apache2
  systemd:
    state: stopped
    name: apache2

- name: restart apache2
  systemd:
    state: restarted
    name: apache2
    daemon_reload: yes

The template files will be taken from role/web/templates/web.conf.j2, which uses Jinja templating, it also takes values from local variables:

<VirtualHost *:80><VirtualHost *:80>
    ServerAdmin {{server_admin_email}}
    DocumentRoot {{server_document_root}}

    ErrorLog ${APACHE_LOG_DIR}/error.log
    CustomLog ${APACHE_LOG_DIR}/access.log combined
</VirtualHost>

The local variables file is located in roles/web/vars/main.yml:

server_admin_email: [email protected]
server_document_root: /var/www/html

Similarly, we will write database roles as well. The following file roles/db/tasks/main.yml includes installation of the database server with assigned passwords when prompted. At the end of the file, we included harden.yml, which executes another set of tasks:

- name: set mysql root password
  debconf:
    name: mysql-server
    question: mysql-server/root_password
    value: "{{ mysql_root_password | quote }}"
    vtype: password

- name: confirm mysql root password
  debconf: 
    name: mysql-server
    question: mysql-server/root_password_again
    value: "{{ mysql_root_password | quote }}"
    vtype: password

- name: install mysqlserver
  apt:
    name: "{{ item }}"
    state: present 
  with_items:
    - mysql-server
    - mysql-client

- include: harden.yml

The harden.yml performs hardening of MySQL server configuration:

- name: deletes anonymous mysql user
  mysql_user:
    user: ""
    state: absent
    login_password: "{{ mysql_root_password }}"
    login_user: root

- name: secures the mysql root user
  mysql_user: 
    user: root
    password: "{{ mysql_root_password }}"
    host: "{{ item }}"
    login_password: "{{mysql_root_password}}"
    login_user: root
 with_items:
   - 127.0.0.1
   - localhost
   - ::1
   - "{{ ansible_fqdn }}"

- name: removes the mysql test database
  mysql_db:
    db: test
    state: absent
    login_password: "{{ mysql_root_password }}"
    login_user: root

- name: enable mysql on startup
  systemd:
    name: mysql
    enabled: yes

  notify:
    - start mysql

The db server role also has roles/db/handlers/main.yml and local variables similar to the web role:

- name: start mysql
  systemd:
    state: started
    name: mysql

- name: stop mysql
  systemd:
    state: stopped
    name: mysql

- name: restart mysql
  systemd:
    state: restarted
    name: mysql
    daemon_reload: yes

The following file is roles/db/vars/main.yml, which has the mysql_root_password while configuring the server. We will see how we can secure these plaintext passwords using ansible-vault in future chapters:

mysql_root_password: R4nd0mP4$$w0rd

Now, we will install PHP and configure it to work with apache2 by restarting the roles/php/tasks/main.yml service:

- name: install php7
  apt:
    name: "{{ item }}"
    state: present
  with_items:
    - php7.0-mysql
    - php7.0-curl
    - php7.0-json
    - php7.0-cgi
    - php7.0
    - libapache2-mod-php7

- name: restart apache2
  systemd:
    state: restarted
    name: apache2
    daemon_reload: yes

To run this playbook, we need to have Ansible installed in the system path. Please refer to http://docs.ansible.com/ansible/intro_installation.html for installation instructions. 

Then execute the following command against the Ubuntu 16.04 server to set up LAMP stack. Provide the password when it prompts for system access for user hodor:

$ ansible-playbook -i inventory site.yml

After successful completion of the playbook execution, we will be ready to use LAMP stack in a Ubuntu 16.04 machine. You might have observed that each task or role is configurable as we need throughout the playbook. Roles give the power to generalize the playbook and customize easily using variables and templating.

 

Summary


We have codified a fairly decent real-world stack for development using a combination of Ansible's features. By thinking about what goes in a LAMP stack overview, we can start by creating the roles. Once we have that thrashed out, the individual tasks are mapped to modules in Ansible. Any task that requires copying of a pre-defined configuration, but with dynamically-generated output, can be done by using variables in our templates and the constructs offered by Jinja2. 

We will use the same approach to various security-related setups that could do with a bit of automation for orchestration, operations, and so on. Once we have a handle on how to do this for a virtual machine running our laptop, it can be repurposed for deploying on your favorite cloud-computing instance as well. The output is human readable and in text, so that it can be added to version control, various roles can be reused as well.  

Now that we have a fairly decent idea of the terms we will be using throughout this book, let's get set for one final piece of the puzzle. In the next chapter, we will learn and understand how we can use automation and scheduling tools, such as Ansible Tower, Jenkins, and Rundeck, to manage and execute playbooks based on certain event triggers or time durations. 

 

 

About the Authors

  • Madhu Akula

    Madhu Akula is a security ninja and security and devops researcher with extensive experience in the industry, ranging from client-facing assignments building scalable and secure infrastructure, to publishing industry-leading research to running training sessions for companies and governments alike.

    Madhu’s research papers are frequently selected for major security industry conferences including Defcon 24, All Day DevOps (2016, 2017), DevSecCon (London, Singapore, Boston), DevOpsDays India, c0c0n, Serverless Summit ToorCon, DefCamp, SkydogCon, NolaCon, and null, and more. Madhu was a keynote speaker for the National Cyber Security conference at Dayananda Sagar College in February 2016.

    When he’s not working with Appsecco’s clients or speaking at events, he’s actively involved in researching vulnerabilities in open source products/platforms such as WordPress, Ntop, and Opendocman. He’s also a contributing bug hunter with Code Vigilant (a project to secure open source software).

    His research has identified many vulnerabilities in over 200 organizations including the U.S. Department of Homeland Security, Google, Microsoft, Yahoo, Adobe, LinkedIn, eBay, AT&T, Blackberry, Cisco, Barracuda, and more. He is also an active member of Bugcrowd, Hackerone, Synack, and more. Madhu has trained over 5000 people in information security for companies and organizations including the Indian Navy and the Ministry of e-services in a leading Gulf state.

    Browse publications by this author
  • Akash Mahajan

    Akash Mahajan is an accomplished security professional with over a decade’s experience in providing specialist application and infrastructure consulting services at the highest levels to companies, governments, and organizations around the world. He has lots of experience in working with clients to provide innovative security insights that truly reflect the commercial and operational needs of the organization, from strategic advice to testing and analysis to incident response and recovery.

    He is an active participant in the international security community and a conference speaker both individually, as chapter lead of the Bangalore chapter of OWASP the global organization responsible for defining the standards for web application security and as a co-founder of NULL India’s largest open security community.

    Akash runs Appsecco, a company focused on Application Security. He authored the book Burp Suite Essentials published by Packt Publishing in November 2014, which is listed as a reference by the creators of Burp Suite.

    Browse publications by this author

Latest Reviews

(4 reviews total)
Geniales Angebotsspecial viel Wissen für wenig Geld
It has the info I wanted.
great book. looking forward for more books from author

Recommended For You

Mastering Ansible - Third Edition

Design, develop, and solve real-world automation and orchestration problems by unlocking the automation capabilities of Ansible.

By James Freeman and 1 more
Learning Ansible 2.7 - Third Edition

Use Ansible to configure your systems, deploy software, and orchestrate advanced IT tasks

By Fabio Alessandro Locati
Learn Ansible

Run Ansible playbooks to launch complex multi-tier applications hosted in public clouds

By Russ McKendrick
Deep Learning with TensorFlow 2 and Keras - Second Edition

Build machine and deep learning systems with the newly released TensorFlow 2 and Keras for the lab, production, and mobile devices

By Antonio Gulli and 2 more