Ansible

This section aims to be a guide of good practices defined by the ArCo architecture for those projects that need Ansible to configure the servers after being provisioned with Terragrunt/Terraform. The purpose of applying these practices is to have a common base for all projects inside the organization so the learning curve is smaller for those changing between different projects using this tool.

We have used to best practices Ansible page as a reference for most of what we are applying.

Structure

ArCo keeps almost the same resource organization recommended by Ansible. But we add a new level to group the inventory and playbooks configuration.

Any project using Ansible should have the following structure:

inventory/
    production                # inventory file for production servers
    staging                   # inventory file for staging environment

    group_vars/
        group1.yml             # here we assign variables to particular groups
        group2.yml
        group3/                # when we have sensible variables, we create a folder for particular groups
            vars.yml           # here we add every no sensible variables and the sensible variables do reference to vaults file
            vaults.yml         # here we add sensible variables with prefix vault_
    host_vars/
        hostname1.yml          # here we assign variables to particular systems
        hostname2.yml

library/                  # if any custom modules, put them here (optional)
module_utils/             # if any custom module_utils to support modules, put them here (optional)
filter_plugins/           # if any custom filter plugins, put them here (optional)

playbooks/
    site.yml                  # master playbook
    webservers.yml            # playbook for webserver tier
    dbservers.yml             # playbook for dbserver tier

    roles/
        common/               # this hierarchy represents a "role"
            tasks/            #
                main.yml      #  <-- tasks file can include smaller files if warranted
            handlers/         #
                main.yml      #  <-- handlers file
            templates/        #  <-- files for use with the template resource
                ntp.conf.j2   #  <------- templates end in .j2
            files/            #
                bar.txt       #  <-- files for use with the copy resource
                foo.sh        #  <-- script files for use with the script resource
            vars/             #
                main.yml      #  <-- variables associated with this role
            defaults/         #
                main.yml      #  <-- default lower priority variables for this role
            meta/             #
                main.yml      #  <-- role dependencies
            library/          # roles can also include custom modules
            module_utils/     # roles can also include custom module_utils
            lookup_plugins/   # or other types of plugins, like lookup in this case

        webtier/              # same kind of structure as "common" was above, done for the webtier role
        monitoring/           # ""
        fooapp/               # ""

Variables naming

In Ansible we have two different types of variables: globals and locals to the role. Each of them has a different naming convention. Local variables always have a prefix that allows to identify the role that variable belongs to. That prefix can be the role name itself or an acronym that identifies that role with no mistakes. On the other hand, global variables don’t have that kind of prefix.

Thanks to this we can take a big Ansible project and identify easily what variables are used for multiple roles and which are local to one single role.

# playbooks/nginx/vars/main.yml
nginx_path: "{{ apps_path }}/nginx"

# inventory/group_vars/all.yml
apps_path: /opt/apps

# inventory/group_vars/nginx.yml
nginx_path: /var/lib/nginx

In this example we can see how the rol ngins uses the global variable app_path. But in our particular case we want to use a different path and override the specific variable of the module. So we need to redefine the variable nginx_path with the new path.

Sensitive information

Sometimes we use files and variables that hold sensitive information like passwords or private keys. In these cases it is mandatory to encrypt their content before the code is submitted to the repository. This way if anyone gets access to the repository or its information won’t be able to read it if they don’t have privileges to do it.

Ansible provides a tool called Ansible Vault to encrypt that information in our Ansible scripts. The default algorithm it uses is AES and the encrypted output looks like this:

the_secret: !vault |
      $ANSIBLE_VAULT;1.1;AES256
      62313365396662343061393464336163383764373764613633653634306231386433626436623361
      6134333665353966363534333632666535333761666131620a663537646436643839616531643561
      63396265333966386166373632626539326166353965363262633030333630313338646335303630
      3438626666666137650a353638643435666633633964366338633066623234616432373231333331
      6564

To organize the sensitive variables in any project, the ArCo recommendation is to split the variables in two files: vaults.yml and vars.yml. The first defines the variable that really contains the sensitive information. These variables' names have to start with the prefix "vault_", so we can easily spot them. The later defines the variables we are actually using: both non sensitive variables and sensitive variables that reference the variables in vaults.yml.

inventory/
    group_vars/
        group3/
            vars.yml
            vaults.yml

# inventory/group_vars/group3/vars.yml
mysql_password: "{{ vault_mysql_password }}"

# inventory/group_vars/group3/vaults.yml
vault_mysql_password: !vault |
      $ANSIBLE_VAULT;1.1;AES256:dev
      62313365396662343061393464336163383764373764613633653634306231386433626436623361
      6134333665353966363534333632666535333761666131620a663537646436643839616531643561
      63396265333966386166373632626539326166353965363262633030333630313338646335303630
      3438626666666137650a353638643435666633633964366338633066623234616432373231333331
      6564

Encrypt a property

For encrypting variables you have to execute the next command:

ansible-vault encrypt_string --vault-id @prompt 'value-to-encrypt' --name 'variable_name'

ansible-vault command asks the vault key when @prompt is used. The result is the encrypt variable. You should modify the uncrypted variable with that value.

  variable_name: !vault |
          $ANSIBLE_VAULT;1.1;AES256
          39343766306430653934316363306262303439313535666338313436613437393063306166333965
          3932303865623065653237363864363065336639653463360a613430383532363161363137386234
          39356239363862663566626334336666623161616662653162353064323863623664633431326361
          6365613762373735660a653937333135333836366234386561313337333064373663633161323832
          64396364623963323934393366323838303738643134373334336264646231373538
  Encryption successful

If you want to use a vault file to encrypt the variable, you could to use the next command:

ansible-vault encrypt_string --vault-id 'vault_file_path' 'value-to-encrypt' --name 'variable_name'

It´s possible to set multiple vault passwords to encrypt different files or variables. The value for --vault-id can specify the type of vault id (propmt, a file path, etc..) and a label for the vault-id ('sublive', 'prelive', 'live', etc..). This feature can be used starting from 2.4 Ansible´s version.

For example, for the vault-id 'sublive'`:

ansible-vault encrypt_string `--vault-id` sublive@prompt 'value-to-encrypt' --name 'variable_name'

Encrypting a file

For encrypting files you have to execute the next command:

ansible-vault create --vault-id dev@prompt foo.yml

Vaults

When working with sensitive data, it will be different in each of the environments we have, e.g. sublive, prelive, live. Just like the actual values differ in all of them, the same should happen with the master password used to encrypt/decrypt the sensitive properties. That way if a password is compromised the other environments will remain safe.

When we encrypt a property or file using Ansible Vault, we can specify the name of the vault to which the resource will be added by using the --vault-id attribute. For example if we want to encrypt a property for the live environment we should execute this command:

ansible-vault encrypt_string `--vault-id` live@prompt 'value-to-encrypt' --name 'variable_name'

Once the property is encrypted we can find in the output the vault used in the encryption.

  variable_name: !vault |
          $ANSIBLE_VAULT;1.1;AES256:live  #<-- Last parameter shows the vault-id
          39343766306430653934316363306262303439313535666338313436613437393063306166333965
          3932303865623065653237363864363065336639653463360a613430383532363161363137386234
          39356239363862663566626334336666623161616662653162353064323863623664633431326361
          6365613762373735660a653937333135333836366234386561313337333064373663633161323832
          64396364623963323934393366323838303738643134373334336264646231373538
  Encryption successful

If no vault is specified, Ansible will use a default vault.

Full Parent Role and Child Role

Whenever a new role is created it should have all needed tasks for what it was created. Let’s say we want to a web server that means to create users, routes, install and configure the server, install the service and restart the server. All those tasks should be defined in our role as independent tasks and the main.yml should call all of them in the proper order. We shouldn’t create independent roles for each of those tasks.

playbooks/
    setup_nginx.yml                   # setup webserver playbook

    roles/
        nginx/
            tasks/
                users.yml             # task which create user and groups for nginx
                directories.yml       # task which create directories to install nginx
                install_nignx.yml     # task which install nginx
                install_service.yml   # task which install the service to start, stop, restart nginx
                restart_service.yml   # task which restart nginx
                main.yml
            handlers/
                main.yml
            templates/
                nginx.conf.j2
            vars/
                main.yml
            defaults/
                main.yml

# playbooks/roles/nginx/tasks/main.yml

- name: Create users and groups
  include: users.yml

- name: Change owner of config, log and data directories
  include: directories.yml

- name: Setup configuration
  include: install_nignx.yml

- name: Install service
  include: install_service.yml

- name: Restart service
  include: restart_service.yml

In case we need to invoke only a particular task of the role we have to create a child role that invokes the parent role file. E.g. if we need a role that only restart the server we must do as follows:

playbooks/
    roles/
        restart_nginx/
            tasks/
                main.yml

# playbooks/roles/restart_nginx/task/main.yml

- include_roles:
    name: "nginx"
    tasks: "restart_service"

This way variable, files and dependencies are in the parent role and if we need to modify something we only have to do it in a single place.

Access using the AWS Bastion

When a project is deployed in the cloud the way to access to the machines and configure them is a but different. Firstly we need to access a machine called bastion. From there we can access the desired machine as long as we have enough privileges for that. To allow this behaviour we have to add two files to our project: ansible.cfg and ssh.config.

ansible.cfg                 # file where you configure the SSH client
ssh.config                  # file where you have the configuration of how to connect to bastion and ansible machines
inventory/
playbooks/
    site.yml
    roles/
        webtier/
        monitoring/
        fooapp/

# ansible.cfg
[ssh_connection]
ssh_args = -o ControlPersist=15m -F ssh.config -q

# ssh.config

Host bastion
    User                   ec2-user
    HostName               100.99.88.77
    ProxyCommand           none
    IdentityFile           ~/.ssh/private_key
    BatchMode              yes
    PasswordAuthentication no
    StrictHostKeyChecking  no

Host *
    ServerAliveInterval    60
    TCPKeepAlive           yes
    ProxyCommand           ssh -i ~/.ssh/private_key -W %h:%p ec2-user@100.99.88.77
    ControlMaster          auto
    ControlPath            ~/.ssh/mux-%r@%h:%p
    ControlPersist         8h
    User                   ec2-user
    IdentityFile           ~/.ssh/private_key
    StrictHostKeyChecking  no

Besides that we also need to add to the folder ` ~/.ssh` the file with the name we specified in the IdentityFile property with the private key to access the machines of that environment.

Usage

The first step to use Ansible is to install it in our machine following this guide.

Before executing any Ansible playbook we have to configure our local environment. For that we will add the private key needed to access the machines we are about to configure to the path ~/.ssh.

If our playbook isn’t using Ansible Vault to manage sensitive information we can execute it with this command:

ansible-playbook -i inventory/  playbooks/setup_cicd.yml

On the contrary, if we have properties encrypted with Ansible Vault we have to execute this other command:

ansible-playbook --vault-id vault-example@prompt -i inventory/  playbooks/setup_cicd.yml
Vault password (vault-example):

In this case Ansible will ask for the master password of the vault and will execute the playbook.