TBRD

Introduction

Organizing and maintaining your virtual machines can become a real mess, especially when you're working on larger projects. As a dev-op, you have to use your time smart and be as effective as possible. That's why Ansible was born and has become so popular.

Ansible is a very powerful tool when it comes to automation and it helps you a lot in organizing your infrastructure. Especially if you use Terraform to start or shut down machines on demand or use an autoscaling configuration.

So you never know how many computers are currently running and what IP addresses they have in your network. So when it comes to maintaining them, you need a solution that does all the hard work for you.

Prerequisites

Let's say we need to manage a simple web application with the following setup using custom VMs in Google Cloud:

- MySQL        > 1
- Readis       > 1
- App Admin    > 1
- App Frontend > 1-5

This is how our application setup could look like

The application instances in particular are configured using auto-scaling and can therefore range in number from one to five. As a Dev-Ops, we want to create an Ansible configuration that organizes and executes automated tasks. This way, our infrastructure can be maintained at any time and the required tasks can be executed on the VMs.

For this example, let's just assume that the project is up and running in GCP and we already have a GCP service account with sufficient permissions to access the resources. We have also created and downloaded a service account key that we will use to authenticate Ansible to GCP.

We have also prepared a set of Ansible playbooks to fulfill all our tasks. Now we just need to create our inventory configuration to distribute the right Ansible tasks to the right VMs.

Authentication for your GCP project

Before we can execute at least one Ansible task on our VMs, we have to connect to GCP and authenticate ourselves.

In our working directory, in which we will execute all Ansible commands, we will create a new directory, which we will call inventory. In it, we need to create a new file inventory.gcp.yaml and add the following content

plugin: gcp_compute
scopes:
  - https://www.googleapis.com/auth/compute
projects:
  - ${GCP_PROJECT}
auth_kind: serviceaccount
service_account_file: ${PATH_TO_KEY}

./inventory/inventory.gcp.yaml

This is basically the minimum configuration required to connect to GCP with Ansible. In most cases, your Ansible installation already has the gcp_compute plugin installed. Otherwise you can find more information here.

Simply replace GCP_PROJECT with the ID of your project and change PATH_TO_KEY with the path to the key file of your service account.

Execute the following command to see if the configuration works

ansible-inventory -i inventory --list

If everything went well, you will see a lot of output about all VMs with their details running in your GCP project, and at the very end you can see the VMs with their IPs.

"ungrouped": {
    "hosts": [
        ...
    ]
}

Define Ansible Groups using Network Tags

Now that we can connect to our GCP project, we need to organize our VMs into groups. Ansible uses a hierarchical way of organizing resources. This means that we always have an implicit root group called all and below that we start to organize our VMs into subgroups and their subgroups.

In our example, we could split our VMs into the following groups and subgroups:

- db
- redis
- app
|- admin
|- fronten

This is how our VMs could be organized into groups

This is exactly the hierarchy we need to have in our Ansible inventory. Let's now add more content to our inventory.gcp.yaml file.

...
groups:
  db: "'db' in tags['items']" 
  redis: "'redis' in tags['items']"
  app: "'app' in tags['items']"
  app_admin: "'app-admin' in tags['items']"
  app_frontend: "'app-frontend' in tags['items']"

./inventory/inventory.gcp.yaml

For this configuration to work, we need to add all tags to our VMs as so-called Network Tags, which can be added when creating or editing VMs. Note that each of our application instances must have the two tags app and app_admin or app_frontend. This ensures that all our application instances are added to the app group, but only the admin VM is added to the app_admin group, and so on.

We can now check our inventory setup by executing the following command again

ansible-inventory -i inventory --list

This time we get almost the same output and our newly defined groups are displayed.

...
"all": {
  "children": [
    "app",
    "db",
    "redis",
  ]
},
"app": {
  "children": [
    "app_admin",
    "app_frontend"
  ],
  "hosts": [
    ...
  ]
},
"app_admin": {
  "hosts": [
    ...
  ]
}
...

Now we are ready to go and can use our playbooks to address the desired groups.

Define Ansible Groups using Labels

In some cases, the use of network tags is not desirable, as they actively assign firewall rules to the VMs and can therefore lead to unexpected results. In this case, there is another way to organize your VMs into groups using labels.

Labels are like environment variables and can have a key and a value. This is pretty handy when it comes to structuring and organizing your infrastructure. I personally don't use them often, but I will show you how to use them.

Now let's add some labels to our VMs. We will define a service label containing the name of the service and for the application instances we will also add an additional component label with the value admin or frontend.

- db
  > service: db
- redis
  > service: redis
- app-admin
  > service: app
  > component: admin
- app-frontend
  > service: app
  > component: frontend

This is how we could organize our labels in our VMs

Now we need to add a few more definitions to our inventory.gcp.yaml file so that Ansible knows how to use the labels.

...
keyed_groups:
  - prefix: gcp
    key: labels

./inventory/inventory.gcp.yaml

The keyed_groups configuration can be used to generate a variety of Ansible groups that use the key and value concatenation of a dictionary, in our case it is our label dictionary. We can now check out our inventory by running the following command again

ansible-inventory -i inventory --list

In this case, we still have all the outputs as before, but in between we find new groups that are automatically generated by Ansible.

...
"gcp_service_db": {
  "hosts": [
    ...
  ]
},
"gcp_service_redis": {
  "hosts": [
    ...
  ]
},
"gcp_service_app": {
  "hosts": [
    ...
  ]
},
"gcp_component_admin": {
  "hosts": [
    ...
  ]
},
"gcp_component_frontend": {
  "hosts": [
    ...
  ]
},
...

Now Ansible has combined all the key and value pairs and created a variety of groups with the prefix we defined. Now we can use these groups and add them to our Ansible playbooks so that the right tasks are executed on the right machines.

Separate your VMs using Filters

In a real-world example, your VMs will most likely be separated by an environment such as development or production. In this case, your dev and prod VMs can all have the same labels and tags, so running your Ansible playbooks will affect all of them.

Let's say we only want to manage our production VMs. In this case, we need to add an additional label environment to all our production VMs and update our inventory file by adding filters so Ansible knows which VMs we want to target specifically.

Let us now add further configurations to our inventory.gcp.yaml file

filters:
  - labels.environment = prod

./inventory/inventory.gcp.yaml

If we now check our inventory, only VMs that have the environment: prod label are listed in the output.

Conclusion

As we have just seen, there are many ways to organize your infrastructure with Ansible. By using them all, we can make our playbooks very specific and ensure that each individual task is only executed on the VMs to which it belongs.

Simplify Ansible Inventory for Dynamic GCP VM Management

Introduction

Prerequisites

Authentication for your GCP project

Define Ansible Groups using Network Tags

Define Ansible Groups using Labels

Separate your VMs using Filters

Conclusion