How to automate your system administration tasks with Ansible

Do you want to sharpen your system administration or Linux skills? Perhaps you have some stuff running on your local LAN and you want to make your life easier—where do you begin? In this article, I'll explain how to set up tooling to simplify administering multiple machines.

When it comes to remote administration tools, SaltStack, Puppet, Chef, and Ansible are a few popular options. Throughout this article, I'll focus on Ansible and explain how it can be helpful whether you have 5 virtual machines or a 1,000.

Our journey begins with the basic administration of multiple machines, whether they are virtual or physical. I will assume you have an idea of what you want to achieve, and basic Linux administration skills (or at least the ability to look up the steps required to perform each task). I will show you how to use the tools, and it is up to you to decide what to do with them.

What is Ansible?

The Ansible website explains the project as "a radically simple IT automation engine that automates cloud provisioning, configuration management, application deployment, intra-service orchestration, and many other IT needs." Ansible can be used to perform the same tasks across a defined set of servers from a centralized location.

If you are familiar with Bash for-loops, you'll find that Ansible operates in a similar fashion. The difference, however, is that Ansible is idempotent. In layman's terms this means that generally Ansible only performs the requested action if a change will occur as a result. For example, if you were to perform a Bash for-loop to create a user across all of your machines, it may look something like this:

for server in serverA serverB serverC; do ssh ${server} "useradd myuser"; done

This would create myuser on serverA, serverB, and serverC; however, it would run the user add command every single time the for-loop was run, whether or not the user existed. An idempotent system will first check whether the user exists, and if it does not, the tool will create it. This is a simplified example, of course, but the benefits of an idempotent tool will become more clear over time.

How does Ansible work?

Ansible translates Ansible playbooks into commands that are run over SSH, which has several benefits when it comes to managing Unix-like environments:

1. Most, if not all of the Unix-like machines you are administering will have SSH running by default.
2. Relying on SSH means that no agent is required on the remote host.
3. In most cases no additional software needs to be installed as Ansible requires Python 2.6 in order to operate. Most, if not all distributions of Linux have this version (or greater) installed by default.
4. Ansible does not require a master node. It can be run from any host that has the Ansible package installed and sufficient SSH access.
5. Although running Ansible in a cron job is possible, by default Ansible only runs when you tell it to.

Setting up SSH key authentication

A common method for using Ansible is to set up passwordless SSH keys to facilitate ease of management. (Using Ansible Vault for passwords and other sensitive information is possible, but is outside the scope of this article.) For now, simply generate an SSH key with the following command as shown in Example 1.

Example 1: Generating An SSH Key

[09:44 user ~]$ ssh-keygen 
Generating public/private rsa key pair.
Enter file in which to save the key (/home/user/.ssh/id_rsa): 
Created directory '/home/user/.ssh'.
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /home/user/.ssh/id_rsa.
Your public key has been saved in /home/user/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:TpMyzf4qGqXmx3aqZijVv7vO9zGnVXsh6dPbXAZ+LUQ user@user-fedora
The key's randomart image is:
+---[RSA 2048]----+
|                 |
|                 |
|              E  |
|       o .   ..  |
|   .  + S    o+. |
|  . .o * .  .+ooo|
| . .+o  o o oo+.*|
|. .ooo* o. *  .*+|
| . o+*BO.o+    .o|
+----[SHA256]-----+

In Example 1, the Enter key is used to accept the defaults. An SSH key can be generated by any unprivileged user and installed in any user's SSH authorized_keys file on the remote system. After the key has been generated, it will need to be copied to a remote host. To do so, run the following command:

ssh-copy-id root@servera

Note: Ansible does not require root access; however, if you choose to use a non-root user, you must configure the appropriate sudo permissions for the tasks you want to accomplish.

You will be prompted for the root password for servera, which will allow your SSH key to be installed on the remote host. After the initial installation of the SSH key, you will no longer be prompted for the root password on the remote host when logging in over SSH.

Installing Ansible

The installation of the Ansible package is only required on the host that generated the SSH key in Example 1. If you are running Fedora, you can issue the following command:

sudo dnf install ansible -y

If you run CentOS, you need to configure Extra Packages for Enterprise Linux (EPEL) repositories:

sudo yum install epel-release -y

Then you can install Ansible with yum:

sudo yum install ansible -y

For Ubuntu-based systems, you can install Ansible from the PPA:

sudo apt-get install software-properties-common -y 
sudo apt-add-repository ppa:ansible/ansible 
sudo apt-get update 
sudo apt-get install ansible -y

If you are using macOS, the recommended installation is done via Python PIP:

sudo pip install ansible

See the Ansible installation documentation for other distributions.

Working with Ansible Inventory

Ansible uses an INI-style file called an Inventory to track which servers it may manage. By default this file is located in /etc/ansible/hosts. In this article, I will use the Ansible Inventory shown in Example 2 to perform actions against the desired hosts (which has been paired down for brevity):

Example 2: Ansible hosts file

[arch]
nextcloud
prometheus
desktop1
desktop2
vm-host15

[fedora]
netflix

[centos]
conan
confluence
7-repo
vm-server1
gitlab

[ubuntu]
trusty-mirror
nwn
kids-tv
media-centre
nas

[satellite]
satellite

[ocp]
lb00
ocp_dns
master01
app01
infra01

Each group, which is denoted via square brackets and a group name (such as [group1]), is an arbitrary group name that can be applied to a set of servers. A server can exist in multiple groups without issue. In this case, I have groups for operating systems (arch, ubuntu, centos, fedora), as well as server function (ocp, satellite). The Ansible host file can handle significantly more advanced functionality than what I am using. For more information, see the Inventory documentation.

Running ad hoc commands

After you have copied your SSH keys to all the servers in your inventory, you are ready to start using Ansible. A basic Ansible function is the ability to run ad hoc commands. The syntax is:

ansible  -a "some command"

For example, if you want to update all of the CentOS servers, you might run:

ansible centos -a 'yum update -y'

Note: Having group names based on the operating system of the server is not necessary. As I will discuss, Ansible Facts can be used to gather this information; however, issuing ad hoc commands becomes more complex when trying to use Facts, and so for convenience I recommend creating a few groups based on operating system if you manage a heterogeneous environment.

This will loop over each of the servers in the group centos and install all of the updates. A more useful ad hoc command would be the Ansible ping module, which is used to verify that a server is ready to receive commands:

ansible all -m ping

This will result in Ansible attempting to log in via SSH to all of the servers in your inventory. Truncated output for the ping command can be seen in Example 3.

Example 3: Ansible ping command output

nwn | SUCCESS => {
    "changed": false, 
    "ping": "pong"
}
media-centre | SUCCESS => {
    "changed": false, 
    "ping": "pong"
}
nas | SUCCESS => {
    "changed": false, 
    "ping": "pong"
}
kids-tv | SUCCESS => {
    "changed": false, 
    "ping": "pong"
}
...

The ability to run ad hoc commands is useful for quick tasks, but what if you want to be able to run the same tasks later, in a repeatable fashion? For that Ansible implements playbooks.

Ansible playbooks for complex tasks

An Ansible playbook is a YAML file that contains all the instructions that Ansible should complete during a run. For the purposes of this exercise, I will not get into more advanced topics such as Roles and Templates. If you are interested in learning more, the documentation is a great place to start.

In the previous section, I encouraged you to use the ssh-copy-id command to propagate your SSH keys; however, this article is focused on how to accomplish tasks in a consistent, repeatable manner. Example 4 demonstrates one method for ensuring, in an idempotent fashion, that an SSH key exists on the target hosts.

Example 4: Ansible playbook "push_ssh_keys.yaml"

---
- hosts: all
  gather_facts: false
  vars:
    ssh_key: '/root/playbooks/files/laptop_ssh_key'
  tasks:
    - name: copy ssh key
      authorized_key:
        key: "{{ lookup('file', ssh_key) }}"
        user: root

In the playbook from Example 4, all of the critical sections are highlighted.

The - hosts: line indicates which host groups the playbook should evaluate. In this particular case, it is going to examine all of the hosts from our Inventory.

The gather_facts: line instructs Ansible to attempt to find out detailed information about each host. I will examine this in more detail later. For now, gather_facts is set to false to save time.

The vars: section, as one might imagine, is used to define variables that can be used throughout the playbook. In such a short playbook as the one in Example 4, it is more a convenience rather than a necessity.

Finally the main section is indicated by tasks:. This is where most of the instructions are located. Each task should have a - name:. This is what is displayed as Ansible is carrying out a run, or playbook execution.

The authorized_key: heading is the name of the Ansible Module that the playbook is using. Information about Ansible Modules can be accessed on the command line via ansible-doc -a; however it may be more convenient to view the documentation in a web browser. The authorized_key module has plenty of great examples to get started with. To run the playbook in Example 4, simply use the ansible-playbook command:

ansible-playbook push_ssh_keys.yaml

If this is the first time adding an SSH key to the box, SSH will prompt you for a password for the root user.

Now that your servers have SSH keys propagated its time to do something a little more interesting.

Ansible and gathering facts

Ansible has the ability to gather all kinds of facts about the target system. This can consume a significant amount of time if you have a large number of hosts. In my experience, it can take 1 to 2 seconds per host, and possibly longer; however, there are benefits to fact gathering. Consider the following playbook used for turning off the ability for users to log in with a password as the root user:

Example 5: Lock down root SSH account

---
- hosts: all
  gather_facts: true
  vars:
  tasks:
    - name: Enabling ssh-key only root access
      lineinfile:
        dest: /etc/ssh/sshd_config
        regexp: '^PermitRootLogin'
        line: 'PermitRootLogin without-password'
      notify: 
        - restart_sshd
        - restart_ssh

  handlers:
    - name: restart_sshd
      service:
        name: sshd
        state: restarted
        enabled: true
      when: ansible_distribution == 'RedHat'
    - name: restart_ssh
      service:
        name: ssh
        state: restarted
        enabled: true
      when: ansible_distribution == 'Debian'

In Example 5 the sshd_config file is modified with the conditional only executes if a distribution match is found. In this case Red Hat-based distributions name their SSH service different than Debian-based, which is the purpose for the conditional statement. Although there are other ways to achieve this same effect, the example helps demonstrate Ansible facts. If you want to see all of the facts that Ansible gathers by default, you can run the setup module on your localhost:

ansible localhost -m setup |less

Any fact that is discovered by Ansible can be used to base decisions upon much the same way the vars: section that was shown in Example 4 is used. The difference is Ansible facts are considered to be built in variables, and thus do not have to be defined by the administrator.

Next steps

Now you have the tools to start investigating Ansible and creating your own playbooks. Ansible is a tool that has so much depth, complexity, and flexibility that it would be impossible to cover everything in one article. This article should be enough to pique your interest and inspire you to explore the possibilities Ansible provides. In my next article, I will discuss the Copy, systemd, service, apt, yum, virt, and user modules. We can combine these to create update and installation playbooks, and to create a basic Git server to store all of the playbooks that may get created.