Build web apps to automate sysadmin tasks

System administrators (sysadmins) waste thousands of hours each year on repetitive tasks. Fortunately, web apps, built using open source tools, can automate a significant portion of that pain away.

For example, it takes only about a day to build a web app using Python and JavaScript to reclaim some of that time. Here is the core structure that any web application must have:

• A backend to persist data
• A web server to host and route traffic
• An HTML user interface
• Interactive JavaScript code to make it more functional
• CSS layout and styling to make it pretty

The scenario: Simplify employee offboarding

Imagine you're a sysadmin at a company with a thousand employees. If the average employee leaves after three years, you have to offboard an employee every single day. That's a significant time sink!

There's a lot to do when an employee leaves: remove their user account from LDAP, revoke GitHub permissions, take them off payroll, update the org chart, redirect their email, revoke their keycard, etc.

As a sysadmin, your job is to automate your job away, so you've already written some offboarding scripts to run the IT side of this automatically. But HR still has to call you and ask you to run each of your scripts, and that's an interruption you can do without.

You decide to dedicate one full day to automating away this problem, saving you hundreds of hours in the long run. (There is another option, which I'll present at the end of this article.)

The app will be a simple portal you can give to HR. When HR enters the departing user's email address, the app runs your offboarding scripts in the background.

Its frontend is built in JavaScript, and the backend is a Python app that uses Flask. It's hosted using Nginx on an AWS EC2 instance (or it could be in your corporate network or private cloud). Let's look at each of these elements in turn, starting with the Python (Flask) app.

Start with the backend

The backend allows you to make an HTTP POST request to a particular URL, passing in the departing employee's email address. The app runs your scripts for that employee and returns success or failure for each script. It uses Flask, a Python web framework that's ideal for lightweight backends like this.

To install Flask, create a Python Virtual Environment, then use pip to install it:

~/offboarding$ virtualenv ~/venv/offboarding
~/offboarding$ source ~/venv/offboarding/bin/activate
(offboarding) ~/offboarding$ pip3 install flask
Collecting flask
  Downloading 
...

Handle a request with Flask

Create HTTP endpoints in Flask by decorating functions with @app.route(<url>, ...), and access the request data using the request variable. Here's a Flask endpoint that reads the employee's email address:

#!/usr/bin/env python3

from flask import Flask, request
app = Flask(__name__)

@app.route('/offboard', methods=['POST'])
def offboard():
    employee_email = request.json.get('employeeEmail')
    print("Running offboarding for employee {} ...".format(employee_email))
    return 'It worked!'

if __name__ == "__main__":
    app.run(threaded=True)

It responds to the HTTP request with status 200 and the text "It worked!" in the body. To check that it works, run the script; this runs the Flask development server, which is good enough for testing and light use (despite the warning).

(offboarding) ~/offboarding$ ./offboarding.py
 * Serving Flask app "offboarding" (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: off
 * Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)

Here's a curl command that makes a request:

~$ curl -X POST \
  -d '{"employeeEmail": "shaun@anvil.works"}' \
  -H "Content-Type: application/json" \
  http://localhost:5000/offboard
It worked!

The final line is the response from the server: it's working! Here's what the server prints:

Running offboarding for employee shaun@anvil.works ...
127.0.0.1 - - [05/Sep/2019 13:10:55] "POST /offboard HTTP/1.1" 200 -

It's up and running! You have an endpoint that can take in your data. Expand this to make it run the pre-existing offboarding scripts.

Run the scripts with Python

To keep things simple, put the scripts in a single folder and iterate over the folder, running anything you find. That way, you don't need to modify the code and restart the server to add new scripts to your offboarding process; you can just copy them into the folder (or create symlinks).

Here's how the Flask app looks when it's been modified to do that (the comments in the code point out some best practices):

#!/usr/bin/env python3

from flask import Flask, request
import subprocess
from pathlib import Path
import os

app = Flask(__name__)

# Set the (relative) path to the scripts directory 
# so we can easily use a different one.
SCRIPTS_DIR = 'scripts'


@app.route('/offboard', methods=['POST'])
def offboard():
    employee_email = request.json.get('employeeEmail')
    print("Running offboarding for employee {} ...".format(employee_email))

    statuses = {}

    for script in os.listdir(SCRIPTS_DIR):
        # The pathlib.Path object is a really elegant way to construct paths 
        # in a way that works cross-platform (IMO!)
        path = Path(SCRIPTS_DIR) / script
        print('  Running {}'.format(path))

        # This is where we call out to the script and store the exit code.
        statuses[script] = subprocess.run([str(path), employee_email]).returncode

    return statuses


if __name__ == "__main__":
    # Running the Flask server in threaded mode allows multiple 
    # users to connect at once. For a consumer-facing app,
    # we would not use the Flask development server, but we expect low traffic!
    app.run(threaded=True)

Put a few executables in the scripts/ directory. Here are some shell commands to do that:

mkdir -p scripts/
cat > scripts/remove_from_ldap.py <<EOF
#!/usr/bin/env python3
print('Removing user from LDAP...')
EOF
cat > scripts/revoke_github_permisisons.py <<EOF
#!/usr/bin/env python3
import sys
sys.exit(1)
EOF
cat > scripts/update_org_chart.sh <<EOF
#!/bin/sh
echo "Updating org chart for $1..."
EOF

chmod +x scripts/*

Now, restart the server, and run the curl request again. The response is a JSON object showing the exit codes of the scripts. It looks like revoke_github_permissions.py failed on this run:

~$ curl -X POST \
  -d '{"employeeEmail": "shaun@anvil.works"}' \
  -H "Content-Type: application/json" \
  http://localhost:5000/offboard
{"remove_from_ldap.py":0,"revoke_github_permissions.py":1,"update_org_chart.sh":0}

Here's the server output; this time it informs us when each script starts running:

Running offboarding for employee shaun@anvil.works ...
  Running scripts/remove_from_ldap.py
  Running scripts/revoke_github_permissions.py
  Running scripts/update_org_chart.sh
127.0.0.1 - - [05/Sep/2019 13:30:55] "POST /offboard HTTP/1.1" 200 -

Now you can run your scripts remotely by making an HTTP request.

Add authentication and access control

So far, the app doesn't do any access control, which means anyone can trigger offboarding for any user. It's easy to see how this could be abused, so you need to add some access control.

In an ideal world, you would authenticate all users against your corporate identity system. But authenticating a Flask app against, for example, Office 365, would take a much longer tutorial. So use "HTTP Basic" username-and-password authentication.

First, install the Flask-HTTPAuth library:

(offboarding) ~/offboarding$ pip3 install Flask-HTTPAuth
Collecting Flask-HTTPAuth
  Downloading …

Now require a username and password to submit the form by adding this code to the top of offboarding.py:

from flask_httpauth import HTTPBasicAuth
from werkzeug.security import generate_password_hash, check_password_hash

app = Flask(__name__)
auth = HTTPBasicAuth()

users = {
    "hr": generate_password_hash("secretpassword"),
}

@auth.verify_password
def verify_password(username, password):
    if username in users:
        return check_password_hash(users.get(username), password)
    return False

@app.route('/offboard', methods=['POST'])
@auth.login_required
def offboard():
  # ... as before …

Specify a username and password for the request to succeed:

~$ curl -X POST \
  -d '{"employeeEmail": "shaun@anvil.works"}' \
  -H "Content-Type: application/json" \
  http://localhost:5000/offboard
Unauthorized Access

ubuntu@ip-172-31-17-9:~$ curl -X POST -u hr:secretpassowrd \
  -d '{"employeeEmail": "shaun@anvil.works"}' \
  -H "Content-Type: application/json" \
  http://localhost:5000/offboard
{"remove_from_ldap.py":0,"revoke_github_permisisons.py":1,"update_org_chart.sh":0}

If the HR department were happy using curl, you'd pretty much be done. But they probably don't speak code, so put a frontend on it. To do this, you must set up a web server.

Set up a web server

You need a web server to present static content to the user. "Static content," refers to the code and data that ends up being used by the user's web browser—this includes HTML, JavaScript, and CSS as well as icons and images.

Unless you want to leave your workstation on all day and carefully avoid tugging the power cable out with your feet, you should host your app on your company's network, private cloud, or another secure remote machine. This example will use an AWS EC2 cloud server.

Install Nginx on your remote machine following the installation instructions:

sudo apt-get update
sudo apt-get install nginx

It's already serving anything put into /var/www/html, so you can just drop your static content into there.

Configure Nginx to talk to Flask

Configure it to be aware of the Flask app. Nginx lets you configure rules about how to host content when the URL matches a certain path. Write a rule that matches the exact path /offboard and forwards the request to Flask:

# Inside the default server {} block in /etc/nginx/sites-enabled/default...
        location = /offboard {
                proxy_pass http://127.0.0.1:5000;
        }

Now restart Nginx.

Imagine your EC2 instance is at 3.8.49.253. When you go to http://3.8.49.253 in your browser, you see the "Welcome to Nginx!" page, and if you make a curl request against http://3.8.49.253/offboard, you get the same results as before. Your app is now online!

There are a couple more things left to do:

• Buy a domain and set up a DNS record (http://3.8.49.253/offboard is not pretty!).
• Set up SSL so that the traffic is encrypted. If you're doing this online, Let's Encrypt is a great free service.

You can figure out these steps on your own; how they work depends strongly on your networking configuration.

Write the frontend to trigger your scripts

It's time to write the frontend HR will use to access the app and start the scripts.

HTML for an input box and button

The frontend will display a text box that HR can use to enter the departing user's email address and a button to submit it to the Flask app. Here's the HTML for that:

<body>
  <input type="email" id="email-box" placeholder="Enter employee email" />
  <input type="button" id="send-button" onclick="makeRequest()" value="Run" />
  <div id="status"></div>
</body>

The empty <div> stores the result of the latest run.

Save that to /var/www/html/offboarding/index.html and navigate to http://3.8.49.253/offboarding. Here's what you get:

It's not very pretty—yet—but it's structurally correct.

JavaScript and jQuery for making the request

See onclick="makeRequest()" in the HTML for the button? It needs a function called makeRequest for the button to call when it's clicked. This function sends the data to the backend and processes the response.

To write it, first add a <script> tag to your HTML file to import jQuery, a really useful JavaScript library that will extract the email address from your page and send the request:

<head>
  <script src="https://code.jquery.com/jquery-3.4.1.min.js"></script>
</head>
...

To make an HTTP POST request using jQuery:

var makeRequest = function makeRequest() {
  // Make an asynchronous request to the back-end
  $.ajax({
    type: "POST",
    url: "/offboard",
    data: JSON.stringify({"employeeEmail": $('#email-box')[0].value}),
    contentType: "application/json"
  })
}

This request is made asynchronously, meaning the user can still interact with the app while they're waiting for a response. The $.ajax returns a promise, which runs the function you pass to its .done() method if the request is successful, and it runs the function you pass to its .fail() method if the request fails. Each of these methods returns a promise, so you can chain them like:

$.ajax(...).done(do_x).fail(do_y)

The backend returns the scripts' exit codes when the request is successful, so write a function to display the exit code against each script name in a table:

function(data) {
  // The process has finished, we can display the statuses.
  var scriptStatuses = data;
  $('#status').html(
    '<table style="width: 100%;" id="status-table"></table>'
  );
  for (script in scriptStatuses) {
    $('#status-table').append(
      '<tr><td>' + script + '</td><td>' + scriptStatuses[script] + '</td></tr>'
    );
  }
}

The $('#status').html() gets the HTML Document Object Model (DOM) element with ID status and replaces the HTML with the string you pass in.

On failure, trigger an alert with the HTTP status code and response body so the HR staff can quote it to alert you if the app breaks in production. The full script looks like this:

var makeRequest = function makeRequest() {
  // Make an asynchronous request to the back-end
  var jqxhr = $.ajax({
    type: "POST",
    url: "/offboard",
    data: JSON.stringify({"employeeEmail": $('#email-box')[0].value}),
    contentType: "application/json"
  }).done(function(data) {
    // The process has finished, we can display the statuses.
    console.log(data);
    var scriptStatuses = data;
    $('#status').html(
      '<table style="width: 100%;" id="status-table"></table>'
    );
    for (script in scriptStatuses) {
      $('#status-table').append('<tr><td>' + script + '</td><td>' + scriptStatuses[script] + '</td></tr>');
    }
  })
  .fail(function(data, textStatus) {
    alert( "error: " + data['statusText']+ " " + data['responseText']);
  })
}

Save this script as /var/www/html/offboarding/js/offboarding.js and include it in your HTML file:

<head>
  <script src="https://code.jquery.com/jquery-3.4.1.min.js"></script>
  <script src="https://opensource.com/js/offboarding.js"></script>
</head>
...

Now when you enter an employee email address and hit Run, the scripts will run and provide their exit codes in the table:

It's still ugly, though! It's time to fix that.

Make it look good

Bootstrap is a good way to style your app in a neutral way. Bootstrap is a CSS library (and more) that offers a grid system to make CSS-based layouts really easy. It gives your app a super clean look and feel, too.

Layout and styling with Bootstrap

Restructure your HTML so things will end up in the right places in Bootstrap's row and column structure: columns go inside rows, which go inside containers. Elements are designated as columns, rows, and containers using the col, row, and container CSS classes, and the card class gives the row a border that makes it look self-contained.

The input boxes are put inside a <form> and the text box gets a <label>. Here's the final HTML for the frontend:

<head>
  <link href="https://stackpath.bootstrapcdn.com/bootstrap/4.3.1/css/bootstrap.min.css" rel="stylesheet" />
  <link href="https://stackpath.bootstrapcdn.com/font-awesome/4.7.0/css/font-awesome.min.css" rel="stylesheet" />
  <script src="https://code.jquery.com/jquery-3.4.1.min.js"></script>
  <script src="https://stackpath.bootstrapcdn.com/bootstrap/4.3.1/js/bootstrap.min.js"></script>
  <script src="https://opensource.com/js/offboarding.js"></script>
</head>

<body>
  <div class="container" style="padding-top: 40px">
    <div class="row card" style="padding: 20px 0">
      <div id="email-input" class="col">
        <form>
          <div class="form-group">
            <label for="email-box">Employee Email</label>
            <input type="email" class="form-control" id="email-box" placeholder="Enter employee email" />
          </div>
            <input type="button" class="btn btn-primary" id="send-button" onclick="makeRequest()" value="Run" />
        </form>
        <div id="status"></div>
      </div>
    </div>
  </div>
</body>

Here's how the app looks now—it's a huge improvement.

Add status icons

One more thing: the app reports the status with 0 for success and 1 for failure, which often confuses people who aren't familiar with Unix. It would be easier for most people to understand if it used something like a checkmark icon for success and an "X" icon for failure.

Use the FontAwesome library to get checkmark and X icons. Just link to the library from the HTML <head>, just as you did with Bootstrap. Then modify the loop in the JavaScript to check the exit status and display a green check if the status is 0 and a red X if the status is anything else:

    for (script in scriptStatuses) {
      var fa_icon = scriptStatuses[script] ? 'fa-times' : 'fa-check';
      var icon_color = scriptStatuses[script] ? 'red' : 'green';
      $('#status-table').append(
        '<tr><td>' + script + '</td><td><i class="fa ' + fa_icon + '" style="color: ' + icon_color + '"></i></td></tr>'
      );
    }

Test it out. Enter an email address, hit Run, and…

Beautiful! It works!

Another option

What a productive day! You built an app that automates an important part of your job. The only downside is you have to maintain a cloud instance, frontend JavaScript, and backend Python code.

But what if you don't have all day to spend automating things or don't want to maintain them forever? A sysadmin has to keep all the many plates spinning, deal with urgent requests, and fight an ever-growing backlog of top-priority tickets. But you might be able to sneak 30 minutes of process improvement on a Friday afternoon. What can you achieve in that time?

If this were the mid-'90s, you could build something in Visual Basic in 30 minutes. But you're trying to build a web app, not a desktop app. Luckily, there is help at hand: You can use Anvil, a service built on open source software to write your app in nothing but Python — this time in 30 minutes:

Full disclosure: Anvil is a commercial service - although everything we do in this article you can do for free!  You can find a step-by-step guide to building this project on the Anvil blog.

No matter which path you take going forward—doing it on your own or using a tool like Anvil, we hope you continue to automate all the things. What kind of processes are you automating? Leave a comment to inspire your fellow sysadmins.