Monitor systemd journals via email

Image by:

Nguyen Nguyen , via Pexels CC0.

Modern Linux systems often use systemd as their init system and manager for jobs and many other functions. Services managed by systemd generally send their output (of all forms: warnings, errors, informational messages, and more) to the systemd journal, not to traditional logging systems like syslog.

In addition to services, Linux systems often have many scheduled jobs (traditionally called cron jobs, even if the system doesn't use cron to run them), and these jobs may either send their output to the logging system or allow the job scheduler to capture the output and deliver it via email.

When managing multiple systems, you can install and configure a centralized log-capture system to monitor their behavior, but the complexity of centralized systems can make them hard to manage.

A simpler solution is to have each system directly send "interesting" output to the administrator(s) by email. For systems using systemd, this can be done using Tim Waugh's journal-brief tool. This tool almost served my needs when I discovered it recently, so, in typical open source fashion, I contributed various patches to add email support to the project. Tim worked with me to get them merged, and now I can use the tool to monitor the 20-plus systems I manage as simply as possible.

Now, early each morning, I receive between 20 and 23 email messages: most of them contain a filtered view of each machine's entire systemd journal (with warnings or more serious messages), but a few are logs generated by scheduled ZFS snapshot-replication jobs that I use for backups. In this article, I'll show you how to set up similar messages.

Install journal-brief

Although journal-brief is available in many Linux package repositories, the packaged versions will not include email support because that was just added recently. That means you'll need to install it from PyPI; I'll show you how to manually install it into a Python virtual environment to avoid interfering with other parts of the installed system. If you have a favorite tool for doing this, feel free to use it.

Choose a location for the virtual environment; in this article, I'll use /opt/journal-brief for simplicity.

Nearly all the commands in this tutorial must be executed with root permissions or the equivalent (noted by the # prompt). However, it is possible to install the software in a user-owned directory, grant that user permission to read from the journal, and install the necessary units as systemd user units, but that is not covered in this article.

Execute the following to create the virtual environment and install journal-brief and its dependencies:

$ python3 -m venv /opt/journal-brief
$ source /opt/journal-brief/bin/activate
$ pip install ‘journal-brief>=1.1.7’
$ deactivate

In order, these commands will:

Create /opt/journal-brief and set up a Python 3.x virtual environment there
Activate the virtual environment so that subsequent Python commands will use it
Install journal-brief; note that the single-quotes are necessary to keep the shell from interpreting the > character as a redirection
Deactivate the virtual environment, returning the shell back to the original Python installation

Also, create some directories to store journal-brief configuration and state files with:

$ mkdir /etc/journal-brief
$ mkdir /var/lib/journal-brief

Configure email requirements

While configuring email clients and servers is outside the scope of this article, for journal-brief to deliver email, you will need to have one of the two supported mechanisms configured and operational.

Option 1: The `mail` command

Many systems have a mail command that can be used to send (and read) email. If such a command is installed on your system, you can verify that it is configured properly by executing a command like:

$ echo "Message body" | mail --subject="Test message" {your email address here}

If the message arrives in your mailbox, you're ready to proceed using this type of mail delivery in journal-brief. If not, you can either troubleshoot and correct the configuration or use SMTP delivery.

To control the generated email messages' attributes (e.g., From address, To address, Subject) with the mail command method, you must use the command-line options in your system's mailer program: journal-brief will only construct a message's body and pipe it to the mailer.

Option 2: SMTP delivery

If you have an SMTP server available that can accept email and forward it to your mailbox, journal-brief can communicate directly with it. In addition to plain SMTP, journal-brief supports Transport Layer Security (TLS) connections and authentication, which means it can be used with many hosted email services (like Fastmail, Gmail, Pobox, and others). You will need to obtain a few pieces of information to configure this delivery mode:

SMTP server hostname
Port number to be used for message submission (it defaults to port 25, but port 587 is commonly used)
TLS support (optional or required)
Authentication information (username and password/token, if required)

When using this delivery mode, journal-brief will construct the entire message before submitting it to the SMTP server, so the From address, To address, and Subject will be supplied in journal-brief's configuration.

Set up configuration and cursor files

Journal-brief uses YAML-formatted configuration files; it uses one file per desired combination of filtering parameters, delivery options, and output formats. For this article, these files are stored in /etc/journal-brief, but you can store them in any location you like.

In addition to the configuration files, journal-brief creates and manages cursor files, which allow it to keep track of the last message in its output. Using one cursor file for each configuration file ensures that no journal messages will be lost, in contrast to a time-based log-delivery system, which might miss messages if a scheduled delivery job can't run to completion. For this article, the cursor files will be stored in /var/lib/journal-brief (you can store the cursor files in any location you like, but make sure not to store them in any type of temporary filesystem, or they'll be lost).

Finally, journal-brief has extensive filtering and formatting capabilities; I'll describe only the most basic options, and you can learn more about its capabilities in the documentation for journal-brief and systemd.journal-fields.

Configure a daily email with interesting journal entries

This example will set up a daily email to a system administrator named Robin at robin@domain.invalid from a server named storage. Robin's mail provider offers SMTP message submission through port 587 on a server named mail.server.invalid but does not require authentication or TLS. The email will be sent from storage-server@domain.invalid, so Robin can easily filter the incoming messages or generate alerts from them.

Robin has the good fortune to live in Fiji, where the workday starts rather late (around 10:00am), so there's plenty of time every morning to read emails of interesting journal entries. This example will gather the entries and deliver them at 8:30am in the local time zone (Pacific/Fiji).

Step 1: Configure journal-brief

Create a text file at /etc/journal-brief/daily-journal-email.yml with these contents:

cursor-file: '/var/lib/journal-brief/daily-journal-email'
output:
  - 'short'
  - ‘systemd’
inclusions:
  - PRIORITY: 'warning'
email:
  suppress_empty: false
  smtp:
    to: '”Robin” <robin@domain.invalid>'
    from: '"Storage Server" <storage-server@domain.invalid>'
    subject: 'daily journal'
    host: 'mail.server.invalid'
    port: 587

This configuration causes journal-brief to:

Store the cursor at the path configured as cursor-file
Format journal entries using the short format (one line per entry) and provide a list of any systemd units that are in the failed state
Include journal entries from any service unit (even the Linux kernel) with a priority of warning, error, or emergency
Send an email even if there are no matching journal entries, so Robin can be sure that the storage server is still operating and has connectivity
Send the email using SMTP

You can test this configuration file by executing a journal-brief command:

$ journal-brief --conf /etc/journal-brief/daily-journal-email

Journal-brief will scan the systemd journal for all new messages (yes, all of the messages it has never seen before), identify any that match the priority filter, and format them into an email that it sends to Robin. If the storage server has been operational for months (or years) and the systemd journal has never been purged, this could produce a very large email message. In addition to Robin not appreciating such a large message, Robin's email provider may not be willing to accept it, so you can generate a shorter message by executing this command:

$ journal-brief -b --conf /etc/journal-brief/daily-journal-email

Adding the -b argument tells journal-brief to inspect only the systemd journal entries from the most recent system boot and ignore any that are older.

After journal-brief sends the email to the SMTP server, it writes a string into the cursor file so that the next time it runs using the same cursor file, it will know where to start in the journal. If the process fails for any reason (e.g., journal entry gathering, entry formatting, or SMTP delivery), the cursor file will not be updated, which means the next time it uses the cursor file, the entries that would have been in the failed email will be included in the next email instead.

Step 2: Set up the systemd service unit

Create a text file at /etc/systemd/system/daily-journal-email.service with:

[Unit]
Description=Send daily journal report

[Service]
ExecStart=/opt/journal-brief/bin/journal-brief --conf /etc/journal-brief/%N.yml
Type=oneshot

This service unit will run journal-brief and specify a configuration file with the same name as the unit file with the suffix removed, which is what %N supplies. Since this service will be started by a timer (see step 3), there is no need to enable or manually start it.

Step 3: Set up the systemd timer unit

Create a text file at /etc/systemd/system/daily-journal-email.timer with:

[Unit]
Description=Trigger daily journal email report

[Timer]
OnCalendar=*-*-* 08:30:00 Pacific/Fiji

[Install]
WantedBy=multi-user.target

This timer will start the daily-journal-email service unit (because its name matches the timer name) every day at 8:30am in the Pacific/Fiji time zone. If the time zone was not specified, the timer would trigger the service at 8:30am in the system time zone configured on the storage server.

To make this timer start every time the system boots, it is WantedBy by the multi-user target. To enable and start the timer:

$ systemctl enable daily-journal-email.timer
$ systemctl start daily-journal-email.timer
$ systemctl list-timers daily-journal-email.timer

The last command will display the timer's status, and the NEXT column will indicate the next time the timer will start the service.

To learn more about systemd timers and building schedules for them, read Use systemd timers instead of cronjobs.

Now the configuration is complete, and Robin will receive a daily email of interesting journal entries.

Monitor the output of a specific service

The storage server has some filesystems on solid-state storage devices (SSD) and runs Fedora Linux. Fedora has an fstrim service that is scheduled to run once per week (using a systemd timer, as in the example above). Robin would like to see the output generated by this service, even if it doesn't generate any warnings or errors. While this output will be included in the daily journal email, it will be intermingled with other journal entries, and Robin would prefer to have the output in its own email message.

Step 1: Configure journal-brief

Create a text file at /etc/journal-brief/fstrim.yml with:

cursor-file: '/var/lib/journal-brief/fstrim'
output: 'short'
inclusions:
  - _SYSTEMD_UNIT:
    - ‘fstrim.service’
email:
  suppress_empty: false
  smtp:
    to: '”Robin” <robin@domain.invalid>'
    from: '"Storage Server" <storage-server@domain.invalid>'
    subject: 'weekly fstrim'
    host: 'mail.server.invalid'
    port: 587

This configuration is similar to the previous example, except that it will include all entries related to a systemd unit named fstrim.service, regardless of their priority levels, and will include only entries related to that service.

Step 2: Modify the systemd service unit

Unlike in the previous example, you don't need to create a systemd service unit or timer, since they already exist. Instead, you want to add behavior to the existing service unit by using the systemd "drop-in file" mechanism (to avoid modifying the system-provided unit file).

First, ensure that the EDITOR environment variable is set to your preferred text editor (otherwise you'll get the default editor on your system), and execute:

$ systemctl edit fstrim.service

Note that this does not edit the existing service unit file; instead, it opens an editor session to create a drop-in file (located at /etc/systemd/system/fstrim.service.d/override.conf).

Paste these contents into the editor and save the file:

[Service]
ExecStopPost=/opt/journal-brief/bin/journal-brief --conf /etc/journal-brief/%N.yml

After you exit the editor, the systemd configuration will reload automatically (which is one benefit of using systemctl edit instead of creating the file directly). Like in the previous example, this drop-in uses %N to avoid duplicating the service name; this means that the drop-in contents can be applied to any service on the system, as long as the appropriate configuration file is created in /etc/journal-brief.

Using ExecStopPost will make journal-brief run after any attempt to run the fstrim.service, whether or not it's successful. This is quite useful, as the email will be generated even if the fstrim.service cannot be started (for example, if the fstrim command is missing or not executable).

Please note that this technique is primarily applicable to systemd services that run to completion before exiting (in other words, not background or daemon processes). If the Type in the Service section of the service's unit file is forking, then journal-brief will not execute until the specified service has stopped (either manually or by a system target change, like shutdown).

The configuration is complete; Robin will receive an email after every attempt to start the fstrim service; if the attempt is successful, then the email will include the output generated by the service.

Monitor without extra effort

With this setup, you can monitor the health of your Linux systems that use systemd without needing to set up any centralized monitoring or logging tools. I find this monitoring method quite effective, as it draws my attention to unusual events on the servers I maintain without requiring any additional effort.

Special thanks to Tim Waugh for creating the journal-brief tool and being willing to accept a rather large patch to add direct email support rather than running journal-brief through cron.