5 reasons sysadmins love systemd

Systemd's speed and ease make it a popular way to manage modern Linux systems.
115 readers like this.
Woman sitting in front of her laptop

kris krüg

As systems administrators know, there's a lot happening on modern computers. Applications run in the background, automated events wait to be triggered at a certain time, log files are written, status reports are delivered. Traditionally, these disparate processes have been managed and monitored with a collection of Unix tools to great effect and with great efficiency. However, modern computers are diverse, with local services running alongside containerized applications, easy access to clouds and the clusters they run on, real-time processes, and more data to process than ever.

Having a unified method of managing them is an expectation for users and a useful luxury for busy sysadmins. For this nontrivial task, the system daemon, or systemd, was developed and quickly adopted by all major Linux distributions.

Of course, systemd isn't the only way to manage a Linux system. There are many alternative init systems, including sysvinit, OpenRC, runit, s6, and even BusyBox, but systemd treats Linux as a unified data set, meant to be manipulated and queried consistently with robust tools. For a busy systems administrator and many users, the speed and ease of systemd is an important feature. Here are five reasons why.

Boot management

Booting a Linux computer can be a surprisingly rare event, if you want it to be. Certainly in the server world, uptimes are often counted in years rather than months or weeks. Laptops and desktops tend to be shut down and booted pretty frequently, although even these are as likely to be suspended or hibernated as they are to be shut down. Either way, the time since the most recent boot event can serve as a sort of session manager for a computer health check. It's a useful way to limit what data you look at when monitoring your system or diagnosing problems.

In the likely event that you can't remember the last time you booted your computer, you can list boot sessions with systemd's logging tool, journalctl:

$ journalctl --list-boots
-42 7fe7c3... Fri 2020-12-04 05:13:59 - Wed 2020-12-16 16:01:23 
-41 332e99... Wed 2020-12-16 20:07:39 - Fri 2020-12-18 22:08:13 
[...]
-1 e0fe5f... Mon 2021-03-29 20:47:46 - Mon 2021-03-29 21:59:29
 0 37fbe4... Tue 2021-03-30 04:46:13 - Tue 2021-03-30 10:42:08

The latest boot sessions appear at the bottom of the list, so you can pipe the output to tail for just the latest boots.

The numbers on the left (42, 41, 1, and 0 in this example) are index numbers for each boot session. In other words, to view logs for only a specific boot session, you can use its index number as reference.

Log reviews

Looking at logs is an important method of extrapolating information about your system. Logs provide a history of much of the activity your computer engages in without your direct supervision. You can see when services launched, when timed jobs ran, what services are running in the background, which activities failed, and more. One of the most common initial troubleshooting steps is to review logs, which is easy to do with journalctl:

$ journalctl --pager-end

The --pager-end (or -e for short) option starts your view of the logs at the end of the journalctl output, so you must scroll up to see events that happened earlier.

Systemd maintains a "catalog" of errors and messages filled with records of errors, possible solutions, pointers to support forums, and developer documentation. This can provide important context to a log event, which can otherwise be a confusing blip in a sea of messages, or worse, could go entirely unnoticed. To integrate error messages with explanatory text, you can use the --catalog (or -x for short) option:

$ journalctl --pager-end --catalog

To further limit the log output you need to wade through, you can specify which boot session you want to see logs for. Because each boot session is indexed, you can specify certain sessions with the --boot option and view only the logs that apply to it:

$ journalctl --pager-end --catalog --boot 42

You can also see logs for a specific systemd unit. For instance, to troubleshoot an issue with your secure shell (SSH) service, you can specify --unit sshd to see only the logs that apply to the sshd daemon:

$ journalctl --pager-end \
--catalog --boot 42 \
--unit sshd

Service management

The first task for systemd is to boot your computer, and it generally does that promptly, efficiently, and effectively. But the task that's never finished is service management. By design, systemd ensures that the services you want to run do indeed start and continue running during your session. This is nicely robust, because in theory even a crashed service can be restarted without your intervention.

Your interface to help systemd manage services is the systemctl command. With it, you can view the unit files that define a service:

$ systemctl cat sshd
# /usr/lib/systemd/system/sshd.service
[Unit]
Description=OpenSSH server daemon
Documentation=man:sshd(8) man:sshd_config(5)
After=network.target sshd-keygen.target
Wants=sshd-keygen.target

[Service]
Type=notify
EnvironmentFile=-/etc/crypto-policies/back-ends/opensshserver.config
EnvironmentFile=-/etc/sysconfig/sshd
ExecStart=/usr/sbin/sshd -D $OPTIONS $CRYPTO_POLICY
ExecReload=/bin/kill -HUP $MAINPID
KillMode=process
Restart=on-failure
RestartSec=42s

[Install]
WantedBy=multi-user.target

Most unit files exist in /usr/lib/systemd/system/ but, as with many important configurations, you're encouraged to modify them with local changes. There's an interface for that, too:

$ systemctl edit sshd

You can see whether a service is currently active:

$ systemctl is-active sshd
active
$ systemctl is-active foo
inactive

Similarly, you can see whether a service has failed with is-failed.

Starting and stopping services is nicely intuitive:

$ systemctl stop sshd
$ systemctl start sshd

And enabling a service to start at boot time is simple:

$ systemctl enable sshd

Add the --now option to enable a service to start at boot time or to start it for your current session.

Timers

Long ago, when you wanted to automate a task on Linux, the canonical tool for the job was cron. There's still a place for the cron command, but there are also some compelling alternatives. For instance, the anacron command is a versatile, cron-like system capable of running tasks that otherwise would have been missed during downtime.

Scheduled events are little more than services activated at a specific time, so systemd manages a cron-like function called timers. You can list active timers:

$ systemctl list-timers
NEXT                          LEFT       
Tue 2021-03-30 12:37:54 NZDT  16min left [...]
Wed 2021-03-31 00:00:00 NZDT  11h left [...]
Wed 2021-03-31 06:42:02 NZDT  18h left [...]

3 timers listed.
Pass --all to see loaded but inactive timers, too.

You can enable a timer the same way you enable a service:

$ systemctl enable myMonitor.timer

Targets

Targets are the final major component of the systemd matrix. A target is defined by a unit file, the same as services and timers. Targets can also be started and enabled in the same way. What makes targets unique is that they group other unit files in an arbitrarily significant way. For instance, you might want to boot to a text console instead of a graphical desktop, so the multi-user target exists. However, the multi-user target is only the graphical target without the desktop unit files as dependencies.

In short, targets are an easy way for you to collect services, timers, and even other targets together to represent an intended state for your machine.

In fact, within systemd, a reboot, a power-off, or a shut-down action is just another target.

You can list all available targets using the list-unit-files option, constraining it with the --type option set to target:

$ systemctl list-unit-files --type target

Taking control with systemd

Modern Linux uses systemd for service management and log introspection. It provides everything from personal Linux systems to enterprise servers with a modern mechanism for monitoring and easy maintenance. The more you use it, the more systemd becomes comfortably predictable and intuitive, and the more you discover how disparate parts of your system are interconnected.

To get better acquainted with systemd, you must use it. And to get comfortable with using it, download our cheat sheet and refer to it often.

What to read next

Learning to love systemd

systemd is the mother of all processes, responsible for bringing the Linux host up to a state where productive work can be done.

Seth Kenlon
Seth Kenlon is a UNIX geek, free culture advocate, independent multimedia artist, and D&D nerd. He has worked in the film and computing industry, often at the same time.

27 Comments

Here is one sysadmin that doesn't love systemd.

I'm not really a fan of it either, but the source of this article is from Red Hat.

Red Hat crowing about their own tool? Inconceivable!

In reply to by MartyMonroe

I think you may be reading too much into this. I use systemd on my RHEL servers (not on my workstation, which runs Slackware as it has for over a decade), and I write articles to promote open source. I do this regardless of who has the most commits in a project's repo (for the record, though, I haven't done a `git blame` to actually count who has the most commits and what entity employs them).

That's all.

In reply to by Teh Billeh

I think it would be great to hear more about people's processes for other init systems ("What init system do you use, and what's the tool chain you use to manage it?"). If you're interested in telling the story of some other init system, submit an article!

In reply to by spooky

systemd is probably the worse init i have ever used. Everything they say is so great about it can easily be done with commands like grep, sed, awk and perl. That doesn't even get to the real problems with it like the binary log files, the spaghetti mess that is the boot order, the un-usable init scripts and worse of all is the feature creep.

In reply to by MartyMonroe

While this article does a good job of summarising the benefits of systemd, it simply ignores the serious issues it introduces. Had it remained as the initially planned system management toolset, I (along with others, I suspect) would have been happy to use it. Unfortunately, the deep integration has lead to it being almost impossible to extricate from tbe dependency chains, and seems to be steadily expanding its reach.
The April 1st 2015 article on Distrowatch was worryingly close to being real, and doesn't appear to be getting less realistic!

I hear what you're saying. I prefer extreme modularity, too, but standardization is useful, and init systems have not historically had that. It seems to me that systemd provides a predictable foundation upon which many other applications can build, and at a pretty low level. I've worked as a sys admin, and I've developed software for Linux; I appreciate the tools systemd provides for the former, and I prefer the uniformity that it provides for the latter.

I'm not a distro maintainer, but that most major distros have migrated to systemd speaks volumes, in my view. You can't get distros to agree on anything, and yet the Big Ones have all adopted systemd. That said, I don't run systemd on all of my systems, so there's still plenty of choice out there. I don't think that's going to change.

In reply to by David Tillotson

systemd is horrible. No matter how much lipstick you put on it, it's still a pig.

Which init system do you use? You should consider writing an article about it, and why you love it. I'm always fascinated to hear about the tools people use, and what it enables them to do. Share the knowledge!

In reply to by spooky

For those who disagree with the article's premise, there is Devuan.

https://www.devuan.org/

It's a mature fork of Debian that preserves init system choice.

Thanks for your article Seth! I don't know much about the workings of system-d even though it's on my system. I'm always eager to learn more.

Glad to hear this was elucidating, Don! It's definitely a testament to systemd that you (and many other users) use it every day but know and care very little about it. It must mean things are working the way they should, and that's a great place to be.

In reply to by Don Watkins

systemd....

- Lead developer la-la's to himself and rubs his ears frantically when somebody brings a valid issue to the table.
- No longer sees it's task as an init system but an entire ecosystem of it's own with far too much integration and applications dependent upon it.
- It's success was based around forcing it's use through big name distros that were already established with large user bases. They had no choice other than to accept the decision of their distros development team.
- In real life use finding errors or broken processes (particularly with application startup) is a nightmare. If it wasn't, the numerous Linux and dev forums wouldn't be full of systemd issues about the same thing. Seriously, look at the small number of tutorials it takes to explain basic OpenRC or SysV startup scripts, then compare with the number of threads where a user can't get a Python script starting up through systemd, maintained in service and errors easily discovered and understood.

Seth, this article is well written and detailed as most of yours are, but suggesting others write articles about the init system they use is avoiding what the commenters are trying to tell you. They don't use another init, they use systemd, because they have no choice; and you already know this to defend your article with.

Before systemd you could swap inits out however you pleased. Sure, you've got to rewrite your config scripts and startups but it wouldn't prevent you installing an application. For example, Devuan is supposed to be Debian with systemd pulled out. Try installing a selection of 10 .deb packages in it and see how many just work. And coming from the other side see how many users can get their systemd service files working first time on a self-written script or from-source package when it used to be as easy as a one-line cron-job and a bash script.

OK, you've called my bluff. I'll write the articles. Watch this space.

In reply to by Moz

Hehe, I'm looking forward to them. OpenRC first if you get chance Seth; on the other hand MX Linux seems to be getting popular these days so maybe it has to be SysV, which will also satisfy those of us still running Slackware.

In reply to by sethkenlon

Weird that you mentioned that. I literally just downloaded MX this morning, with the intent of using it as the basis for the first article. But I'm eager to revisit OpenRC, too, so who knows what I'll actually get to first!

In reply to by Moz

Well said. I have one system that I ended up *abandoning* because startup with systemd was unreliable on it, and nobody on the forum was willing to actually lend a helping hand - and this from someone who has been working with UNIX since 6th edition and has learned LOTS of new things. Systemd is a *disaster*.

In reply to by Moz

I don't overly care about which init system is running as long as they work and for the most part systemd has just worked for me, maybe I'm just lucky. As far as creating new unit files and such, yeah, there is a learning curve compared to just throwing together a bash script, but for some people they both involve the same level of arcane magic. Fortunately, I haven't had to create many from scratch and when I've gotten them squared away they seem to just work. Does systemd still have issues and is it getting into everything? Sure, but like Seth pointed out, at least the interface to it is relative consistent and some feature can simply be turned off and an alternative used. I find it more productive to adapt and learn as much about the systems I admin than whine about change (even if it's perceived as change for changes sake). Just my $.02

how is bash not consistent? the worse part of systemd is way past the init scripts. Its the binary log files. The feature creep, why would you ever want init to do logging? Or resolve hostnames.

systemd is just dumb.

In reply to by iceburg

Nice information

Many thanks, Seth, for the effort you spent to help to understand the not-so-easy-world of systemd. Instead of cursing this tool, it is wiser to try to ride it!

mv systemd some-other-reality

Creative Commons LicenseThis work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License.