Booting Linux faster

Doing Linux kernel and firmware development leads to lots of reboots and lots of wasted time.
145 readers like this.

Of all the computers I've ever owned or used, the one that booted the quickest was from the 1980s; by the time your hand moved from the power switch to the keyboard, the BASIC interpreter was ready for your commands. Modern computers take anywhere from 15 seconds for a laptop to minutes for a small home server to boot. Why is there such a difference in boot times?

A microcomputer from the 1980s that booted straight to a BASIC prompt had a very simple CPU that started fetching and executing instructions from a memory address immediately upon getting power. Since these systems had BASIC in ROM, there was no loading time—you got to the BASIC prompt really quickly. More complex systems of that same era, such as the IBM PC or Macintosh, took a significant time to boot (~30 seconds), although this was mostly due to having to read the operating system (OS) off a floppy disk. Only a handful of seconds were spent in firmware before being able to load an OS.

Modern servers typically spend minutes, rather than seconds, in firmware before getting to the point of booting an OS from disk. This is largely due to modern systems' increased complexity. No longer can a CPU just come up and start executing instructions at full speed; we've become accustomed to CPU frequency scaling, idle states that save a lot of power, and multiple CPU cores. In fact, inside modern CPUs are a surprising number of simpler CPUs that help start the main CPU cores and provide runtime services such as throttling the frequency when it gets too hot. On most CPU architectures, the code running on these cores inside your CPU is provided as opaque binary blobs.

On OpenPOWER systems, every instruction executed on every core inside the CPU is open source software. On machines with OpenBMC (such as IBM's AC922 system and Raptor's TALOS II and Blackbird systems), this extends to the code running on the Baseboard Management Controller as well. This means we can get a tremendous amount of insight into what takes so long from the time you plug in a power cable to the time a familiar login prompt is displayed.

If you're part of a team that works on the Linux kernel, you probably boot a lot of kernels. If you're part of a team that works on firmware, you're probably going to boot a lot of different firmware images, followed by an OS to ensure your firmware still works. If we can reduce the hardware's boot time, these teams can become more productive, and end users may be grateful when they're setting up systems or rebooting to install firmware or OS updates.

Over the years, many improvements have been made to Linux distributions' boot time. Modern init systems deal well with doing things concurrently and on-demand. On a modern system, once the kernel starts executing, it can take very few seconds to get to a login prompt. This handful of seconds are not the place to optimize boot time; we have to go earlier: before we get to the OS.

On OpenPOWER systems, the firmware loads an OS by booting a Linux kernel stored in the firmware flash chip that runs a userspace program called Petitboot to find the disk that holds the OS the user wants to boot and kexec() to it. This code reuse leverages the efforts that have gone into making Linux boot quicker. Even so, we found places in our kernel config and userspace where we could improve and easily shave seconds off boot time. With these optimizations, booting the Petitboot environment is a single-digit percentage of boot time, so we had to find more improvements elsewhere.

Before the Petitboot environment starts, there's a prior bit of firmware called Skiboot, and before that there's Hostboot. Prior to Hostboot is the Self-Boot Engine, a separate core on the die that gets a single CPU core up and executing instructions out of Level 3 cache. These components are where we can make the most headway in reducing boot time, as they take up the overwhelming majority of it. Perhaps some of these components aren't optimized enough or doing as much in parallel as they could be?

Another avenue of attack is reboot time rather than boot time. On a reboot, do we really need to reinitialize all the hardware?

Like any modern system, the solutions to improving boot (and reboot) time have been a mixture of doing more in parallel, dealing with legacy, and (arguably) cheating. 


Stewart Smith will present Booting faster at linux.conf.au, January 21-25 in Christchurch, New Zealand.

User profile image.
Stewart currently works for IBM in the Linux Technology Center on OPAL, the OpenPOWER Abstraction Layer – open source firmware for POWER systems. He currently finds it weird when you don’t have the source code to your firmware and doubly weird that you may not be running a daily build.

9 Comments

My box at home (core i3 3.5Ghz, 8GB RAM, rotational disk) boots from cold to login prompt in 54 seconds. That's running OpenSuse. Once I've logged it, the box fires off Firefox, Sypheed, my popup calendar and a script to tell me the backup which runs at shutdown works. 22 seconds later we're ready to go.

My work box, similar spec but running Windows 10 takes 2 1/2 minutes from cold to login. When I've logged in, it fires off Firefox and Outlook mail client and whatever group policy does. 90-100 seconds later, it's finished stuffing about (all the hourglasses have gone) and I can do work.

The lower spec'ed Windows 7 box I also have at work and use sometimes, boots quicker and logs in faster too. My employer also plonked a Mac on my desk. It's a i5 processor and boots and logs in faster than W10 but not as fast as my home Linux box.

You could speed it up by factor of 2 or more if you compress your entire OS with Squashfs, using either lzo or lz4 compression, and put an overlay mount over it. Requires manual fiddling with your initrd, which can be destroyed by automated update.

In reply to by MartyMonroe

This is something we've considered looking into. However it's currently a tiny fraction of total boot time we'd save if we halved that boot time, and the investment to do that is quite high as we'd need a way to sign, verify, and measure the squashfs image for secure and trusted boot.

So our next batch of efforts are going to be on other firmware components :)

In reply to by Szycha (not verified)

As always, an informative article, but as all too often, a bit depressing. In order even to study the boot process, it is necessary to obtain a fully-open system. As with LibreBoot, the hardware ends up being IBM hardware. Perhaps when it comes to software freedom (or the combination of software and firmware freedom?) there's some sort of "no free lunch" principle:

https://twitter.com/JonErlichman/status/1083812854854307840

It's a lot more affordable to obtain than it has been in the past though with the Raptor Blackbird system - a microATX board that can be bought with a 4 core CPU for $1200.

Perhaps other hardware vendors should follow our lead and respect freedom :)

In reply to by Lori

It's not the boot time that drives me nuts. It's the shutdown times now that all the majors insist on using systemd.

What I'd like to see is an article shutdown faster. In a simple desktop environment, 30 sec per service should be sufficient to stabilize. I've searched to no avail for ways to change the shutdown time manually when or after a service starts, or some way to truncate a shutdown job when it's running.

An article covering this is greatly needed.

It'd be great if somebody went and attacked that, I feel that too. It's not my current area of expertise though. Perhaps someone who works deep in linux distributions can write one?

In reply to by Guy Scree (not verified)

Oceanographer, Computer Science Teacher, Independent Software Developer
Systems Architect (FOSS & WinDoze) with Mixed Language Programming & XUL aka Mozilla Sources

In reply to by stewart-ibm

The shutdown should definitely be optimized

Creative Commons LicenseThis work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License.