Version control for your Linux /etc/ directory

507 readers like this.
Penguins on beach
Image by:

Original photo by Rikki Endsley. CC BY-SA 4.0

One of the many advantages of Linux, UNIX, and similar operating systems is that everything is a file and that most of your configuration is done via text files, allowing you to easily read and write to them with any tool you choose. To monitor your configuration, you have many tools for automating configurations, providing you with detailed control over how your systems are configured. You can easily compare different versions of a configuration file to see any changes. Tracking specific configurations and changes over time allows you to:

More Linux resources

  • See what changed recently when getting a wake-up page
  • Revert changes when package management stomps all over your finely crafted snowflake of a configuration
  • Restore a file when an inadvertent rm hits production rather than the test server you thought you were on
  • See what has changed when multiple people (and tools) work on a system
  • Investigate changes after a script kiddie gets in

For decades, we've been tracking changes in text files with version control systems (VCS). We have the tools and we use them all the time. Version control for your source code gives you features such as tracking file history, diffing versions, reverting to an old version, and branching.

Your /etc/ directory is where most Linux distributions store your system's configuration files, and tracking /etc/ has some extra challenges beyond what a version control system tracks. Etckeeper helps us with those extra requirements. Etckeeper prefers git, but also works with several other version control systems: mercurial, bazaar, or darcs.

Other tools such as configuration management and the distribution's package management also control configurations, but they don't track all the files in version control.

Configuration management systems (CMS), such as Puppet, Chef, and Ansible, do not track every file in /etc/. For many files, they only verify that certain contents are in place and ignore the rest of the file. For instance, Puppet will verify a user exists in /etc/passwd, but will ignore changes to accounts it's not managing.

Package files have base configuration files and maybe also scripts for automagical configuration, but they don't track modifications. Indeed, they might stomp over local configuration during a package upgrade. They shouldn't, but sometimes they do.

Etckeeper will track everything except the repo directory (for instance, /etc/.git) if it is not configured to ignore some files. The default configuration does ignore some files that don't need tracking, such as ephemeral files like /etc/mtab and cache files like /etc/ld.so.cache. Etckeeper uses the version control system's native ignore file and that file is tracked in the version control repo. For instance, /etc/.gitignore is pre-populated with the files that etckeeper will ignore, but /etc/.gitignore is tracked by git.

Some services are a little persnickety about configuration file ownership and permissions and even empty directories, all things not traditionally tracked well by a version control system. Etckeeper augments the underlying version control system by tracking this important metadata in /etc/.etckeeper. Once again, this file is tracked in the version control system.

Etckeeper also has hooks to work with the package management system and check in changes after package installs and upgrades.

Getting started with etckeeper

Once installed, use init to initialize the repo and then check in the current state.

$ sudo etckeeper init
$ sudo etckeeper commit -m "Initial checkin"

Once initialized, /etc/ is a repo in your version control system. Native version control tools can be used to manage files.

$ sudo touch /etc/testfile
$ sudo git -C /etc add testfile
$ sudo git -C /etc commit -m "A test file"
$ echo "10.10.10.1111 nextcloud" | sudo tee -a /etc/hosts >> /dev/null
$ sudo git -C /etc/ diff -U0 hosts
$ diff --git a/hosts b/hosts
index 97f1792..ecc187a 100644
--- a/hosts
+++ b/hosts
@@ -9,0 +10 @@ ff02::2 ip6-allrouters
+10.10.10.1111  nextcloud
$ sudo git -C /etc/ commit hosts -m "Added nextcloud box"
$ sudo git -C /etc/ revert HEAD -m "oops, wrong IP"
$ echo "10.10.10.111    nextcloud" | sudo tee -a /etc/hosts >> /dev/null
$ sudo git -C /etc/ commit hosts -m "Correctly added nextcloud box"

Etckeeper also provides a version control system wrapper. One advantage of using the wrapper is that sudo etckeeper commit will log the account using sudo rather than root.

$ sudo etckeeper commit
$ sudo etckeeper vcs status
$ sudo etckeeper vcs -C apache2 status

Want to see what configuration changes were made by an upgrade? Review your version control system logs.

commit 327f09703be712aab263b9565fede5c21f378732
Author: der.hans <etckeeper@example.com>
Date:   Wed Feb 15 07:24:24 2017 -0700

    committing changes in /etc after apt run
    
    Package changes:
    +vim-addon-manager 0.5.3
    +software-properties-kde 0.92.25debian1
    +vim-fugitive 2.1-2

Etckeeper is a great tool for capturing changes to system configuration over time. It complements native version control system capabilities with wrappers to track important metadata, such as permissions and ownership and work with package management systems. See the etckeeper documentation for further usage examples such as how to clone an /etc/ repo from one system to another.

Tags
Linux
SCaLE
der.hans
Avatar
der.hans is a technology and entrepreneurial veteran. As a volunteer der.hans endeavors to help build community through user group and conference leadership. He is co-chair of Open Source Career Day (OSCD) at Southern California Linux Expo (SCaLE) and chair of the Finance and Partnership committees for SeaGL.
More about me

7 Comments

Avatar
AB | March 1, 2017
No readers like this yet.

Nice article; another product that helps watch for changes is something like Tripwire.

It may be worthwhile to note that a lot of the benefit here is already managed by systems using BtrFS for the root filesystem. Seeing changes, rolling forward/back those changes (plus along with other directory structures where other important files, like the kernel, system libraries, etc., all live, is all magically handled with BtrFS snapshots. Integrate that with the system package manager and you have a powerful solution. This has been the norm for a year or two now with openSUSE and SUSE Linux Enterprise products. We've used them in our organization and these features really give us a lot of flexibility, both by tracking /etc changes and also by tracking the rest of the important directories.

Avatar
der.hans | March 2, 2017
No readers like this yet.

Yeah, tripwire will watch everything if you want. Last I used it was many years ago, does it have an option for keeping revisions and being able to view previous content?

I will have to think about snapshotting like with BtrFS. Maybe I can add that to my presentation on Saturday.

I think it doesn't cover all of the features I want, but it might provide sufficient features to cover the general use case. Like tripwire, it will also cover more of the system than just /etc/.

In any case, I really like etckeeper with snapshotting :).

Avatar
Utku | March 25, 2017
No readers like this yet.

Or just use a configuration management (CM) tool (such as Ansible, Puppet, etc.) and make all your changes to a repository of your CM tool.

Avatar
Andreas | March 28, 2017
No readers like this yet.

Or just use rsnapshot (http://rsnapshot.org/) to have backups of whatever you need, differing versions etc. without spending too much diskspace.

Avatar
storm9c1 | March 28, 2017
No readers like this yet.

Good article. It's always an excellent idea to use a version control system to track changes in system files. And even more interesting to think about managing entire systems that way. For over a decade, I've been designing and engineering a suite of tools, processes and workflows that work with packaging systems (rpm and deb) to allow packages to be used for configuration management, configuration overlays, verification and self-governance through the packaging system (ie: rpm -V). I know that packaging systems work on a large scale. And the concern about overwriting local changes is exactly what a true CM system should do, since all changes should go through the packaging system. No exceptions. I encourage the author and readers to take the version control concept to the next level and look into authoring native OS packages. Then all that's needed is to manage the package build trees (or, ie: SRPMs) with a version control system. The nice thing about this approach is that all Linux systems (and even legacy Unix systems for that matter) have a native packaging system. No additional software is needed on the client side to manage hundreds, thousands, or even millions of nodes. Tools like Ansible can be used to deploy packages, but can also be divorced from CM, so there is no need to marry a system like Ansible, Puppet or Chef. Which ever way the wind blows, whatever a corporation wants to use for lifecycle management, the packaging system will always remain as the one truth. Food for thought and I'm glad to hear that folks are thinking this way.

Avatar
der.hans | March 30, 2017
No readers like this yet.

Utku - Configuration Mangement Systems (CMS) such as puppet, ansible and chef are great and provide part of what I describe in the article. Professional environments should be using CMS or other reproducable, automated, scalable tools for deployment. In most cases a CMS does not track all files in /etc/, so I suggest keeping /etc/ in etckeeper on the CMS managed server in addition to the CMS.

I admit that might be a hard case to make in a virtualized environment, so consider the missing features and whether or not they matter for your environment.

Avatar
der.hans | March 30, 2017
No readers like this yet.

Andreas - I love rsnapshot and have some backups that are very much dependent on it. I feel it doesn't provide all the features of etckeeper. Specifically it generally doesn't keep all changes and it doesn't automagically kick off after package management changes.

rsnapshot does provide an important additional feature, namely off-system backups. You can use git to get the off-system backup, but rsnapshot is also a great option for that.

Related Content

Creative Commons LicenseThis work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License.