How to teach student sys admins

No readers like this yet.
Book stack
Image by:

Image by Kate Ter Haar. Modified by Opensource.com. CC BY-SA 2.0.

Several years ago, when I was working on my Master's degree in Library Science, I took a course on Unix (Solaris) system administration. The course was supposed to involve setting up a web server on a Sun workstation, starting with a fresh, bare-metal install of the Solaris operating system and building things up from there.

Sounds great, right? I certainly thought so, but after the course started, the professor was told to tone things down because someone higher up in the university was concerned that student-administered machines could get hacked and the department would be held accountable. The course was modified so that we had to do a bunch of smaller, less complex tasks by logging into a workstation the professor had complete control over. We did not get to install Apache and PHP, but we did get to work through several interesting and challenging problems. As good as the course ended up being, it was still a pale imitation of what it could have been had we been given the freedom to administer our own machines.

A lot has changed technology-wise since I took that course, and because a new academic year is starting, I thought this was a great time to have a look at what a modern system administration class's homework should look like. Should a student's first system administration experience come with a safety net, so that there are not too many security problems when they make mistakes? Or should a course be as realistic as possible, even if that is supposedly dangerous? I lean heavily toward the "realistic" side, and I am going to share an excellent example of someone teaching that way. If you have different ideas, please share them in the comments section below.

Realistic system administration homework

Tom Clark from Otago Polytechnic gave a presentation at Linux.conf.au earlier this year that provided an excellent example of a modern, realistic system administration course. The content of the course is presented using Clark's three main teaching principles:

  1. Doing something is better than just talking about it.
  2. Use a consistent thread of development so that topics build on each other, and so that lessons from the first day are still relevant on the final day.
  3. Realism, which involves real tasks, real tools, and real assessment.

These three ideas permeate the course and are used to provide the students with a realistic, hands-on learning experience.

The learning experience

Students spend the 16-week long course learning practical skills using real tools. To support their systems, students learn about using support tickets and documentation by using RT and MediaWiki. To deploy and maintain their systems, they learn about configuration management using Puppet, system monitoring using Nagios, and backup and recovery using Bacula. But the broad concepts are more important than the specific software packages I just mentioned. The point is to learn, for example, configuration management, not to be trained to use Puppet. The software used by Clark is used because it works for him, but the software is flexible and changeable.

The major project involves pairs of students deploying, maintaining, and supporting a deployment of ownCloud. They are responsible for deploying on time and keeping the system up and running with a minimum of downtime for a two-week period. During this time, Clark functions as a user and a manager opening support tickets. In his user role, he requests help with password resets and other basic tasks. As a manager, he requests configuration changes, new Nagios checks, and other administrative tasks. Students are expected to handle the issues promptly, document their processes, and communicate effectively.

In addition to opening support tickets, Clark makes things difficult for the students by intentionally breaking things. He has root access on all the servers, so he just logs in and tweaks things, such as shutting down MySQL. At some point during the project, Clark does something brutal—basically simulating a real-world hacking by deleting important files and defacing the site. He says that for "optimal learning experience" he does this around 3:00 AM.

Grading is based on uptime and the students' handling of support tickets and other issues. Clark uses his own Nagios instance to monitor uptime and he reviews their ticket logs and wiki to see how well the students handled issues and worked together as a team. He then meets with the students face-to-face to discuss their performance.

My own experience as a student was sub-par because someone was worried that a bunch of students might mess up and cause problems for the department and/or university, even though messing up is part of the learning process, and the chance for our workstations to be hacked was minuscule. Clark's students, on the other hand, get to experience situations like they would in a real-world setting, except for the fact that if something does go horribly wrong, Clark is there to step in and resolve the problem, even if that means just shutting down the offending virtual machine.

Overall, Clark's course seems like an excellent way to introduce students to system administration. Students get hands-on, practical experience with a wide variety of software, and the experience gives them something to talk about during job interviews. A college course should not necessarily be job training in itself, but it should help develop skills that can be used in real situations. Clark's course provides experience with real tools and, more importantly, helps develops the problem-solving and critical-thinking skills needed in the modern job market.

I would love to hear what professional sys admins and people who teach system administration—both in and outside academia—think. As Clark notes in his presentation, we know far more about teaching programming than we do about teaching system administration. So if you have any thoughts on the subject, feel free to leave them in the comments section below.

Back to
School

This article is part of the Back to School series focused on open source projects and tools for students of all levels.

Tags
Education
Sysadmin
Linux

7 Comments

Avatar
Scott Dowdle | September 1, 2015
No readers like this yet.

I watched that video when it was first released and enjoyed it.

I've taught a Linux sysadmin class every Spring for the last 7 years and while the content of the class has changed some over that time, the setup hasn't changed much. Originally I had one 4 core server with 16GB of RAM running OpenVZ (first on EL5 and later EL6 and hopefully EL7 next Spring if Virtuozzo 7 is done by then). I had a "course server" container that had a public IP address on which each student has a regular user account. Then on the same host there was a container for each student that had a private IP address. Students have root access to their own container... but to access it they have to first login to the course container... and then ssh from the course container to their own container. While that isn't a perfect security solution, I think it helps some. Students have to have good passwords and ssh brute force attack mitigation is in place.

For the last two classes I got a second server running KVM with more RAM (1GB per student or better is a good idea if possible) and more disk space... and I also gave each student a KVM VM. The container ran EL6 and the KVM VM ran EL7... so students got access to sysvinit (upstart in compatibility mode) and systemd. There was a secondary course server KVM VM with a public IP from which they could get to their student KVM VM that had a private IP address.

We actually covered OpenVZ, containers, and KVM near the end of the class so students hopefully understood the environment at the end.

I didn't break things nor have a trouble-ticket system but they had plenty of hands on experience with the package manager, installing service, and configuring them... and I would verify their homework assignments and give plenty of feedback. The class had quite a bit of lecture material (in-class and screencast recording for anyone who missed class).

I agree, hands-on is definitely required for students to get much out of the class. I don't think I could get away with expecting students to fix a server at 3AM though. :)

Avatar
Don Watkins | September 1, 2015
No readers like this yet.

I think this is great and wish that such course were offered at the high school level. I learned a lot from students who built their own Linux systems at home and then shared their insights and work experience with me as I was learning Linux myself. I also offered to teach a course in Linux that would have included some basic system administration, really more of a familiarity course using VirtualBox to host Linux distros on Windows machines. That offer was turned down about five years ago by a local educational agency. I was trained as a CCNA instructor about a dozen years ago and subsequently thought that Linux system administration could and should be taught as part of programs like that at vocational education centers. There are just so many options that are possible and so many topics that could be covered and interconnected in such course including IP subnetting, routing, web hosting, file serving with Samba and NFS along with virtualization with different platforms.

Avatar
Tom Clark | September 1, 2015
No readers like this yet.

Thanks for this article. It's satisfying to see my worked affirmed in this way, and I hope that it will lead to good discussions that help us improve our teaching.

One thing I'd like to point out is that this sort of class works because the students work very hard at it. There's nothing a lecturer can do to save a class if the students don't buy in to the process. I think I get that buy-in because the students recognise the value of what they are asked to do, They're more than willing to work hard provided that they understand why they are expected to do so.

Avatar
Carl T. Miller | September 8, 2015
No readers like this yet.

Great video! Thanks for allowing your presentation to be online.

I'm currently developing the course material for two advanced Linux/Unix system administration classes, and I appreciate what you're doing. In my case the students' VMs are not powered on 24x7, so I need to tailor my material accordingly.

The idea of giving real life tasks is key. One of my primary goals is to ensure students know what documentation is available, how to sort out potentially misleading sources (you'd be surprised how many times students take the first answer from a Google search), and how to find answers for themselves. As a former instructor taught me, you don't need to know all the answers, you just need to know where to find them.

Avatar
Dan Mossor | September 8, 2015
No readers like this yet.

Great article! I am in a Computer Information Systems class at a university here in the states, and as a seasoned IT professional I am helping design the next evolution of coursework for the students in this program. I never thought of a class such as this, but you can bet I'll be working up plans to implement it.

One nit to pick, though - your link for Request Tracker takes you to the internal GNOME sysadmin ticketing site, where people looking for information on RT won't find any. The link you should have used is https://www.bestpractical.com/rt/

Avatar
Joshua Allen Holm | September 10, 2015
No readers like this yet.

Dan,

Thank you for your "nitpick". We've change the link in the article to point to the site you suggested.

Avatar
saturno | September 24, 2015
No readers like this yet.

Very nice!

Can i ask what infrastructure resources are used for this class?
Are you using an internal (at Otago Polytechnic) virtualization solution, how many servers each team have to manage, do you deploy the infrastructure for the class on some public cloud?
Thanks again, it was very interesting to watch your talk.

Related Content

Creative Commons LicenseThis work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License.