Harvard goes PaaS with SELinux Sandbox

We are using a policy module type enforcement (te) file here to extend sandbox_t.

First, we needed to specify that sandbox_t is required for this module. Second, we used the corenet interfaces to allow sandbox_t to connect to any TCP port and send/recv from any UDP port. We also enabled the sysnet_dns_name_resolve interface that actually allows the use of the DNS resolver, and automatically uses other policy to allow us to create TCP and UDP sockets. During testing, we noticed that the shell was using /dev/urandom for connections, so we also allowed this. 

 

Our policy file finished, we compiled it using the supplied make file:

# make -f /usr/share/selinux/devel/Makefile mysandbox.pp

Finally, we installed the compiled policy package, and retested to see if we could connect to the network:

# semodule -i mysandbox.pp

# runcon -t sandbox_t nc localhost 80

sandbox_t is now allowed to connect to port 80.

Harvard would then need to distribute and install the compiled policy file (mysandbox.pp) on all of their test servers by running these same commands. Puppet and Chef are excellent tools for automating this.

Multiple domains

Harvard had another idea—they wanted to allow professors to upload their own test suites along with their students' code. However, the current setup presented a problem:

Students' code could affect the professors test suites!

If I can modify the professors test to always return success, I can probably improve my grades.

We decided to write a new policy where the processes on the system would run with two different types: sandbox_staff_t and sandbox_t. We also created two file types, sandbox_staff_file_t and sandbox_file_t. The $HOME directory and all its contents were labeled sandbox_staff_file_t. $HOME now contained two directories, staff and student. The student directory's contents were labeled sandbox_file_t. The staff tests were stored in the staff directory, labeled sandbox_staff_file_t. The rules for the process types were that sandbox_t can manage all content in sandbox_file_t, and sandbox_staff_t can manage all content in sandbox_staff_file_t and sandbox_file_t.

The sandbox_staff_t process needed to be able to execute scripts within the student directory, but it could not execute as sandbox_staff_t or this would allow the students' scripts to attack the staff directory. We needed a transition rule saying that when the sandbox_staff_t process executes a file labeled sandbox_file_t it transitions to sandbox_t. We also wanted sandbox_staff_t to be allowed to send signals, i.e. 'kill,' to the sandbox_t processes and allow it to ptrace them.

Here is the policy we wrote for this:

policy_module(mysandbox,2.0)

gen_require(`

type sandbox_t;

type sandbox_file_t;

attribute sandbox_file_type;

')

# extend sandbox to be allowed to connect to TCP and

# UDP sockets

corenet_tcp_connect_all_ports(sandbox_t)

corenet_udp_sendrecv_all_ports(sandbox_t)

sysnet_dns_name_resolve(sandbox_t)

dev_read_urand(sandbox_t)

# Create new sandbox domain sandbox_staff_t, this

# interface also adds the rules to allow sandbox_staff_t to

# manage all content labeled sandbox_file_t.

sandbox_domain_template(sandbox_staff)

# Create a type for all staff content

type sandbox_staff_file_t, sandbox_file_type;

files_type(sandbox_staff_file_t)

# Rules to allow sandbox_staff_t, to manage content in

# sandbox_staff_file_t.

manage_files_pattern(sandbox_staff_t, sandbox_staff_file_t, sandbox_staff_file_t)

manage_dirs_pattern(sandbox_staff_t, sandbox_staff_file_t, sandbox_staff_file_t)

manage_sock_files_pattern(sandbox_staff_t, sandbox_staff_file_t, sandbox_staff_file_t)

manage_fifo_files_pattern(sandbox_staff_t, sandbox_staff_file_t, sandbox_staff_file_t)

manage_lnk_files_pattern(sandbox_staff_t, sandbox_staff_file_t, sandbox_staff_file_t)

 

# Now we write rules to control what happens when

# sandbox_staff_t executes a program,

 

# If the process executes a program labeled

# sandbox_staff_file_t it will run as sandbox_staff_t

can_exec(sandbox_staff_t, sandbox_staff_file_t)

 

# If the sandbox_staff_t process runs a file labeled

# sandbox_file_t, it will transition to sandbox_t.

domtrans_pattern(sandbox_staff_t, sandbox_file_t,sandbox_t)

 

# Allow the sandbox_staff_t process to control the

# sandbox_t process.

allow sandbox_staff_t sandbox_t:process { signal_perms ptrace };

ps_process_pattern(sandbox_staff_t, sandbox_t)

 

 

# Allow the sandbox_t process to search through the

# sandbox_staff_t directories

allow sandbox_t sandbox_staff_file_t:dir search_dir_perms;

 

# Since the tests are going to be executing shells, shell

# programs seem to want to read /proc files

kernel_read_system_state(sandbox_t)

kernel_read_system_state(sandbox_staff_t)

Compile and install:

# make -f /usr/share/selinux/devel/Makefile mysandbox.pp

# semodule -i mysandbox.pp

Test to make sure we can run as sandbox_staff_t:

# runcon -t sandbox_staff_t id -Z

staff_u:unconfined_r:sandbox_staff_t:s0-s0:c0.c1023

Good, it worked. Now, setup a test environment.

Setting up a test environment

First, setup a home directory "class" with sub-directories staff and student, and write test scripts which output the SELinux context that the scripts are running with and "Hello World."

$ mkdir -p class/staff class/student

$ cat > class/staff/test.sh

#!/bin/sh

id -Z

echo "Hello world staff"

$ cat > class/student/test.sh

#!/bin/sh

id -Z

echo "Hello world student"

$ chmod +x class/staff/test.sh class/student/test.sh

Set the context of the directories, default for all to sandbox_staff_file_t, and class/student to sandbox_file_t;

$ chcon -R -t sandbox_staff_file_t class

$ chcon -R -t sandbox_file_t class/student

Check the labels using find:

$ find class -printf "%p\t\t%Z\n"

class  staff_u:object_r:sandbox_staff_file_t:s0
class/student  staff_u:object_r:sandbox_file_t:s0
class/student/test.sh  staff_u:object_r:sandbox_file_t:s0
class/staff  staff_u:object_r:sandbox_staff_file_t:s0
class/staff/test.sh  staff_u:object_r:sandbox_staff_file_t:s0

Begin testing.

$ id -Z

staff_u:staff_r:staff_t:s0-s0:c0.c1023

$ runcon -t sandbox_t id -Z

staff_u:unconfined_r:sandbox_t:s0-s0:c0.c1023

$ runcon -t sandbox_t touch class/bad

touch: cannot touch ‘class/bad’: Permission denied

$ runcon -t sandbox_t touch class/staff/bad

touch: cannot touch ‘class/staff/bad’: Permission denied

Excellent! This shows that a student process cannot write to the class or staff directories. If we allowed student code to write to these directories, it could create a .bashrc or profile file and trick the sandbox_staff_t process into doing something bad.

$ runcon -t sandbox_t touch class/student/good

This shows the student's code can write to their own directory, which is what we intended.

Verify that the sandbox_staff_t process can write to all directories:

$ runcon -t sandbox_staff_t touch class/staff/good
$ runcon -t sandbox_staff_t touch class/student/good

With this test passing, it's time to make sure that the context transition is working.

# Test to make sure the sandbox_staff_t process will transition to the proper context when running the students code, ie programs labeled sandbox_file_t.

$ runcon -t sandbox_staff_t class/staff/test.sh

staff_u:unconfined_r:sandbox_staff_t:s0-s0:c0.c1023

"Hello world staff"

$ runcon -t sandbox_staff_t class/student/test.sh

staff_u:unconfined_r:sandbox_staff_t:s0-s0:c0.c1023

"Hello world student"

Oops, this looks wrong! Why didn't the process run as sandbox_t? It looks like the transition failed.

The policy says that when a sandbox_staff_t process runs a sandbox_file_t, it should transition to sandbox_t. Instead we see the process running as sandbox_staff_t, so we know something is broken. Let me explain what really happened here.

When the runcon or sandbox command is executed, these commands tell the kernel to label the next process they execute as the specified type. In this case, runcon tells the kernel to run the class/student/test/sh as sandbox_staff_t.

In order to get the transitions to work the way we expect, we need to execute a program (/bin/sh) as the sandbox_staff_t and then allow that process to execute class/student/test.sh:

$ runcon -t sandbox_staff_t sh -c class/student/test.sh

staff_u:unconfined_r:sandbox_t:s0-s0:c0.c1023

"Hello world student"

This shows the process is running as sandbox_t, which means our policy is working correctly. Harvard can now test the sandbox with a command such as:

# sandbox -t sandbox_staff_t -h class staff/test.sh

Note: Monitoring the audit.log's to see what the students are doing can be daunting. Each day, someone needs to review the /var/log/audit/audit.log for AVC messages, which is one of the things I do for OpenShift.

It's easy to get overwhelmed by the number of AVCs generated by students exploring the system. I see many users doing stuff like find /, rpm -q, su, sudo, passwd--most of which I consider harmless becasue this is just users probing the system. Truth be told, if I was given an account on a system, I would probably do the same. But we found that when we removed the ability for a normal user to execute these programs the noise level was reduced considerably.

I recommend that you remove the "other" execute permission flag following: rpm, yum, debuginfo-install, traceroute, iptables, iptables-multi, dmesg, tcpdump, mtr, ip, su, sudo, denyhosts, chsh, chfn. Also modify the applications group to the wheel group and add any administors to the wheel group.

For example:

# chmod o-x /usr/bin/rpm

# chgrp wheel /usr/bin/rpm

# ls -l /usr/bin/rpm
-rwxr-xr--. 1 root wheel 27280 Jul 21 22:04 /usr/bin/rpm

# grep wheel /etc/group
wheel:x:10:dwalsh

This change would prevent student accounts from executing these commands and triggering AVC's that you are probably not concerned about.