Cluster computing on the Raspberry Pi with Kubernetes

Ever wanted to make your very own cloud? Now you can! All it takes is some cheap open source hardware and open source software. For about $200, I was able to set up four Raspberry Pi 2s with the Kubernetes cloud operating system using Fabric8.

Fabric8 is an open source DevOps and integration platform that works out of the box on any Kubernetes or OpenShift environment and provides continuous delivery, management, ChatOps, and a Chaos Monkey. In short, it's the best framework there is to develop and manage your microservices on. (Full disclosure: I work on the Red Hat Fabric8 project.)

Never before was there a better match between a software architecture and the hardware it runs on. The Pis are completely silent (no fan) and have a quite powerful quad-core CPU, while the microservices architecture makes each process relatively small so it can run on the 1GB RAM memory constraint of each node. You can simply add more Pis if you need more computing power. Lastly, it's just plain fun to play with the hardware instead of logging into a remote virtual machine at Amazon.

1. Required hardware

To build this four-Pi setup I used:

• 4 Raspberry Pi 2s
• 4 16GB MicroSD cards (Class 10)
• 1 60W power supply with USB outlets
• 4 short USB to Micro USB cables (for powering the Pis)
• 4 short Cat 5 network cables
• 1 longer Cat 5 network cable to hook into your network
• 1 network hub (Mine is an old five-port, 10/100MBps I dusted off)
• LEGOs (Trust me, it feels good to build your own!)

It's important to get Class 10 MicroSD cards, as this is the one component that greatly affects the speed of the system. You can easily get away with a smaller 20W power supply for four Pis, but I'm planning on adding more later so I got a bigger one. The network port on the Pi is 100MBps, so you can use your old hub.

2. Flashing the MicroSD cards

Now we need to get the operating system onto the MicroSD Cards. I'm using a Hypriot image. I tested a bunch of others, including Pidora, Redsleeve/CentOS, and Arch Linux, but concluded that Hypriot is the best for me at this point (mostly because of the native Docker support and the image being small). You can download the latest Hypriot image here.

1. Insert the memory card in the card reader. Do not mount it. If it automounts, unmount the card. Then write the image to the card. On my OS X machine, the command for doing so is:
   

   sudo dd bs=1m if=hypriot-rpi-20151103-224349.img of=/dev/diskn
   

Replace 'n' with the number of disk found using the disk utility: On Mac OS X 10.8.x Mountain Lion, "Verify Disk" (before unmounting) will display the BSD name as "/dev/disk2s1" or similar. Please double check or else you maybe formatting a disk you did not intend to format!
2. Wait. This may take 10 minutes or so.
3. Do this for all four MicroSD cards. You can work on your LEGO casing while waiting.

3. Power up and add DNS

You can now power up the first Pi. By default, use root/hypriot to log in. You will need to set up DNS for each of the Pis. If you don't want to hook up a monitor and keyboard to figure out its IP and MAC address, then you can use nmap. If you have a DNS service, provision it there. Otherwise, name the Pis in /etc/hosts. I named my Pis rpi-master, rpi-node-1, rpi-node-2, and rpi-node-3. After a quick reboot of each Pi, you'll see the names reflected in the cmd prompt. Note that if you skip the DNS step, your Kubernetes cluster will not work. I added the following to my /etc/hosts files on all machines:

192.168.1.9     rpi-master
192.168.1.21    rpi-node-1
192.168.1.18    rpi-node-2
192.168.1.23    rpi-node-3

4. Add swap (optional)

I added 1GB of swap space to my rpi-master because it runs a few more Kubernetes components. Some people seem concerned that frequent writes to the MicroSD card will make it fail quickly. I therefore decided to set the swappiness such that it only uses the swap as a last resort.

dd if=/dev/zero of=/swap/swapfile bs=1M count=1024
mkswap /swap/swapfile
swapon /swap/swapfile

Now you can check with top that you have a 1GB of swap space.

To make the swap permanent between reboots, add

/swap/swapfile none swap sw 0 0

to your /etc/fstab file. Finally, in /etc/sysctl.conf, I've set the swappiness to 1:

vm.swappiness = 1

"1" means it will only use swap when RAM use exceeds 99%.

5. Install Kubernetes

These instructions are to install the plain vanilla Kubernetes from Google. I had some issues compiling OpenShift 3 due to some 64-bit dependencies, and I will get back to that later. I would like go support both. All the code I'm using is checked in under https://github.com/Project31. If you want to compile your own Kubernetes binaries, check out my blog post about it.

5.1. Install Kubernetes master

The Kubernetes master runs the Kubernetes REST API, a scheduler, kubernetes-proxy, and a replication controller. It also uses etcd as a key-value store that is replicated with between other Kubernetes masters. For more details, please see the Kubernetes documentation.

^{Kubernetes master components.}

To install these components onto the master, log in to the master node and run:

git clone git@github.com:Project31/kubernetes-installer-rpi.git
cd kubernetes-install-rpi
./build-master.sh

Verify the running Docker containers using docker ps:

and the now set export KUBERNETES_MASTER=http://rpi-master:8080 so your kubectl can connect the Kubernetes REST API on the master. Now let take a look at the running pods using kubectl get pods

We can see that the kube-controller-rpi-master pod contains runs five Docker containers with the Kubernetes services mentioned above.

5.2. Install Kubernetes node

The Kubernetes node only runs the Kubernetes proxy and the pods.

^{Kubernetes client components.}

To install these components on a node, log in to the node and run:

git clone git@github.com:Project31/kubernetes-installer-rpi.git
cd kubernetes-install-rpi

Now edit the kube-procy.yaml and set --master=http://rpi-master:8080 to your Kubernetes master. Then edit the kubelet.service file and set your master's Kubernetes REST endpoint there as well (which in my case is http://192.168.1.9:8080).

Now you can run the install:

./build-worker.sh

And verify our proxy came up using docker ps:

The proxy is running! Set export KUBERNETES_MASTER=http://rpi-master:8080 so your kubectl can connect the Kubernetes REST API on the master. Now verify the nodes are all registered:

kubectl get nodes

Yay, it worked!

6. Open Docker for remote connections

We need to fix up the Docker configuration on the master so that it accepts remote connections and we can deploy something to it. Open the /etc/default/docker file for editing and set the DOCKER_OPTS:

DOCKER_OPTS="-H tcp://192.168.1.9:2375 -H unix:///var/run/docker.sock --storage-driver=overlay -D"

Specify the IP address of machine, or you can user '0.0.0.0' to bind to all interfaces. Now we can remotely push Docker images to the master.

7. Deploy a simple service

Let's deploy a simple service and scale to two pods to make sure things are working correctly:

kubectl -s http://localhost:8080 run httpd --image=hypriot/rpi-busybox-httpd --port=80
kubectl scale --replicas=2 rc httpd
kubectl get pods -o wide

We see that even though we executed the command on the master, it started a pod on node-2 and node-3:

8. Conclusion

First of all, it works! The Kubernetes master may not have much memory left to run services, but the nodes, on the other hand, have almost an entire 1GB available for Docker containers. If you ever need more computing power, you can simply add nodes. The extremely low price of this hardware platform, combined with the Kubernetes cloud OS and the Fabric8 DevOps and iPaaS capabilities make this truly disruptive technology. It really is a cloud in a box for just $200.