Troubleshoot Kubernetes with the power of tmux and kubectl

Image by:

kris krüg

Kubernetes is a thriving open source container orchestration platform that offers scalability, high availability, robustness, and resiliency for applications. One of its many features is support for running custom scripts or binaries through its primary client binary, kubectl. Kubectl is very powerful and allows users to do anything with it that they could do directly on a Kubernetes cluster.

Troubleshooting Kubernetes with aliases

Anyone who uses Kubernetes for container orchestration is aware of its features—as well as the complexity it brings because of its design. For example, there is an urgent need to simplify troubleshooting in Kubernetes with something that is quicker and has little need for manual intervention (except in critical situations).

There are many scenarios to consider when it comes to troubleshooting functionality. In one scenario, you know what you need to run, but the command's syntax—even when it can run as a single command—is excessively complex, or it may need one or two inputs to work.

For example, if you frequently need to jump into a running container in the System namespace, you may find yourself repeatedly writing:

kubectl --namespace=kube-system exec -i -t <your-pod-name>

To simplify troubleshooting, you could use command-line aliases of these commands. For example, you could add the following to your dotfiles (.bashrc or .zshrc):

alias ksysex='kubectl --namespace=kube-system exec -i -t'

This is one of many examples from a repository of common Kubernetes aliases that shows one way to simplify functions in kubectl. For something simple like this scenario, an alias is sufficient.

Switching to a kubectl plugin

A more complex troubleshooting scenario involves the need to run many commands, one after the other, to investigate an environment and come to a conclusion. Aliases alone are not sufficient for this use

case; you need repeatable logic and correlations between the many parts of your Kubernetes deployment. What you really need is automation to deliver the desired output in less time.

Consider 10 to 20—or even 50 to 100—namespaces holding different microservices on your cluster. What would be helpful for you to start troubleshooting this scenario?

You would need something that can quickly tell which pod in which namespace is throwing errors.
You would need something that can watch logs of all the pods in a namespace.
You might also need to watch logs of certain pods in a specific namespace that have shown errors.

Any solution that covers these points would be very useful in investigating production issues as well as during development and testing cycles.

To create something more powerful than a simple alias, you can use kubectl plugins. Plugins are like standalone scripts written in any scripting language but are designed to extend the functionality of your main command when serving as a Kubernetes admin.

To create a plugin, you must use the proper syntax of kubectl-<your-plugin-name> to copy the script to one of the exported pathways in your $PATH and give it executable permissions (chmod +x).

After creating a plugin and moving it into your path, you can run it immediately. For example, I have kubectl-krawl and kubectl-kmux in my path:

$ kubectl plugin list
The following compatible plugins are available:

/usr/local/bin/kubectl-krawl
/usr/local/bin/kubectl-kmux

$ kubectl kmux

Now let's explore what this looks like when you power Kubernetes with tmux.

Harnessing the power of tmux

Tmux is a very powerful tool that many sysadmins and ops teams rely on to troubleshoot issues related to ease of operability—from splitting windows into panes for running parallel debugging on multiple machines to monitoring logs. One of its major advantages is that it can be used on the command line or in automation scripts.

I created a kubectl plugin that uses tmux to make troubleshooting much simpler. I will use annotations to walk through the logic behind the plugin (and leave it for you to go through the plugin's full code):

#NAMESPACE is namespace to monitor.
#POD is pod name
#Containers is container names

# initialize a counter n to count the number of loop counts, later be used by tmux to split panes.
n=0;

# start a loop on a list of pod and containers
while IFS=' ' read -r POD CONTAINERS
do

           # tmux create the new window for each pod
            tmux neww $COMMAND -n $POD 2>/dev/null

           # start a loop for all containers inside a running pod
	for CONTAINER in ${CONTAINERS//,/ }
	do

  	if [ x$POD = x -o x$CONTAINER = x ]; then
    	# if any of the values is null, exit.
    	warn "Looks like there is a problem getting pods data."
    	break
  	fi
           
            # set the command to execute
  	COMMAND=”kubectl logs -f $POD -c $CONTAINER -n $NAMESPACE”
 	# check tmux session
  	if tmux has-session -t <session name> 2>/dev/null;
  	then
    	<set session exists>
  	else
    	<create session>
  	fi

           # split planes in the current window for each containers
  	tmux selectp -t $n \; \
    	splitw $COMMAND \; \
    	select-layout tiled \;

           # end loop for containers
	done

           # rename the window to identify by pod name
	tmux renamew $POD 2>/dev/null
       
            # increment the counter
	((n+=1))

# end loop for pods
done< <(<fetch list of pod and containers from kubernetes cluster>)

# finally select the window and attach session
 tmux selectw -t <session name>:1 \; \
  attach-session -t <session name>\;

After the plugin script runs, it will produce output similar to the image below. Each pod has its own window, and each container (if there is more than one) is split by the panes in its pod window, streaming logs as they arrive. The beauty of tmux can be seen below; with the proper configuration, you can even see which window has activity going on (see the white tabs).

Conclusion

Aliases are always helpful for simple troubleshooting in Kubernetes environments. When the environment gets more complex, a kubectl plugin is a powerful option for using more advanced scripting. There are no limits on which programming language you can use to write kubectl plugins. The only requirements are that the naming convention in the path is executable, and it doesn't have the same name as an existing kubectl command.

To read the complete code or try the plugins I created, check my kube-plugins-github repository. Issues and pull requests are welcome.