kubicluster

kubicluster is a bash script collection that helps you setup a kubernetes cluster inside virtual machines from scratch. It’s predominant purpose is to help you to learn the ins and outs of kubernetes. You might also use it to use it as an inspirational basis to (semi)-automate your kubernetes cluster management.

If you are looking for a more sophisticated, production-ready approach, checkout https://github.com/kubernetes-sigs/kubespray or use kubeadm.

DISCLAIMER: kubicluster is by all means a constant work in progress, if you have questions, suggestions, found a bug, extended it, and so on … please contribute it as an issue and/or a PR to this repository

Campsite principle: kubicluster is a contribution aggregating hundreds of bits and pieces of resources into a set of scripts to automate a small cluster setup. It would not have been possible to be assembled and coded without all those people contributing their bits and pieces of knowledge as open-knowledge and open-source. If you benefited from kubicluster, consider contributing a little bit to it (or to any other part of the kubernetes ecosystem), so it will be a little bit easier for the next one using it. Some inspiration for contributing to kubicluster can be found in the roadmap in this README.

Suported tool version combination(s)

It currently supports:

virsh version

# should yield the following version numbers (or later versions) on your hypervisor
Compiled against library: libvirt 5.0.0
Using library: libvirt 5.0.0
Using API: QEMU 5.0.0
Running hypervisor: QEMU 3.1.0

If you would like to see other options or versions supported as well, you are welcome to open a PR.

kubicluster emphasizes runtime security and workload isolation. Therefore, it uses kata containers as the default runtime and runc as an optional on-demand runtime. Calico is not yet compatible with kata containers as the underlying container runtime for the same reason, why docker host network support is not working inside kata containers (https://github.com/kata-containers/documentation/blob/master/Limitations.md#docker—nethost). Therefore kubicluster runs calico pods using runc as the low-level runtime.

Prerequisites

You should know the basics about kvm, virsh and libvirt-qemu (or any other underlying virtualisation you might want to use). You should be familiar with the command line. All scripts are written in bash to keep the entry level barrier as low as possible (picking a higher level programming makes it more difficult to clearly see how all the tools interact on the command line level and how they depend on each other, more good arguments can be found here: https://github.com/my-own-kind/mokctl-docs/blob/master/docs/faq.md#why-bash).

You should have a clean virtual machine image with a fresh, empty install of debian 9 that can be used as a template for controller nodes and worker nodes. Inside this VM the root user needs to have ssh access using a key file. This can be revoked once the kubicluster setup is completed.

Acquire basic knowledge about kubernetes

Some good resources besides https://kubernetes.io/docs/home/ to learn more about kubernetes are listed here by topic. Take your time and invest some hours reading through them before starting with kubernetes, it will pay off.

Container Runtimes

KVM and virtualisation

Kubernetes networking with Calico

Kubernetes networking alternatives

Beyond the basics

Other tutorials, that might be more fitting for your future use cases

Basic Ideas / “Features”

General

Architecture of the resulting cluster

Everything should run inside virtual machines, so we can easily (re)create a cluster on a hypervisor with affecting the hypervisor and port the cluster to a different hypervisor / server easily.

There are 4 Layers in our cluster:

To setup a cluster from scratch, each layer has to be setup consecutively from (1) to (4).

Handling variables and arguments

Supporting the customisation of some variables through environment variables and through command arguments, is both convenient, but also allows to play around a bit more with them and to learn about the dependencies between all components, the commands to install them and configure them.

Security aspects

SSH access into virtual machines

Authentication over an SSH_KEY (instead of password) is considered safer and allows for an automation without waiting for a user interaction to provide a password. The ssh access of the root user can be limited to the hypervisor(s) on which the virtual machines will be running inside the VM template. If the virtual machines are only accessible via ssh from the hypervisor itself, but not from the outside world, an attacker would have to gain access to the hypervisor (which - in a production setup - should be hardened and protected through various means like firewall by default) to be able to gain access to a virtual machine / a node. If an attacker gains access to a hypervisor, the whole server including it’s virtual machines would have to be considered compromised.

Certificates and authentication

Kubernetes relies heavily on certificates to manage access and authentication. To be future proof for a good while the default RSA keylength to be used to generate certificates is set to 8192. Any keylength of 2048 and higher is considered save. A custom keylength can be defined with the environment variable RSA_KEYLENGTH.

Networking

Nodes/Virtual machines should be able to communicate over a network with each other that is separate from their default connection to their respective host. This has two advantages:

Usage

Installation

You can simply install kubicluster by cloning this repository and adding a symbolic link pointing to the ./kubicluster script or directly calling it.

git clone https://github.com/sray/kubicluster.git
ln -s $(pwd)/kubicluster/kubicluster /usr/local/bin/kubicluster

The template vm

The template image (vm.qcow2), which you intend to use as a base image for your kubicluster nodes, needs to have two interfaces defined in it’s /etc/network/interfaces. To setup a minimal development cluster, this is usually not a standard requirement, but it is good practice in production environment to distribute different types of traffic across different networks and interfaces. Therefore, in kubicluster, having two interfaces from the start is a must. While it may take a little more time to understand this and set this up (if you are using kubicluster, there is no difference at all since kubicluster creates and manages both networks on the hypervisor for you), it gives you the opportunity to debug networking problems in your production cluster later on more easily, if your development environment resembles it more closely.

The following shows the configuration of two interfaces in /etc/network/interfaces inside a template VM using kubicluster default values:

# The primary network interface
allow-hotplug enp1s0
iface enp1s0 inet static
	address 192.168.122.254/24
	gateway 192.168.122.1

# The vm cluster network interface
allow-hotplug enp7s0
iface enp7s0 inet static
	address 192.168.24.254/24
	gateway 192.168.24.1

When executing the step/sub-command kubicluster create-vms those default values are overwritten. If the definition of those two network interfaces deviates from the expected default values, you need to provide them as arguments to kubicluster create-vms. The gateways need to be set correctly inside the template vm already (otherwise your template vm would not be managable through ssh plus testing the connection between the cluster bridge and the template vm helps in debugging networking problems later on).

Any other interface specific configuration can be added (e.g. mtu in the following overview) and will not be changed by kubicluster.

# The primary network interface                             ### corresponding command line argument
allow-hotplug ${TEMPLATE_DEFAULT_CONNECTION_INTERFACE}      # -tif=enp1s0|--template-interface=enp1s0
iface ${TEMPLATE_DEFAULT_CONNECTION_INTERFACE} inet static  # -tif=enp1s0|--template-interface=enp1s0
	address ${TEMPLATE_DEFAULT_CONNECTION_IP}/24              # -tip=192.168.122.254|--template-ip=192.168.122.254
	gateway 192.168.122.1

# The vm cluster network interface
allow-hotplug ${TEMPLATE_CLUSTER_CONNECTION_INTERFACE}      # -tcif=enp7s0|--template-cluster-interface=enp7s0
iface ${TEMPLATE_DEFAULT_CONNECTION_INTERFACE} inet static  # -tcif=enp7s0|--template-cluster-interface=enp7s0
	address ${TEMPLATE_CLUSTER_CONNECTION_IP}/24              # -tcip=192.168.24.254|--template-cluster-ip=192.168.24.254
	gateway 192.168.24.1
	mtu 1400

Standard flow to create a small development cluster

The following creates a small cluster on one hypervisor with one controller (also hosting the one etcd instance) and two worker nodes.

# set the c-level net for network connections between hypervisor and a vm
HYPERVISOR_NET=192.168.122
# set the c-level net for network connections between vms (kubernetes cluster traffic will go through them)
# note: this specifices the VM cluster network ON TOP OF WHICH the kubernetes cluster 10.32.i.i/16 will run
# note: so far only c-nets within 192.168.0.0/16 are supported
VM_CLUSTER_NET=192.168.24
CONTROLLER_01=kubi-controller-01,${VM_CLUSTER_NET}.11,${HYPERVISOR_NET}.11
WORKER_0001=kubi-worker-0001,${VM_CLUSTER_NET}.21,${HYPERVISOR_NET}.21
WORKER_0002=kubi-worker-0002,${VM_CLUSTER_NET}.22,${HYPERVISOR_NET}.22
# note: you manage IP allocation, so be sure you do not have IP conflicts between nodes on the same hypervisor

# checkout the optional arguments template-(interface|ip) and template-cluster-(interface|ip) for create vms
./kubicluster create-vms help
# if your template vm uses other interface names or different ips for those interfaces add the respective arguments to the following create-vms commands (e.g. --template-cluster-interface=enp7s0)

# DECIDE between A and B:
# A: in a dev environment (vms will only be able to communicate with one another if they are running on the same hypervisor)
./kubicluster prepare path/to/vm.qcow2 path/to/vm-root-ssh_rsa -tip=${VM_CLUSTER_NET} -ndev
./kubicluster create-vms ${CONTROLLER_01} ${WORKER_0001} ${WORKER_0002} -ndev

# B: in a production environment (multi-hypervisor support, cluster network is bridged)
./kubicluster prepare path/to/vm.qcow2 path/to/vm-root-ssh_rsa -tip=${VM_CLUSTER_NET}
./kubicluster create-vms ${CONTROLLER_01} ${WORKER_0001} ${WORKER_0002}

# generate _C_ertificates a_N_d _C_onfiguration files
RSA_KEYLENGTH=2048 ./kubicluster cnc -c ${CONTROLLER_01} -w ${WORKER_0001} -w ${WORKER_0002}
./kubicluster create-controllers -c ${CONTROLLER_01} --force-etcd-data-reset # etcd data have to be reset on initial install to initialise encryption of etcd
./kubicluster create-workers -c ${CONTROLLER_01} -w ${WORKER_0001} -w ${WORKER_0002}

Running these commands to setup a cluster is rather straight forward. If you want to look under the hood and run all the sub-commands yourself, checkout the *) section in the sub-command argument parsing within the sub-command shell script files.

If you later want to add another worker (e.g. kubi-worker-0003) to your cluster, you only need to execute the following:

HYPERVISOR_NET=192.168.122
# set the c-level net for network connections between vms (kubernetes cluster traffic will go through them)
# note: this specifices the VM cluster network ON TOP OF WHICH the kubernetes cluster 10.32.i.i/16 will run
# note: so far only c-nets within 192.168.0.0/16 are supported
VM_CLUSTER_NET=192.168.24
CONTROLLER_01=kubi-controller-01,${VM_CLUSTER_NET}.11,${HYPERVISOR_NET}.11
WORKER_0003=kubi-worker-0003,${VM_CLUSTER_NET}.23,${HYPERVISOR_NET}.23

./kubicluster create-vms ${WORKER_0003} -ndev
# or without the -ndev setting if you are using a real bridge linked to a physical interface or simply managed by yourself directly in the OS:
# ./kubicluster create-vms ${WORKER_0003}
./kubicluster cnc for_worker_nodes -w ${WORKER_0003} -c ${CONTROLLER_01}
./kubicluster create-workers -c ${CONTROLLER_01} -w ${WORKER_0003}

Diving deeper

The kubicluster command and each sub-command show information on how to use it and which environment variables and which command arguments are supported when called with help:

./kubicluster help
./kubicluster prepare help
./kubicluster create-vms help
./kubicluster cnc help
./kubicluster create-controllers help
./kubicluster create-workers help

All scripts are written in plain bash/shell. If you dive into them and feel there could be more explanation or a clarifying comment would be helpful in a place that is not yet commented (enough) for beginners, please consider contributing to the community and open a PR.

What each script / sub-command is doing

kubicluster prepare

NOTE: install_dependencies and setup_virtualisation should run on any debian/ubuntu hypervisor without problems, for other distros you need to execute the corresponding commands yourself, or better yet - submit a PR to this repo ;-)

kubicluster create-vms

kubicluster cnc

kubicluster create-controllers

kubicluster create-workers

In case, you run into networking issues, that you cannot solve and would like to start from scratch, you can delete some of the kubernetes resources to force a redeployment of calico networking pods before re-running the create-workers sub-command:

kubectl delete daemonset calico-node -n kube-system
kubectl delete deployment calico-kube-controllers -n kube-system
# then re-run
/kubicluster create-workers ...

Road Map