MicroK8s in LXD
MicroK8s can also be installed inside an LXD container. This is a great way, for example, to test out clustered MicroK8s without the need for multiple physical hosts.
You can install LXD via snaps:
sudo snap install lxd sudo lxd init
Add the MicroK8s LXD profile
MicroK8s requires some specific settings to work within LXD (these are explained in more detail below). These can be applied using a custom profile. The first step is to create a new profile to use:
lxc profile create microk8s
Once created, we’ll need to add the rules. If you’re using ZFS, you’ll need this version or, if you’re using ext4, you’ll need this one. There is a section at the end of this document to describe what these rules do.
Download the profile:
# for ZFS wget https://raw.githubusercontent.com/ubuntu/microk8s/master/tests/lxc/microk8s-zfs.profile -O microk8s.profile # for ext4 wget https://raw.githubusercontent.com/ubuntu/microk8s/master/tests/lxc/microk8s.profile -O microk8s.profile
We can now pipe that file into the LXD profile.
cat microk8s.profile | lxc profile edit microk8s
And then clean up.
Start an LXD container for MicroK8s
We can now create the container that MicroK8s will run in.
lxc launch -p default -p microk8s ubuntu:20.04 microk8s
Note that this command uses the ‘default’ profile, for any existing system settings (networking, storage, etc.) before also applying the ‘microk8s’ profile - the order is important.
Install MicroK8s in an LXD container
First, we’ll need to install MicroK8s within the container.
lxc exec microk8s -- sudo snap install microk8s --classic
Load AppArmor profiles on boot
When the LXD container boots it needs to load the AppArmor profiles required by MicroK8s or else you may get the error:
cannot change profile for the next exec call: No such file or directory
To automate the profile loading first enter the LXD container with:
lxc shell microk8s
Then create an
rc.local file to perform the profile loading:
cat > /etc/rc.local <<EOF #!/bin/bash apparmor_parser --replace /var/lib/snapd/apparmor/profiles/snap.microk8s.* exit 0 EOF
chmod +x /etc/rc.local
Accessing MicroK8s Services Within LXD
Assuming you left the default bridged networking when you initially setup LXD, there is minimal effort required to access MicroK8s services inside the LXD container.
Simply note the eth0 interface IP address from
lxc list microk8s
and use this to access services running inside the container.
Exposing Services To Node
You’ll need to expose the deployment or service to the container itself before you can access it via the LXD container’s IP address. This can be done using
kubectl expose. This example will expose the deployment’s port 80 to a port assigned by Kubernetes.
In this example, we will use Microbot as it provides a simple HTTP endpoint to expose. These steps can be applied to any other deployment.
First, let’s deploy Microbot (please note this image only works on
lxc exec microk8s -- sudo microk8s kubectl create deployment microbot --image=dontrebootme/microbot:v1
Then check that the deployment has come up.
lxc exec microk8s -- sudo microk8s kubectl get all NAME READY STATUS RESTARTS AGE pod/microbot-6d97548556-hchb7 1/1 Running 0 21m NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/kubernetes ClusterIP 10.152.183.1 <none> 443/TCP 21m NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/microbot 1/1 1 1 21m NAME DESIRED CURRENT READY AGE replicaset.apps/microbot-6d97548556 1 1 1 21m
As we can see, Microbot is running. Let’s expose it to the LXD container.
lxc exec microk8s -- sudo microk8s kubectl expose deployment microbot --type=NodePort --port=80 --name=microbot-service
We can now get the assigned port. In this example, it’s
lxc exec microk8s -- sudo microk8s kubectl get service microbot-service NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE microbot-service NodePort 10.152.183.188 <none> 80:32750/TCP 27m
With this, we can access Microbot from our host but using the container’s address that we noted earlier.
The dashboard addon has a built in helper. Start the Kubernetes dashboard
lxc exec microk8s -- microk8s dashboard-proxy
127.0.0.1 with the container’s IP address we noted earlier.
Explanation of the custom rules
boot.autostart: “true”: Always start the container when LXD starts. This is needed to start the container when the host boots.
linux.kernel_modules: Comma separated list of kernel modules to load before starting the container
lxc.apparmor.profile=unconfined: Disable AppArmor. Allow the container to talk to a bunch of subsystems of the host (eg /sys) (see ). By default AppArmor will block nested hosting of containers, however Kubernetes needs to host Docker containers. Docker containers need to be confined based on their profiles thus we rely on confining them and not the hosts. If you can account for the needs of the Docker containers you could tighten the AppArmor profile instead of disabling it completely, as suggested in .
lxc.cap.drop=: Do not drop any capabilities . For justification see above.
lxc.mount.auto=proc:rw sys:rw: Mount proc and sys rw . For privileged containers, lxc over-mounts part of /proc as read-only to avoid damage to the host. Kubernetes will complain with messages like “Failed to start ContainerManager open /proc/sys/kernel/panic: permission denied”
security.nesting: “true”: Support running LXD (nested) inside the container.
security.privileged: “true”: Runs the container in privileged mode, not using kernel namespaces [4, 5]. This is needed because hosted Docker containers may need to access for example storage devices (See comment in ).
devices: disable /sys/module/nf_conntrack/parameters/hashsize and /sys/module/apparmor/parameters/enabled: Hide two files owned by the host from the LXD containers. Containers cannot disable AppArmor or set the size of the connection tracking table.
-  Container nesting: https://stgraber.org/2012/05/04/lxc-in-ubuntu-12-04-lts/.
-  Capabilities https://stgraber.org/2014/01/01/lxc-1-0-security-features/.
-  Mount proc and sys in privileged containers: https://github.com/lxc/lxd/issues/3042.
-  Unprivileged containers: https://unix.stackexchange.com/questions/177030/what-is-an-unprivileged-lxc-container/177031#177031.
-  Privileged containers: http://blog.benoitblanchon.fr/lxc-unprivileged-container/.
-  Lxc Security https://wiki.ubuntu.com/LxcSecurity.
-  Connections tracking: https://wiki.khnet.info/index.php/Conntrack_tuning.
Last updated 11 months ago. Help improve this document in the forum.