It is important to recognise that things can go wrong. But MicroK8s gives you tools to help work out what has gone wrong, as detailed below. Be sure to check out the common issues section for help resolving the most frequently encountered problems.

Checking logs

If a pod is not behaving as expected, the first port of call should be the logs.

First determine the resource identifier for the pod:

microk8s kubectl get pods

This will list the currently available pods, for example:

NAME                                 READY   STATUS             RESTARTS   AGE
mk8s-redis-7647889b6d-vjwqm          1/1     Running            0          2m24s

You can then use kubectl to view the log. For example, for the simple redis pod above:

microk8s kubectl logs mk8s-redis-7647889b6d-vjwqm

Examining the configuration

If the problem you are experiencing indicates a problem with the configuration of the Kubernetes components themselves, it could be helpful to examine the arguments used to run these components.

These are available in the directory ${SNAP_DATA}/args, which on Ubuntu should point to /var/snap/microk8s/current.
Note that the $SNAP_DATA environment variable itself is only available to the running snap. For more information on the snap environment, check the snap documentation.

Using the built-in inspection tool

MicroK8s ships with a script to compile a complete report on MicroK8s and the system which it is running on. This is essential for bug reports, but is also a useful way of confirming the system is (or isn’t) working and collecting all the relevant data in one place.

To run the inspection tool, enter the command (admin privilege is required to collect all the data):

sudo microk8s inspect

You should see output similar to the following:

Inspecting services
  Service snap.microk8s.daemon-cluster-agent is running
  Service snap.microk8s.daemon-flanneld is running
  Service snap.microk8s.daemon-containerd is running
  Service snap.microk8s.daemon-apiserver is running
  Service snap.microk8s.daemon-apiserver-kicker is running
  Service snap.microk8s.daemon-proxy is running
  Service snap.microk8s.daemon-kubelet is running
  Service snap.microk8s.daemon-scheduler is running
  Service snap.microk8s.daemon-controller-manager is running
  Service snap.microk8s.daemon-etcd is running
  Copy service arguments to the final report tarball
Inspecting AppArmor configuration
Gathering system information
  Copy processes list to the final report tarball
  Copy snap list to the final report tarball
  Copy VM name (or none) to the final report tarball
  Copy disk usage information to the final report tarball
  Copy memory usage information to the final report tarball
  Copy server uptime to the final report tarball
  Copy current linux distribution to the final report tarball
  Copy openSSL information to the final report tarball
  Copy network configuration to the final report tarball
Inspecting kubernetes cluster
  Inspect kubernetes cluster

Building the report tarball
  Report tarball is at /var/snap/microk8s/1031/inspection-report-20191104_153950.tar.gz

This confirms the services that are running, and the resultant report file can be viewed to get a detailed look at every aspect of the system.

Common issues

Node is not ready when RBAC is enabled...

Ensure the hostname of your machine name does not contain capital letters or underscores. Kubernetes normalizes the machine name causing its registration to fail.

To fix this you can change the hostname or use the --hostname-override argument in kubelet’s configuration in /var/snap/microk8s/current/args/kubelet.

My dns and dashboard pods are CrashLooping...

The cni network plugin used by MicroK8s creates a cni0 interface (cbr0 on pre v1.16 releases) when the first pod is created.

If you have ufw enabled, you'll need to allow traffic on this interface:

sudo ufw allow in on cni0 && sudo ufw allow out on cni0
My pods can't reach the internet or each other (but my MicroK8s host machine can)...

Make sure packets to/from the pod network interface can be forwarded to/from the default interface on the host via the iptables tool. Such changes can be made persistent by installing the iptables-persistent package:

   sudo iptables -P FORWARD ACCEPT
   sudo apt-get install iptables-persistent

or, if using ufw:

   sudo ufw default allow routed

The MicroK8s inspect command can be used to check the firewall configuration:

   microk8s inspect

A warning will be shown if the firewall is not forwarding traffic.

My log collector is not collecting any logs...

By default container logs are located in /var/log/pods/{id}. You have to mount this location in your log collector for that to work. Following is an example diff for fluent-bit:

@@ -36,6 +36,9 @@
            - name: varlibdockercontainers
              mountPath: /var/lib/docker/containers
              readOnly: true
   +        - name: varlibdockercontainers
   +          mountPath: /var/snap/microk8s/common/var/lib/containerd/
   +          readOnly: true
            - name: fluent-bit-config
              mountPath: /fluent-bit/etc/
          terminationGracePeriodSeconds: 10
   @@ -45,7 +48,7 @@
              path: /var/log
          - name: varlibdockercontainers
   -          path: /var/lib/docker/containers
   +          mountPath: /var/snap/microk8s/common/var/lib/containerd/
          - name: fluent-bit-config
              name: fluent-bit-config
My pods are not starting and I use ZFS...

Microk8s switched to containerd as its container runtime in release 492. When run on ZFS, containerd must be configured to use ZFS snapshots. Presently neither Microk8s nor containerd perform this automatically so you must manually update the configuration. Instructions on how to do this are documented here.

My home directory is not in /home or is on NFS and I can't get Microk8s to work...

While not strictly a Microk8s issue, snaps generally do not work out of the box if your home directory is mounted via NFS, or if it is not located directly under /home. See snapd bugs #1662552 and #1620771 for further information and possible workarounds.

Reporting a bug

If you cannot solve your issue and believe the fault may lie in MicroK8s, please file an issue on the project repository.

To help us deal effectively with issues, it is incredibly useful to include the report obtained from microk8s inspect, as well as any additional logs, and a summary of the issue.

Last updated 8 days ago.