Troubleshooting

It is important to recognise that things can go wrong. But MicroK8s gives you tools to help work out what has gone wrong, as detailed below. Be sure to check out the common issues section for help resolving the most frequently encountered problems.

Checking logs

If a pod is not behaving as expected, the first port of call should be the logs.

First determine the resource identifier for the pod:

microk8s.kubectl get pods

This will list the currently available pods, for example:

NAME                                 READY   STATUS             RESTARTS   AGE
mk8s-redis-7647889b6d-vjwqm          1/1     Running            0          2m24s

You can then use kubectl to view the log. For example, for the simple redis pod above:

microk8s.kubectl logs mk8s-redis-7647889b6d-vjwqm

Examining the configuration

If the problem you are experiencing indicates a problem with the configuration of the Kubernetes components themselves, it could be helpful to examine the arguments used to run these components. These are available in the directory ${SNAP_DATA}/args, which on Ubuntu should point to /var/snap/microk8s/current. Note that the $SNAP_DATA environment variable itself is only available to the running snap. For more information on the snap environment, check the snap documentation.

Using the built-in inspection tool

MicroK8s ships with a script to compile a complete report on MicroK8s and the system which it is running on. This is essential for bug reports, but is also a useful way of confirming the system is (or isn’t) working and collecting all the relevant data in one place.

To run the inspection tool, enter the command (admin privilege is required to collect all the data):

sudo microk8s.inspect

You should see output similar to the following:

Inspecting services
  Service snap.microk8s.daemon-cluster-agent is running
  Service snap.microk8s.daemon-flanneld is running
  Service snap.microk8s.daemon-containerd is running
  Service snap.microk8s.daemon-apiserver is running
  Service snap.microk8s.daemon-apiserver-kicker is running
  Service snap.microk8s.daemon-proxy is running
  Service snap.microk8s.daemon-kubelet is running
  Service snap.microk8s.daemon-scheduler is running
  Service snap.microk8s.daemon-controller-manager is running
  Service snap.microk8s.daemon-etcd is running
  Copy service arguments to the final report tarball
Inspecting AppArmor configuration
Gathering system information
  Copy processes list to the final report tarball
  Copy snap list to the final report tarball
  Copy VM name (or none) to the final report tarball
  Copy disk usage information to the final report tarball
  Copy memory usage information to the final report tarball
  Copy server uptime to the final report tarball
  Copy current linux distribution to the final report tarball
  Copy openSSL information to the final report tarball
  Copy network configuration to the final report tarball
Inspecting kubernetes cluster
  Inspect kubernetes cluster

Building the report tarball
  Report tarball is at /var/snap/microk8s/1031/inspection-report-20191104_153950.tar.gz

This confirms the services that are running, and the resultant report file can be viewed to get a detailed look at every aspect of the system.

Common issues (and their solutions!)

My dns and dashboard pods are CrashLooping...

The cni network plugin used by MicroK8s creates a cni0 interface (cbr0 on pre v1.16 releases) when the first pod is created.

If you have ufw enabled, you'll need to allow traffic on this interface:

sudo ufw allow in on cni0 && sudo ufw allow out on cni0
My pods can't reach the internet or each other (but my MicroK8s host machine can)...

Make sure packets to/from the pod network interface can be forwarded to/from the default interface on the host via the iptables tool. Such changes can be made persistent by installing the iptables-persistent package:

   sudo iptables -P FORWARD ACCEPT
      sudo apt-get install iptables-persistent
   

or, if using ufw:

   sudo ufw default allow routed
   

The MicroK8s inspect command can be used to check the firewall configuration:

   microk8s.inspect
   

A warning will be shown if the firewall is not forwarding traffic.

My log collector is not collecting any logs...

By default container logs are located in /var/log/pods/{id}. You have to mount this location in your log collector for that to work. Following is an example diff for fluent-bit:

@@ -36,6 +36,9 @@
            - name: varlibdockercontainers
              mountPath: /var/lib/docker/containers
              readOnly: true
   +        - name: varlibdockercontainers
   +          mountPath: /var/snap/microk8s/common/var/lib/containerd/
   +          readOnly: true
            - name: fluent-bit-config
              mountPath: /fluent-bit/etc/
          terminationGracePeriodSeconds: 10
   @@ -45,7 +48,7 @@
              path: /var/log
          - name: varlibdockercontainers
            hostPath:
   -          path: /var/lib/docker/containers
   +          mountPath: /var/snap/microk8s/common/var/lib/containerd/
          - name: fluent-bit-config
            configMap:
              name: fluent-bit-config
   

Reporting a bug

If you cannot solve your issue and believe the fault may lie in MicroK8s, please file an issue on the project repository.

To help us deal effectively with issues, it is incredibly useful to include the report obtained from microk8s.inspect, as well as any additional logs, and a summary of the issue.

We appreciate your feedback on the docs. You can edit this page or file a bug here.