NOTE: Launch configurations are available starting from MicroK8s 1.27 and newer
Launch configurations schema
The schema is defined in schema.go. For a full example config file, see full.yaml
Launch configurations version 0.1.0
---
# 'version' is the semantic version of the configuration file format.
version: 0.1.0
# 'persistentClusterToken' is a persistent token that may be used to authenticate join requests made to the local node.
# Persistent token length should be 32 characters. Other nodes will be able to join using the following command:
# $ microk8s join $THIS_NODE_IP:25000/a74cddf30d2408d49fcd748a26021c6a
persistentClusterToken: "a74cddf30d2408d49fcd748a26021c6a"
# 'join' is configuration to join the local node to an already existing MicroK8s cluster.
join:
url: "10.10.10.10:25000/a74cddf30d2408d49fcd748a26021c6a" # 'url' is the cluster join URL
worker: true # set 'worker' to true to join as a worker node
# 'extraSANs' is a list of extra Subject Alternate Names to add to the local API server.
extraSANs:
- 10.10.10.10
- microk8s.example.com
# 'extraKubeAPIServerArgs' is extra arguments to add to the local node kube-apiserver.
# Set a value to null to remove it from the arguments.
extraKubeAPIServerArgs:
--authorization-mode: RBAC,Node
--event-ttl: null
# 'extraKubeletServerArgs' is extra arguments to add to the local node kubelet.
# Set a value to null to remove it from the arguments.
extraKubeletArgs:
--cluster-dns: 10.152.183.10
# 'extraKubeProxyArgs' is extra arguments to add to the local node kube-proxy.
# Set a value to null to remove it from the arguments.
extraKubeProxyArgs:
--cluster-cidr: 10.1.0.0/16
# 'extraKubeControllerManagerArgs' is extra arguments to add to the local node kube-controller-manager.
# Set a value to null to remove it from the arguments.
extraKubeControllerManagerArgs:
--leader-elect-lease-duration: 30s
--leader-elect-renew-deadline: 15s
# 'extraKubeSchedulerArgs' is extra arguments to add to the local node kube-scheduler.
# Set a value to null to remove it from the arguments.
extraKubeSchedulerArgs:
--leader-elect-lease-duration: 30s
--leader-elect-renew-deadline: 15s
# 'extraContainerdArgs' is extra arguments to add to the local node containerd service.
# Set a value to null to remove it from the arguments.
extraContainerdArgs:
-l: debug
# 'extraContainerdEnv' is extra environment variables (e.g. proxy configuration) for the local node containerd service.
# Set a value to null to remove it from the environment.
extraContainerdEnv:
http_proxy: http://squid.internal:3128
https_proxy: http://squid.internal:3128
# 'extraDqliteArgs' is extra arguments for the local node k8s-dqlite service.
# Set a value to null to remove it from the arguments.
extraDqliteArgs:
--disk-mode: "true"
# 'extraDqliteEnv' is extra environment variables for the local node k8s-dqlite service.
# Set a value to null to remove it from the environment.
extraDqliteEnv:
LIBRAFT_TRACE: "1"
LIBDQLITE_TRACE: "1"
# 'extraMicroK8sClusterAgentArgs' is extra arguments for the local node cluster-agent service.
extraMicroK8sClusterAgentArgs:
--min-tls-version: tls13
# 'extraMicroK8sClusterAgentEnv' is extra environment variables for the local node cluster-agent service.
extraMicroK8sClusterAgentEnv:
GOFIPS: "1"
# 'extraMicroK8sAPIServerProxyArgs' is extra arguments for the local node apiserver-proxy service (kube-apiserver proxy for worker nodes).
extraMicroK8sAPIServerProxyArgs:
--refresh-interval: "0"
# 'extraMicroK8sAPIServerProxyEnv' is extra environment variables for the local node apiserver-proxy service (kube-apiserver proxy for worker nodes).
extraMicroK8sAPIServerProxyEnv:
GOFIPS: "1"
# 'extraEtcdArgs' is extra arguments to add to the local node etcd service.
# Set a value to null to remove it from the arguments.
extraEtcdArgs:
--log-level: debug
--enable-pprof: "true"
# 'extraEtcdEnv' is extra environment variables (e.g. proxy configuration) for the local node etcd service.
# Set a value to null to remove it from the environment.
extraEtcdEnv:
GOFIPS: "1"
# 'extraFlanneldArgs' is extra arguments to add to the local node flanneld service.
# Set a value to null to remove it from the arguments.
extraFlanneldArgs:
--kube-subnet-mgr: "true"
--kubeconfig-file: $SNAP_DATA/credentials/kubelet.config
--etcd-endpoints: null
--etcd-cafile: null
--etcd-certfile: null
--etcd-keyfile: null
# 'extraFlanneldEnv' is extra environment variables (e.g. proxy configuration) for the local node flanneld service.
# Set a value to null to remove it from the environment.
extraFlanneldEnv:
GOFIPS: "1"
# 'extraConfigFiles' is extra service configuration files to create (e.g. for configuring kube-apiserver encryption at rest).
# These files will be written at $SNAP_DATA/args/<filename>.
extraConfigFiles:
flannel-network-mgr-config: |
{"Network": "10.1.0.0/16", "Backend": {"Type": "vxlan"}}
# 'addonRepositories' is a list of addon repositories to configure on the local node.
addonRepositories:
- name: github-repo
url: https://github.com/org/repo
reference: master # 'reference' is branch or tag to check out instead of the default
- name: local-repo
url: /path/to/local/repo # 'url' can be a local directory
# 'addons' is a list of addons to enable or disable.
addons:
- name: dns # 'name' is the name of the addon to enable.
- name: gpu
args: [--version=v22.9.1] # 'args' is extra arguments that will be passed to the enable script.
# refer to the documentation of each addon for more information.
- name: registry
disable: true # 'disable' should be set when the addon must be disabled instead.
# 'containerdRegistryConfigs' is used to configure registry mirrors. the key is the repository name (e.g. "docker.io")
# and the value will be written to the respective hosts.toml file
containerdRegistryConfigs:
# Use `http://my.proxy:5000` as a DockerHub registry mirror.
docker.io: |
[host."http://my.proxy:5000"]
capabilities = ["pull", "resolve"]
Examples
The following examples present launch configurations for common tasks. The examples are meant to be composable, you can mix and match as many of these configs as you want to suit your needs:
Enable DNS
Enable DNS addon, use host resolv.conf for upstream nameservers or fallback to 1.1.1.1.
---
version: 0.1.0
addons:
- name: dns
# These arguments will be set by the 'dns' addon. We set them manually to avoid
# unnecessary Kubernetes restarts while the cluster is bootstrapping.
extraKubeletArgs:
--cluster-dns: 10.152.183.10
--cluster-domain: cluster.local
Enable DNS, RBAC, Metrics-Server, Dashboard, Hostpath Storage, Ingress and Cert-Manager
---
version: 0.1.0
addons:
- name: dns
- name: rbac
- name: metrics-server
- name: dashboard
- name: hostpath-storage
- name: ingress
- name: cert-manager
# These arguments will be set by the 'dns' and 'rbac' addons. We set them manually to avoid
# unnecessary Kubernetes restarts while the cluster is bootstrapping.
extraKubeletArgs:
--cluster-dns: 10.152.183.10
--cluster-domain: cluster.local
extraKubeAPIServerArgs:
--authorization-mode: RBAC,Node
Enable DNS with specific nameserver
Enable DNS addon and use a specific upstream nameserver (by specifying it as argument to the DNS addon):
---
version: 0.1.0
addons:
- name: dns
args: [1.1.1.1, 8.8.8.8]
Configure private registry mirrors
Configure private registry mirrors for docker.io and registry.k8s.io. Requires that you have manually pushed required images to the registry mirrors, or that you have set them up as a pull-through cache:
---
version: 0.1.0
addons:
- name: dns
containerdRegistryConfigs:
docker.io: |
[host."http://dockerhub.internal.mirror:15050"]
capabilities = ["pull", "resolve"]
registry.k8s.io: |
[host."http://registryk8sio.internal.mirror:15050"]
capabilities = ["pull", "resolve"]
Configure proxy for access to image registries
Containerd will use this proxy when pulling images from the upstream repositories:
---
version: 0.1.0
addons:
- name: dns
extraContainerdEnv:
http_proxy: http://squid.internal:3128
https_proxy: http://squid.internal:3128
no_proxy: 10.0.0.0/8,127.0.0.1,192.168.0.0/16,172.16.0.0/12
Configure Subject Alternate Names for kube-apiserver certificates
Ensure the kube-apiserver can be accessed using the hostname k8s.cluster1.infra1.pod.
---
version: 0.1.0
extraSANs:
- k8s.cluster1.infra1.prod
Configure Kubernetes services
Decrease lease times for kube-scheduler and kube-controller-manager from their default values. Also, tell kube-apiserver to prefer InternalIP when talking to the kubelets:
---
version: 0.1.0
extraKubeAPIServerArgs:
--kubelet-preferred-address-types: InternalIP,Hostname,InternalDNS,ExternalDNS,ExternalIP
extraKubeControllerManagerArgs:
--leader-elect-lease-duration: 30s
--leader-elect-renew-deadline: 15s
extraKubeSchedulerArgs:
--leader-elect-lease-duration: 30s
--leader-elect-renew-deadline: 15s
Configure host interfaces
Configure host interfaces used by Kubernetes services. In this example, the host uses 10.0.1.10 for control plane traffic, and 10.0.2.10 for NodePort services.
---
version: 0.1.0
extraKubeAPIServerArgs:
--advertise-address: 10.0.1.10
extraKubeletArgs:
--node-ip: 10.0.1.10
extraKubeProxyArgs:
--nodeport-addresses: 10.0.2.10
Custom addon repositories
Configure custom addon repositories and enable custom addons:
---
version: 0.1.0
addonRepositories:
- name: custom
url: https://github.com/neoaggelos/microk8s-addons-extras
addons:
- name: custom/clusterapi
Configure secrets encryption at rest
Create an extra config file with the encryption at rest config and pass it to kube-apiserver:
---
version: 0.1.0
extraKubeAPIServerArgs:
--encryption-provider-config: "$SNAP_DATA/args/encryption-config.yaml"
extraConfigFiles:
encryption-config.yaml: |
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources:
- secrets
- configmaps
providers:
- aescbc:
keys:
- name: key1
secret: GneEFCl9TpamK4CcxRJcHFgUuFUnfTTnSW/LzwWRY5g= # use 'openssl rand -base64 32'
- identity: {}
For more details on the required configurations, see Encrypting Confidential Data at Rest | Kubernetes
Configure auditing policies
Create an extra config file with the auditing policy configuration and pass it to kube-apiserver:
---
version: 0.1.0
extraKubeAPIServerArgs:
--audit-policy-file: "$SNAP_DATA/args/audit-policy.yaml"
--audit-log-path: "-"
extraConfigFiles:
audit-policy.yaml: |
apiVersion: audit.k8s.io/v1
kind: Policy
omitStages:
- "RequestReceived"
rules:
- level: RequestResponse
resources:
- group: ""
resources: ["configmaps"]
- level: Metadata
resources:
- group: ""
resources: ["pods/log", "pods/status"]
For more details on the required configurations, see Auditing | Kubernetes
Configure clustering
Form a multi-node cluster that automatically provisions itself after installing. This requires separate configs for the init node and the rest of the cluster nodes:
On the cluster init node, configure a persistent token:
# configuration for init node of cluster
---
version: 0.1.0
persistentClusterToken: 3f6380c7c2950e74128ff037878b4925
On the rest of the nodes, specify the join string using the persistent token. This is identical to running microk8s join $join_url:
# configuration for other control plane nodes
---
version: 0.1.0
join:
url: $FIRST_NODE_IP:25000/3f6380c7c2950e74128ff037878b4925
# configuration for other worker nodes
---
version: 0.1.0
join:
url: $FIRST_NODE_IP:25000/3f6380c7c2950e74128ff037878b4925
worker: true
3-node HA cluster with KubeOVN
On the cluster init node, enable kube-ovn and configure a cluster token:
---
version: 0.1.0
persistentClusterToken: 3f6380c7c2950e74128ff037878b4925
addons:
- name: dns
- name: kube-ovn
args: [--force]
On the other two nodes, use a multi-part configuration to enable kube-ovn and then join the cluster:
---
version: 0.1.0
addons:
- name: kube-ovn
args: [--force]
---
version: 0.1.0
join:
url: $FIRST_NODE_IP:25000/3f6380c7c2950e74128ff037878b4925
Specify a different service CIDR (before MicroK8s 1.28)
By default, MicroK8s uses the CIDR 10.152.183.0/24 for ClusterIP services. You may change it to 10.200.0.0/16 using the following config:
---
version: 0.1.0
extraKubeAPIServerArgs:
--cluster-service-ip-range: 10.200.0.0/16
extraKubeletArgs:
--cluster-dns: 10.200.0.10
--cluster-domain: cluster.local
extraSANs:
- 10.200.0.1
addons:
- name: dns
args: [/etc/resolv.conf, 10.200.0.10]
Kubernetes will use the first IP address of the specified range as the kubernetes service address, so we include it in the certificate Subject Alternate Names. Further, we set the ClusterIP address for the DNS service.
Specify a custom pod and service CIDR (requires MicroK8s 1.28 or newer)
Use 10.2.0.0/16 for pods and 10.94.0.0/24 for services. Make sure to add the 10.94.0.1 as an extra SAN as this address will be used by the kube-apiserver. The DNS addon will automatically use a valid service IP for the coredns service.
---
version: 0.2.0
extraCNIEnv:
IPv4_CLUSTER_CIDR: "10.2.0.0/16"
IPv4_SERVICE_CIDR: "10.94.0.0/24"
extraSANs:
- 10.94.0.1
addons:
- name: dns
Specify kubelet arguments
Configure kubelet for CPU isolation.
---
version: 0.1.0
extraKubeletArgs:
--feature-gates: "CPUManager=true,TopologyManager=true"
--system-reserved: "cpu=500m"
--kube-reserved: "cpu=500m"
--cpu-manager-policy: "static"
--reserved-cpus: "1,2"