MicroK8s - Add a Windows worker node to MicroK8s

This guide explains how to add Windows worker nodes to MicroK8s. This how-to guide explains how to join a Windows Server 2022 to an existing MicroK8s cluster running Calico.

Note: Currently Windows nodes may be affected by a bug in the Windows container code for Server 2022/July update. Please see this issue for more details and a workaround:Container networking on Kubernetes broken after Server 2022 July 2024 / KB5040437 (OS Build 20348.2582) update · Issue #516 · microsoft/Windows-Containers · GitHub

Requirements

A node running Windows (tested with Windows Server 2022).
A MicroK8s cluster using the Calico CNI (enabled by default since 1.19).

Prepare the MicroK8s cluster

Determine the exact version of Kubernetes running in the cluster, e.g. 1.27.1. You can use the following command:
```
microk8s kubectl get node -o wide
```
Determine the exact version of Calico running in the MicroK8s cluster, e.g. 3.25.0. For this, you can inspect the image used by the calico-node containers:
```
microk8s kubectl get ds/calico-node -n kube-system -o jsonpath='{.spec.template.spec.containers[?(.name=="calico-node")].image}{"\n"}'
```
Generate a kubeconfig file for the MicroK8s cluster. You will need this to run calicoctl commands, and later copy it to the Windows node.
```
mkdir -p ~/.kube
microk8s config > ~/.kube/config
```
In order for Windows pods to schedule, strict affinity must be set to true. This is required to prevent Linux nodes from borrowing IP addresses from Windows nodes. This can be set with the calicoctl binary. Install the calicoctl binary with:
```
CALICO_VERSION="3.25.0"
curl -L https://github.com/projectcalico/calico/releases/download/v$CALICO_VERSION/calicoctl-linux-amd64 -o calicoctl
chmod +x ./calicoctl
```
(set the environment variable to the appropriate version of Calico for your system)
Then, set strict affinity to true with the following command:
```
./calicoctl ipam configure --strictaffinity=true --allow-version-mismatch
```

Prepare the Windows node

All the commands shown next have to be run in a PowerShell window as Administrator.

Install the containerd container runtime. The machine might restart during this operation.

Invoke-WebRequest -UseBasicParsing "https://raw.githubusercontent.com/microsoft/Windows-Containers/Main/helpful_tools/Install-ContainerdRuntime/install-containerd-runtime.ps1" -o install-containerd-runtime.ps1

.\install-containerd-runtime.ps1

Create directory c:\k, create the kubeconfig file c:\k\config. After creating, open the file with notepad, paste the contents of the kubeconfig file of the MicroK8s cluster and save.
```
mkdir c:\k
New-Item c:\k\config

notepad c:\k\config
```

Install Calico

Retrieve the install-calico-windows.ps1 script from the Calico GitHub releases page. NOTE: do not worry about the calico version in the URL, we will pick the Calico version to install later.
```
Invoke-WebRequest -Uri https://github.com/projectcalico/calico/releases/download/v3.25.1/install-calico-windows.ps1 -OutFile c:\k\install-calico-windows.ps1
```

Download Calico and Kubernetes binaries using the following command (replace 1.27.1 with the Kubernetes version and 3.25.0 with the Calico version running in the cluster):

c:\k\install-calico-windows.ps1 -ReleaseBaseURL "https://github.com/projectcalico/calico/releases/download/v3.25.0" -ReleaseFile "calico-windows-v3.25.0.zip" -KubeVersion "1.27.1" -DownloadOnly "yes" -ServiceCidr "10.152.183.0/24" -DNSServerIPs "10.152.183.10"

Configure the CNI bin and configuration directories and then install the Calico services. If the vSwitch is not yet created, this will temporarily affect network connectivity for a few seconds.

$ENV:CNI_BIN_DIR="c:\program files\containerd\cni\bin"
$ENV:CNI_CONF_DIR="c:\program files\containerd\cni\conf"
c:\calicowindows\install-calico.ps1
c:\calicowindows\start-calico.ps1

If successful, the output should look like this:

Starting Calico...
This may take several seconds if the vSwitch needs to be created.
Waiting for Calico initialisation to finish...
Waiting for Calico initialisation to finish...StoredLastBootTime , CurrentLastBootTime 5/21/2023 8:21:24 AM
Waiting for Calico initialisation to finish...StoredLastBootTime , CurrentLastBootTime 5/21/2023 8:21:24 AM
Calico initialisation finished.
Done, the Calico services are running:

Status   Name               DisplayName
------   ----               -----------
Running  CalicoFelix        Calico Windows Agent
Running  CalicoNode         Calico Windows Startup

Install Kubernetes services (kubelet and kube-proxy):

c:\calicowindows\kubernetes\install-kube-services.ps1

Add a firewall rule for incoming connections to the Windows kubelet node service. This is for kubectl logs and kubectl exec commands to work with pods running in Windows nodes:
```
New-NetFirewallRule -Name 'Kubelet-In-TCP' -DisplayName 'Kubelet (node)' -Enabled True -Direction Inbound -Protocol TCP -Action Allow -LocalPort 10250
```

Configure the network adapter name expected by the kubelet service.

Run ipconfig to retrieve the IP configuration of the machine’s network adapters. An example output is shown below, where the network adapter name is vEthernet (tapeb67a82f-a1):

Windows IP Configuration


Ethernet adapter vEthernet (tapeb67a82f-a1):

Connection-specific DNS Suffix  . : teststack.internal
Link-local IPv6 Address . . . . . : fe80::d17b:a2e:c02c:1163%8
IPv4 Address. . . . . . . . . . . : 172.16.1.177
Subnet Mask . . . . . . . . . . . : 255.255.255.0
Default Gateway . . . . . . . . . : 172.16.1.1

Then, open c:\calicowindows\kubernetes\kubelet-service.ps1 with notepad and edit the default value of the InterfaceName parameter to match the one above.

# c:\calicowindows\kubernetes\kubelet-service.ps1
Param(
    [string]$NodeIp="",
    [string]$InterfaceName="vEthernet (tapeb67a82f-a1)"
)

Save the file and exit notepad.

Clean service arguments for kubelet. Open c:\calicowindows\kubernetes\kubelet-service.ps1 with notepad and ensure that:
- For MicroK8s version 1.26 or newer, remove the argument --logtostderr=true
- For MicroK8s version 1.27 or newer, remove the argument --container-runtime=remote
Save the file and exit notepad.

Start Kubernetes services

Start-Service kubelet
Start-Service kube-proxy

Verify the Windows node

Ensure that the Windows node appears in the output of microk8s kubectl get nodes -o wide, for example:

NAME   STATUS   ROLES    AGE    VERSION   INTERNAL-IP    EXTERNAL-IP   OS-IMAGE                                    KERNEL-VERSION      CONTAINER-RUNTIME
u1     Ready    <none>   3h3m   v1.27.1   172.16.1.49    <none>        Ubuntu 20.04.5 LTS                          5.4.0-139-generic   containerd://1.6.15
w1     Ready    <none>   105m   v1.27.1   172.16.1.177   <none>        Windows Server 2022 Datacenter Evaluation   10.0.20348.587      containerd://1.6.6

Start a nanoserver pod on the Windows node and wait for it to come up.
```
microk8s kubectl run test-pod -it --image mcr.microsoft.com/windows/nanoserver:ltsc2022 --overrides '{"spec": {"nodeSelector": {"kubernetes.io/os": "windows"}}}'
```
The default timeout for the pod to come up is 60 seconds, which might not be enough for the nanoserver image to be fetched and the pod to be started. In that case, simply wait for the pod to start and then attach using microk8s kubectl attach -it test-pod

Ensure that you can curl the kubernetes service, e.g. with curl https://kubernetes --insecure

Microsoft Windows [Version 10.0.20348.1726]
(c) Microsoft Corporation. All rights reserved.

C:\>curl https://kubernetes --insecure
{
"kind": "Status",
"apiVersion": "v1",
"metadata": {},
"status": "Failure",
"message": "Unauthorized",
"reason": "Unauthorized",
"code": 401
}

Uninstall the Windows worker node

On the Windows machine:

Stop and uninstall Kubernetes services:

c:\calicowindows\kubernetes\uninstall-kube-services.ps1

Stop and uninstall Calico services:
```
c:\calicowindows\uninstall-calico.ps1
```

On the MicroK8s machine:

Remove the node from MicroK8s (replace w1 with the name of the Windows node):
```
microk8s kubectl delete node w1
```

There is no specific upgrade path for a Windows node. If you have upgraded MicroK8s to a later version, and wish the Windows node to remain compatible, you should first uninstall the node components (see above), then re-install the newer version.

Add a Windows worker node to MicroK8s