Step 1: Update Kubelet-Config ConfigMap

A cluster created with Kubeadm will have a ConfigMap named kubelet-config in Namespace kube-system. This ConfigMap will be used if new Nodes are added to the cluster.

➜ ssh cks8930 

➜ candidate@cks8930:~# k -n kube-system edit cm kubelet-config
# kubectl -n kube-system edit cm kubelet-config
apiVersion: v1
data:
  kubelet: |
    apiVersion: kubelet.config.k8s.io/v1beta1
    kind: KubeletConfiguration
    ...
    volumeStatsAggPeriod: 0s
    containerLogMaxSize: 5Mi
    containerLogMaxFiles: 3
kind: ConfigMap
metadata:
  name: kubelet-config
  namespace: kube-system
...

Above we can see that we simply added the two new arguments to data.kubelet.

A new Node added to the cluster, both control plane and worker, would use this KubeletConfiguration containing the changes. That KubeletConfiguration from the ConfigMap will also be used during a kubeadm upgrade.

Step 2: Update Control Plane Kubelet-Config

To find the Kubelet-Config path we can check the Kubelet process:

➜ candidate@cks8930:~# sudo -i

➜ root@cks8930:~# ps aux | grep kubelet
root        7418  2.0  4.8 1927756 98748 ?       Ssl  11:38   1:56 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml
...

Above we see it's specified via the argument --config=/var/lib/kubelet/config.yaml. We could also check the Kubeadm config for the Kubelet:

➜ root@cks8930:~# find / | grep kubeadm
/var/lib/dpkg/info/kubeadm.md5sums
/var/lib/dpkg/info/kubeadm.list
/var/lib/kubelet/kubeadm-flags.env
/usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf
...

➜ root@cks8930:~# cat /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf
# Note: This dropin only works with kubeadm and kubelet v1.11+
[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf"
Environment="KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml"
...

Above we see the argument --config being set. And we should see that our changes are still missing in that file:

➜ root@cks8930:~# grep containerLog /var/lib/kubelet/config.yaml

➜ root@cks8930:~#

We go ahead and download the latest Kubelet-Config:

➜ root@cks8930:~# kubeadm upgrade node phase kubelet-config
[upgrade] Reading configuration from the cluster...
[upgrade] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[upgrade] Backing up kubelet config file to /etc/kubernetes/tmp/kubeadm-kubelet-config1186317096/config.yaml
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[upgrade] The configuration for this node was successfully updated!
[upgrade] Now you should go ahead and upgrade the kubelet package using your package manager.

➜ root@cks8930:~# grep containerLog /var/lib/kubelet/config.yaml
containerLogMaxFiles: 3
containerLogMaxSize: 5Mi

Now we just need to restart the Kubelet:

➜ root@cks8930:~# service kubelet restart

We can verify the changes:

➜ root@cks8930:~# kubectl get --raw "/api/v1/nodes/cks8930/proxy/configz" | jq
...
    "containerLogMaxSize": "5Mi",
    "containerLogMaxFiles": 3,
...

Step 3: Update Worker Node Kubelet-Config

We should see that the existing Kubelet-Config on the worker node is still unchanged:

➜ root@cks8930:~# ssh cks8930-node1

➜ root@cks8930-node1:~# grep containerLog /var/lib/kubelet/config.yaml

➜ root@cks8930-node1:~#

So we go ahead and apply the updates:

➜ root@cks8930-node1:~# kubeadm upgrade node phase kubelet-config
[upgrade] Reading configuration from the cluster...
[upgrade] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[upgrade] Backing up kubelet config file to /etc/kubernetes/tmp/kubeadm-kubelet-config948054586/config.yaml
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[upgrade] The configuration for this node was successfully updated!
[upgrade] Now you should go ahead and upgrade the kubelet package using your package manager.

➜ root@cks8930-node1:~# grep containerLog /var/lib/kubelet/config.yaml
containerLogMaxFiles: 3
containerLogMaxSize: 5Mi

➜ root@cks8930-node1:~# service kubelet restart

And verify the changes:

➜ root@cks8930-node1:~# kubectl get --raw "/api/v1/nodes/cks8930-node1/proxy/configz" | jq
...
    "containerLogMaxSize": "5Mi",
    "containerLogMaxFiles": 3,
...