Kubernetes cgroup driver misconfiguration

by Gasim   Last Updated January 12, 2018 23:00 PM

Default Docker installation in CentOS starts with systemd Cgroup. I installed Kubernetes from official YUM repo and systemd drop-in 10-kubeadm.conf has the following contents:

[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf"
Environment="KUBELET_SYSTEM_PODS_ARGS=--pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true"
Environment="KUBELET_NETWORK_ARGS=--network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin"
Environment="KUBELET_DNS_ARGS=--cluster-dns=10.96.0.10 --cluster-domain=cluster.local"
Environment="KUBELET_AUTHZ_ARGS=--authorization-mode=Webhook --client-ca-file=/etc/kubernetes/pki/ca.crt"
Environment="KUBELET_CADVISOR_ARGS=--cadvisor-port=0"
Environment="KUBELET_CGROUP_ARGS=--cgroup-driver=systemd"
Environment="KUBELET_CERTIFICATE_ARGS=--rotate-certificates=true --cert-dir=/var/lib/kubelet/pki"
ExecStart=
ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_SYSTEM_PODS_ARGS $KUBELET_NETWORK_ARGS $KUBELET_DNS_ARGS $KUBELET_AUTHZ_ARGS $KUBELET_CADVISOR_ARGS $KUBELET_CGROUP_ARGS $KUBELET_CERTIFICATE_ARGS $KUBELET_EXTRA_ARGS

I also tried to get the environmental variables to see if the systemd drop-in overrides properly (systemctl show --property=Environment kubelet | cat):

Environment=KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf\x20--kubeconfig=/etc/kubernetes/kubelet.conf KUBELET_SYSTEM_PODS_ARGS=--pod-manifest-path=/etc/kubernetes/manifests\x20--allow-privileged=true KUBELET_NETWORK_ARGS=--network-plugin=cni\x20--cni-conf-dir=/etc/cni/net.d\x20--cni-bin-dir=/opt/cni/bin KUBELET_DNS_ARGS=--cluster-dns=10.96.0.10\x20--cluster-domain=cluster.local KUBELET_AUTHZ_ARGS=--authorization-mode=Webhook\x20--client-ca-file=/etc/kubernetes/pki/ca.crt KUBELET_CADVISOR_ARGS=--cadvisor-port=0 KUBELET_CGROUP_ARGS=--cgroup-driver=systemd KUBELET_CERTIFICATE_ARGS=--rotate-certificates=true\x20--cert-dir=/var/lib/kubelet/pki

When I initialize the cluster using kubeadm init --apiserver-advertise-address=X.X.X.X --pod-network-cidr=10.244.0.0/16, the process is successful. However, running kubelet version gives me a Cgroup misconfiguration error:

error: failed to run Kubelet: failed to create kubelet: misconfiguration: kubelet cgroup driver: "cgroupfs" is different from docker cgroup driver: "systemd"

Versions

  • CentOS Linux release 7.4.1708 (Core)
  • Kubernetes Version (kubectl version): 1.9 (GitVersion: 1.9.1; Go Version: go1.9.2)
  • Docker version (docker version): 1.12.6

What is the right solution for this problem? I tried to change Cgroup of Kubernetes to cgroupfs and kubeadm kept giving me "kubelet unresponsive" error. How can I make Kubernetes to accept CGroup parameter given to it?



Related Questions


cgroup throttle on child device does not work

Updated May 03, 2017 20:00 PM

How do I set cgroup limits for systemd user slices?

Updated November 29, 2017 21:00 PM

Centos 7 & Hard CPU Limits

Updated April 24, 2015 06:00 AM

Enable cgroups on Beowulf compute nodes

Updated August 28, 2017 21:00 PM