前言

去年12月份，当Kubernetes社区宣布1.20版本之后会逐步弃用dockershim，当时也有很多自媒体在宣传Kubernetes弃用Docker。其实，我觉得这是一种误导，也许仅仅是为了蹭热度。

dockershim是Kubernetes的一个组件，其作用是为了操作Docker。Docker是在2013年面世的，而Kubernetes是在2016年，所以Docker刚开始并没有想到编排，也不会知道会出现Kubernetes这个庞然大物（它要是知道，也不会败的那么快…）。但是Kubernetes在创建的时候就是以Docker作为容器运行时，很多操作逻辑都是针对的Docker，随着社区越来越健壮，为了兼容更多的容器运行时，才将Docker的相关逻辑独立出来组成了dockershim。

正因为这样，只要Kubernetes的任何变动或者Docker的任何变动，都必须维护dockershim，这样才能保证足够的支持，但是通过dockershim操作Docker，其本质还是操作Docker的底层运行时Containerd，而且Containerd自身也是支持CRI（Container Runtime Interface），那为什么还要绕一层Docker呢？是不是可以直接通过CRI和Containerd进行交互？这也是社区希望启动dockershim的原因之一吧。

再来看看启动dockershim究竟对用户、对维护者有多少影响。

对上层用户来说，其实并没有影响，因为上层已经屏蔽调了这些细节，只管用就可以了。更多的影响只是针对我们这些YAML工程师，因为我们主要是考虑用哪个容器运行时，如果继续用Docker，以后版本升级有没有影响？如果不用Docker，维护的成本、复杂度、学习成本会不会增加？其实我们是想多了，事情也远没那么复杂，喜欢用docker的依旧可以用docker，想用containerd的就用containerd，改动也不大，后面也有相关的部署文档。而且也只是kubernetes社区不再维护dockershim而已，Mirantis 和 Docker 已经决定之后共同合作维护 dockershim 组件，也就是说dockershim依然可以作为连接docker的桥梁，只是从kubernetes内置携带改成独立的而已。

那什么是containerd呢？

Containerd是从Docker中分离的一个项目，旨在为Kubernetes提供容器运行时，负责管理镜像和容器的生命周期。不过Containerd是可以抛开Docker独立工作的。它的特性如下：

  • 支持OCI镜像规范，也就是runc
  • 支持OCI运行时规范
  • 支持镜像的pull
  • 支持容器网络管理
  • 存储支持多租户
  • 支持容器运行时和容器的生命周期管理
  • 支持管理网络名称空间

Containerd和Docker在命令使用上的一些区别主要如下：

功能 Docker Containerd
显示本地镜像列表 docker images crictl images
下载镜像 docker pull crictl pull
上传镜像 docker push 无
删除本地镜像 docker rmi crictl rmi
查看镜像详情 docker inspect IMAGE-ID crictl inspecti IMAGE-ID
显示容器列表 docker ps crictl ps
创建容器 docker create crictl create
启动容器 docker start crictl start
停止容器 docker stop crictl stop
删除容器 docker rm crictl rm
查看容器详情 docker inspect crictl inspect
attach docker attach crictl attach
exec docker exec crictl exec
logs docker logs crictl logs
stats docker stats crictl stats

可以看到使用方式大同小异。

下面介绍一下使用kubeadm安装K8S集群，并使用containerd作为容器运行时的具体安装步骤。

环境说明

主机节点

IP地址 系统 内核
192.168.0.5 CentOS7.6 3.10
192.168.0.125 CentOS7.6 3.10

软件说明

软件 版本
kubernetes 1.20.5
containerd 1.4.4

环境准备

（1）在每个节点上添加 hosts 信息：
$ cat /etc/hosts

  1. 192.168.0.5 k8s-master
  2. 192.168.0.125 k8s-node01

（2）禁用防火墙：

  1. $ systemctl stop firewalld
  2. $ systemctl disable firewalld

（3）禁用SELINUX：

  1. $ setenforce 0
  2. $ cat /etc/selinux/config
  3. SELINUX=disabled

（4）创建/etc/sysctl.d/k8s.conf文件，添加如下内容：

  1. net.bridge.bridge-nf-call-ip6tables = 1
  2. net.bridge.bridge-nf-call-iptables = 1
  3. net.ipv4.ip_forward = 1

（5）执行如下命令使修改生效：

  1. $ modprobe br_netfilter
  2. $ sysctl -p /etc/sysctl.d/k8s.conf

（6）安装 ipvs

  1. $ cat > /etc/sysconfig/modules/ipvs.modules <<EOF
  2. #!/bin/bash
  3. modprobe -- ip_vs
  4. modprobe -- ip_vs_rr
  5. modprobe -- ip_vs_wrr
  6. modprobe -- ip_vs_sh
  7. modprobe -- nf_conntrack_ipv4
  8. EOF
  9. $ chmod 755 /etc/sysconfig/modules/ipvs.modules && bash /etc/sysconfig/modules/ipvs.modules && lsmod | grep -e ip_vs -e nf_conntrack_ipv4

上面脚本创建了的/etc/sysconfig/modules/ipvs.modules文件，保证在节点重启后能自动加载所需模块。使用lsmod | grep -e ip_vs -e nf_conntrack_ipv4命令查看是否已经正确加载所需的内核模块。

（7）安装了 ipset 软件包：

  1. $ yum install ipset -y

为了便于查看 ipvs 的代理规则，最好安装一下管理工具 ipvsadm：

  1. $ yum install ipvsadm -y

（8）同步服务器时间

  1. $ yum install chrony -y
  2. $ systemctl enable chronyd
  3. $ systemctl start chronyd
  4. $ chronyc sources

（9）关闭 swap 分区：

  1. $ swapoff -a

（10）修改/etc/fstab文件，注释掉 SWAP 的自动挂载，使用free -m确认 swap 已经关闭。swappiness 参数调整，修改/etc/sysctl.d/k8s.conf添加下面一行：

  1. vm.swappiness=0

执行sysctl -p /etc/sysctl.d/k8s.conf使修改生效。

（11）接下来可以安装 Containerd

  1. $ yum install -y yum-utils \
  2.  device-mapper-persistent-data \
  3.  lvm2
  4. $ yum-config-manager \
  5.  --add-repo \
  6.  https://download.docker.com/linux/centos/docker-ce.repo
  7. $ yum list | grep containerd

可以选择安装一个版本，比如我们这里安装最新版本：

  1. $ yum install containerd.io-1.4.4 -y

（12）创建containerd配置文件：

  1. mkdir -p /etc/containerd
  2. containerd config default > /etc/containerd/config.toml
  3. # 替换配置文件
  4. sed -i "s#k8s.gcr.io#registry.cn-hangzhou.aliyuncs.com/google_containers#g"  /etc/containerd/config.toml
  5. sed -i '/containerd.runtimes.runc.options/a\ \ \ \ \ \ \ \ \ \ \ \ SystemdCgroup = true' /etc/containerd/config.toml
  6. sed -i "s#https://registry-1.docker.io#https://registry.cn-hangzhou.aliyuncs.com#g"  /etc/containerd/config.toml

（13）启动Containerd:

  1. systemctl daemon-reload
  2. systemctl enable containerd
  3. systemctl restart containerd

在确保 Containerd安装完成后，上面的相关环境配置也完成了，现在我们就可以来安装 Kubeadm 了，我们这里是通过指定yum 源的方式来进行安装，使用阿里云的源进行安装：

  1. cat <<EOF > /etc/yum.repos.d/kubernetes.repo
  2. [kubernetes]
  3. name=Kubernetes
  4. baseurl=http://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
  5. enabled=1
  6. gpgcheck=0
  7. repo_gpgcheck=0
  8. gpgkey=http://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg
  9.  http://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
  10. EOF

然后安装 kubeadm、kubelet、kubectl（我安装的是最新版，有版本要求自己设定版本）：

  1. $  yum install -y kubelet-1.20.5 kubeadm-1.20.5 kubectl-1.20.5

设置运行时：

  1. $ crictl config runtime-endpoint /run/containerd/containerd.sock

可以看到我们这里安装的是 v1.20.5版本，然后将 kubelet 设置成开机启动：

  1. $ systemctl daemon-reload
  2. $ systemctl enable kubelet && systemctl start kubelet

到这里为止上面所有的操作都需要在所有节点执行配置。

**

初始化集群

初始化Master

然后接下来在 master 节点配置 kubeadm 初始化文件，可以通过如下命令导出默认的初始化配置：

  1. $ kubeadm config print init-defaults > kubeadm.yaml

然后根据我们自己的需求修改配置，比如修改 imageRepository 的值，kube-proxy 的模式为 ipvs，需要注意的是由于我们使用的containerd作为运行时，所以在初始化节点的时候需要指定cgroupDriver为systemd

  1. apiVersion: kubeadm.k8s.io/v1beta2
  2. bootstrapTokens:
  3. - groups:
  4.   - system:bootstrappers:kubeadm:default-node-token
  5.   token: abcdef.0123456789abcdef
  6.   ttl: 24h0m0s
  7.   usages:
  8.   - signing
  9.   - authentication
  10. kind: InitConfiguration
  11. localAPIEndpoint:
  12.   advertiseAddress: 192.168.0.5 
  13.   bindPort: 6443
  14. nodeRegistration:
  15.   criSocket: /run/containerd/containerd.sock 
  16.   name: k8s-master
  17.   taints:
  18.   - effect: NoSchedule
  19.     key: node-role.kubernetes.io/master
  20. ---
  21. apiServer:
  22.   timeoutForControlPlane: 4m0s
  23. apiVersion: kubeadm.k8s.io/v1beta2
  24. certificatesDir: /etc/kubernetes/pki
  25. clusterName: kubernetes
  26. controllerManager: {}
  27. dns:
  28.   type: CoreDNS
  29. etcd:
  30.   local:
  31.     dataDir: /var/lib/etcd
  32. imageRepository: registry.cn-hangzhou.aliyuncs.com/google_containers
  33. kind: ClusterConfiguration
  34. kubernetesVersion: v1.20.5
  35. networking:
  36.   dnsDomain: cluster.local
  37.   podSubnet: 172.16.0.0/16
  38.   serviceSubnet: 10.96.0.0/12
  39. scheduler: {}
  40. ---
  41. apiVersion: kubeproxy.config.k8s.io/v1alpha1
  42. kind: KubeProxyConfiguration
  43. mode: ipvs
  44. ---
  45. apiVersion: kubelet.config.k8s.io/v1beta1
  46. kind: KubeletConfiguration
  47. cgroupDriver: systemd

然后使用上面的配置文件进行初始化：

  1. $ kubeadm init --config=kubeadm.yaml
  3. [init] Using Kubernetes version: v1.20.5
  4. [preflight] Running pre-flight checks
  5. [preflight] Pulling images required for setting up a Kubernetes cluster
  6. [preflight] This might take a minute or two, depending on the speed of your internet connection
  7. [preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
  8. [certs] Using certificateDir folder "/etc/kubernetes/pki"
  9. [certs] Generating "ca" certificate and key
  10. [certs] Generating "apiserver" certificate and key
  11. [certs] apiserver serving cert is signed for DNS names [k8s-master kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.0.5]
  12. [certs] Generating "apiserver-kubelet-client" certificate and key
  13. [certs] Generating "front-proxy-ca" certificate and key
  14. [certs] Generating "front-proxy-client" certificate and key
  15. [certs] Generating "etcd/ca" certificate and key
  16. [certs] Generating "etcd/server" certificate and key
  17. [certs] etcd/server serving cert is signed for DNS names [k8s-master localhost] and IPs [192.168.0.5 127.0.0.1 ::1]
  18. [certs] Generating "etcd/peer" certificate and key
  19. [certs] etcd/peer serving cert is signed for DNS names [k8s-master localhost] and IPs [192.168.0.5 127.0.0.1 ::1]
  20. [certs] Generating "etcd/healthcheck-client" certificate and key
  21. [certs] Generating "apiserver-etcd-client" certificate and key
  22. [certs] Generating "sa" key and public key
  23. [kubeconfig] Using kubeconfig folder "/etc/kubernetes"
  24. [kubeconfig] Writing "admin.conf" kubeconfig file
  25. [kubeconfig] Writing "kubelet.conf" kubeconfig file
  26. [kubeconfig] Writing "controller-manager.conf" kubeconfig file
  27. [kubeconfig] Writing "scheduler.conf" kubeconfig file
  28. [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
  29. [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
  30. [kubelet-start] Starting the kubelet
  31. [control-plane] Using manifest folder "/etc/kubernetes/manifests"
  32. [control-plane] Creating static Pod manifest for "kube-apiserver"
  33. [control-plane] Creating static Pod manifest for "kube-controller-manager"
  34. [control-plane] Creating static Pod manifest for "kube-scheduler"
  35. [etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
  36. [wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
  37. [kubelet-check] Initial timeout of 40s passed.
  38. [apiclient] All control plane components are healthy after 70.001862 seconds
  39. [upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
  40. [kubelet] Creating a ConfigMap "kubelet-config-1.20" in namespace kube-system with the configuration for the kubelets in the cluster
  41. [upload-certs] Skipping phase. Please see --upload-certs
  42. [mark-control-plane] Marking the node k8s-master as control-plane by adding the labels "node-role.kubernetes.io/master=''" and "node-role.kubernetes.io/control-plane='' (deprecated)"
  43. [mark-control-plane] Marking the node k8s-master as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
  44. [bootstrap-token] Using token: abcdef.0123456789abcdef
  45. [bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
  46. [bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes
  47. [bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
  48. [bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
  49. [bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
  50. [bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
  51. [kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
  52. [addons] Applied essential addon: CoreDNS
  53. [addons] Applied essential addon: kube-proxy
  55. Your Kubernetes control-plane has initialized successfully!
  57. To start using your cluster, you need to run the following as a regular user:
  59.   mkdir -p $HOME/.kube
  60.   sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  61.   sudo chown $(id -u):$(id -g) $HOME/.kube/config
  63. Alternatively, if you are the root user, you can run:
  65.   export KUBECONFIG=/etc/kubernetes/admin.conf
  67. You should now deploy a pod network to the cluster.
  68. Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  69.   https://kubernetes.io/docs/concepts/cluster-administration/addons/
  71. Then you can join any number of worker nodes by running the following on each as root:
  73. kubeadm join 192.168.0.5:6443 --token abcdef.0123456789abcdef \
  74.     --discovery-token-ca-cert-hash sha256:446623b965cdb0289c687e74af53f9e9c2063e854a42ee36be9aa249d3f0ccec

拷贝 kubeconfig 文件

  1. $ mkdir -p $HOME/.kube
  2. $ sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  3. $ sudo chown $(id -u):$(id -g) $HOME/.kube/config

**

添加节点

记住初始化集群上面的配置和操作要提前做好，将 master 节点上面的 $HOME/.kube/config 文件拷贝到 node 节点对应的文件中，安装 kubeadm、kubelet、kubectl，然后执行上面初始化完成后提示的 join 命令即可：

  1. # kubeadm join 192.168.0.5:6443 --token abcdef.0123456789abcdef \
  2. >     --discovery-token-ca-cert-hash sha256:446623b965cdb0289c687e74af53f9e9c2063e854a42ee36be9aa249d3f0ccec 
  3. [preflight] Running pre-flight checks
  4. [preflight] Reading configuration from the cluster...
  5. [preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
  6. [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
  7. [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
  8. [kubelet-start] Starting the kubelet
  9. [kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...
  11. This node has joined the cluster:
  12. * Certificate signing request was sent to apiserver and a response was received.
  13. * The Kubelet was informed of the new secure connection details.
  15. Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

如果忘记了上面的 join 命令可以使用命令kubeadm token create —print-join-command重新获取。

执行成功后运行 get nodes 命令：

  1. $ kubectl get no
  2. NAME         STATUS     ROLES                  AGE   VERSION
  3. k8s-master   NotReady   control-plane,master   29m   v1.20.5
  4. k8s-node01   NotReady   <none>                 28m   v1.20.5

可以看到是 NotReady 状态，这是因为还没有安装网络插件，接下来安装网络插件，可以在文档 https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/ 中选择我们自己的网络插件，这里我们安装 calio:

  1. $ wget https://docs.projectcalico.org/v3.8/manifests/calico.yaml

因为有节点是多网卡，所以需要在资源清单文件中指定内网网卡
$ vi calico.yaml

  1. ......
  2. spec:
  3.  containers:
  4.  - env:
  5.  - name: DATASTORE_TYPE
  6.    value: kubernetes
  7.  - name: IP_AUTODETECTION_METHOD # DaemonSet中添加该环境变量
  8.    value: interface=eth0 # 指定内网网卡
  9.  - name: WAIT_FOR_DATASTORE
  10.    value: "true"
  11. - name: CALICO_IPV4POOL_CIDR # 由于在init的时候配置的172网段，所以这里需要修改
  12.   value: "172.16.0.0/16"
  14. ......

安装calico网络插件

  1. $ kubectl apply -f calico.yaml

隔一会儿查看 Pod 运行状态：

  1. # kubectl get pod -n kube-system 
  2. NAME                                      READY   STATUS              RESTARTS   AGE
  3. calico-kube-controllers-bcc6f659f-zmw8n   0/1     ContainerCreating   0          7m58s
  4. calico-node-c4vv7                         1/1     Running             0          7m58s
  5. calico-node-dtw7g                         0/1     PodInitializing     0          7m58s
  6. coredns-54d67798b7-mrj2b                  1/1     Running             0          46m
  7. coredns-54d67798b7-p667d                  1/1     Running             0          46m
  8. etcd-k8s-master                           1/1     Running             0          46m
  9. kube-apiserver-k8s-master                 1/1     Running             0          46m
  10. kube-controller-manager-k8s-master        1/1     Running             0          46m
  11. kube-proxy-clf4s                          1/1     Running             0          45m
  12. kube-proxy-mt7tt                          1/1     Running             0          46m
  13. kube-scheduler-k8s-master                 1/1     Running             0          46m

网络插件运行成功了，node 状态也正常了：

  1. # kubectl get nodes 
  2. NAME         STATUS   ROLES                  AGE   VERSION
  3. k8s-master   Ready    control-plane,master   47m   v1.20.5
  4. k8s-node01   Ready    <none>                 46m   v1.20.5

用同样的方法添加另外一个节点即可。**

配置命令自动补全

  1. yum install -y bash-completion
  2. source /usr/share/bash-completion/bash_completion
  3. source <(kubectl completion bash)
  4. echo "source <(kubectl completion bash)" >> ~/.bashrc

踩坑

在1.20版本以上，当使用nfs做存储的时候，在创建PVC时，会报以下错误：

  1. I0323 08:41:25.264754       1 controller.go:987] provision "default/test-nfs-pvc2" class "nfs-client-storageclass": started
  2. E0323 08:41:25.267631       1 controller.go:1004] provision "default/test-nfs-pvc2" class "nfs-client-storageclass": unexpected error getting claim reference: selfLink was empty, can't make reference

这是因为kubernetes1.20.0废弃了selfLink，解决办法是重新加回来，如下在kube-apiserver.yaml中添加如下参数：

  1. $ vim /etc/kubernetes/manifests/kube-apiserver.yaml
  2. # 增加一行
  3. - --feature-gates=RemoveSelfLink=false

然后重新apply以下使之生效：

  1. kubectl apply -f /etc/kubernetes/manifests/kube-apiserver.yaml

参考文档

【1】：https://github.com/containerd/containerd/issues/4857
【2】：https://github.com/containerd/containerd
【3】：https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.20.md