13、实践 - 13.4、kubeadm集群更改证书 - 《Kubernetes》

获取现在集群的证书信息
- 获取apiserver的证书信息
- 获取ectd的证书详情
创建证书
- 创建CA服务端证书签名请求配置文件openssl.conf
创建集群的key和CA
创建 Certificates
生成kubernetes各组件配置文件并应用
更新证书
- Master节点
- Node节点

未完待测…………………………..

kubeadm集群默认证书是1年，如果在部署前更换的直接修改源代码，然后编译即可。如果已经部署了，需要更新证书，则可以参考下面的方法。

集群信息，单master

# kubectl get node
NAME         STATUS   ROLES    AGE    VERSION
k8s-master   Ready    master   118d   v1.17.2
k8s-node01   Ready    node01   118d   v1.17.2
k8s-node02   Ready    node02   118d   v1.17.2

获取现在集群的证书信息

获取apiserver的证书信息

# openssl x509 -noout -text -in /etc/kubernetes/pki/apiserver.crt
....
X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage: 
                TLS Web Server Authentication
            X509v3 Subject Alternative Name: 
                DNS:k8s-master, DNS:kubernetes, DNS:kubernetes.default, DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster.local, IP Address:10.96.0.1, IP Address:10.1.10.128
....

从上面得到签发的DNS和IP详情，如下：

DNS.1=k8s-master
DNS.2=kubernetes
DNS.3=kubernetes.default
DNS.4=kubernetes.default.svc
DNS.5=kubernetes.default.svc.cluster.local
IP.1=10.96.0.1
IP.2=10.1.10.128

获取ectd的证书详情

# openssl x509 -noout -text -in /etc/kubernetes/pki/etcd/server.crt
.....
        X509v3 extensions:
            X509v3 Basic Constraints: 
                CA:FALSE
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage: 
                TLS Web Server Authentication, TLS Web Client Authentication
            X509v3 Subject Alternative Name: 
                DNS:k8s-master, DNS:localhost, IP Address:127.0.0.1, IP Address:10.1.10.128, IP Address:0:0:0:0:0:0:0:1
.....

从上面得到签发的DNS和IP详情，如下：

DNS.1=k8s-master
DNS.2=localhost
IP.1=127.0.0.1
IP.2=10.1.10.128
IP.3=0:0:0:0:0:0:0:1

创建证书

我们只需要在一个节点上进行证书生成，生成的证书分发到其他节点即可。我们创建一个目录用于保存新建的证书文件，mkdir /root/k8s/newssl -p && cd `` /root/k8s/newssl

创建CA服务端证书签名请求配置文件`openssl.conf`

[ req ]
default_bits = 2048
default_md = sha256
distinguished_name = req_distinguished_name
[req_distinguished_name]
[ v3_ca ]
basicConstraints = critical, CA:TRUE
keyUsage = critical, digitalSignature, keyEncipherment, keyCertSign
[ v3_req_server ]
basicConstraints = CA:FALSE
keyUsage = critical, digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth
[ v3_req_client ]
basicConstraints = CA:FALSE
keyUsage = critical, digitalSignature, keyEncipherment
extendedKeyUsage = clientAuth
[ v3_req_apiserver ]
basicConstraints = CA:FALSE
keyUsage = critical, digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names_cluster
[ v3_req_etcd ]
basicConstraints = CA:FALSE
keyUsage = critical, digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth, clientAuth
subjectAltName = @alt_names_etcd
[ alt_names_cluster ]
DNS.1=k8s-master
DNS.2=kubernetes
DNS.3=kubernetes.default
DNS.4=kubernetes.default.svc
DNS.5=kubernetes.default.svc.cluster.local
IP.1=10.96.0.1
IP.2=10.1.10.128
[ alt_names_etcd ]
DNS.1=k8s-master
DNS.2=localhost
IP.1=127.0.0.1
IP.2=10.1.10.128
IP.3=0:0:0:0:0:0:0:1

注意替换alt_names_cluster和alt_names_etcd的内容

创建集群的key和CA

需要创建的CA信息如下：

路径	Common Name	描述
ca.crt,key	kubernetes	Kubernetes general CA
etcd/ca.crt,key	kubernetes	For all etcd-related functions
front-proxy-ca.crt,key	kubernetes	For the front-end proxy

要注意 CA 中 CN(Common Name) 与 O(Organization) 等内容是会影响Kubernetes组件认证的。

CA (Certificate Authority) 是自签名的根证书，用来签名后续创建的其它证书
CN (Common Name), apiserver 会从证书中提取该字段作为请求的用户名 (User Name)
O (Organization), apiserver 会从证书中提取该字段作为请求用户所属的组 (Group)

（1）、创建kubernetes CA

openssl genrsa -out ca.key 2048
openssl req -x509 -new -nodes -key ca.key \
  -subj "/CN=kubernetes" -config openssl.conf \
  -extensions v3_ca -out ca.crt -days 3560

（2）、创建etcd CA

mkdir -p etcd
openssl genrsa -out etcd/ca.key 2048
openssl req -x509 -new -nodes -key etcd/ca.key \
  -subj "/CN=kubernetes" -config openssl.conf \
  -extensions v3_ca -out etcd/ca.crt -days 3560

（3）、创建front-proxy CA

openssl genrsa -out front-proxy-ca.key 2048
openssl req -x509 -new -nodes -key front-proxy-ca.key \
  -subj "/CN=kubernetes" -config openssl.conf \
  -extensions v3_ca -out front-proxy-ca.crt -days 3560

创建 Certificates

将要创建的 Certificates有:

Name	Key	Certificates	Common Name	Organization
etcd/server	etcd/server.key	etcd/server.crt	master
etcd/peer	etcd/peer.key	etcd/peer.crt	master
etcd/healthcheck-client	etcd/healthcheck-client.key	etcd/healthcheck-client.crt	kube-etcd-healthcheck-client	system:masters
apiserver-etcd-client	apiserver-etcd-client.key	apiserver-etcd-client.crt	kube-apiserver-etcd-client	system:masters
apiserver	apiserver.key	apiserver.crt	kube-apiserver
apiserver-kubelet-client	apiserver-kubelet-client.key	apiserver-kubelet-client.crt	kube-apiserver-kubelet-client	system:masters
front-proxy-client	front-proxy-client.key	front-proxy-client.crt	front-proxy-client
kube-scheduler	kube-scheduler.key	kube-scheduler.crt	system:kube-scheduler
sa(kube-controller-manager)	sa.key(sa.pub)	kube-controller-manager.crt	system:kube-controller-manager
admin(kubectl)	admin.key	admin.crt	kubernetes-admin	system:masters
kubelet	kubelet.key	kubelet.crt	system:node:master	system:nodes

（1）、创建etcd/server

openssl genrsa -out etcd/server.key 2048
openssl req -new -key etcd/server.key \
  -subj "/CN=master" -out etcd/server.csr
openssl x509 -in etcd/server.csr -req -CA etcd/ca.crt \
  -CAkey etcd/ca.key -CAcreateserial -extensions v3_req_etcd \
  -extfile openssl.conf -out etcd/server.crt -days 3560

（2）、创建etcd/peer

openssl genrsa -out etcd/peer.key 2048
openssl req -new -key etcd/peer.key \
  -subj "/CN=master" -out etcd/peer.csr
openssl x509 -in etcd/peer.csr -req -CA etcd/ca.crt \
  -CAkey etcd/ca.key -CAcreateserial -extensions v3_req_etcd \
  -extfile openssl.conf -out etcd/peer.crt -days 3560

（3）、创建etcd/healthcheck-client

openssl genrsa -out etcd/healthcheck-client.key 2048
openssl req -new -key etcd/healthcheck-client.key \
  -subj "/CN=kube-etcd-healthcheck-client/O=system:masters" \
  -out etcd/healthcheck-client.csr
openssl x509 -in etcd/healthcheck-client.csr -req -CA etcd/ca.crt \
  -CAkey etcd/ca.key -CAcreateserial -extensions v3_req_etcd \
  -extfile openssl.conf -out etcd/healthcheck-client.crt -days 3560

（4）、创建apiserver-etcd-client

openssl genrsa -out apiserver-etcd-client.key 2048
openssl req -new -key apiserver-etcd-client.key \
  -subj "/CN=kube-apiserver-etcd-client/O=system:masters" \
  -out apiserver-etcd-client.csr
openssl x509 -in apiserver-etcd-client.csr -req -CA etcd/ca.crt \
  -CAkey etcd/ca.key -CAcreateserial -extensions v3_req_etcd \
  -extfile openssl.conf -out apiserver-etcd-client.crt -days 3560

（5）、创建apiserver

openssl genrsa -out apiserver.key 2048
openssl req -new -key apiserver.key \
  -subj "/CN=kube-apiserver" -config openssl.conf \
  -out apiserver.csr
openssl x509 -req -in apiserver.csr -CA ca.crt -CAkey ca.key \
  -CAcreateserial -extensions v3_req_apiserver \
  -extfile openssl.conf -out apiserver.crt -days 3560

（6）、创建apiserver-kubelet-client

openssl genrsa -out apiserver-kubelet-client.key 2048
openssl req -new -key apiserver-kubelet-client.key \
  -subj "/CN=kube-apiserver-kubelet-client/O=system:masters" \
  -out apiserver-kubelet-client.csr
openssl x509 -req -in apiserver-kubelet-client.csr -CA ca.crt -CAkey ca.key \
  -CAcreateserial -extensions v3_req_client \
  -extfile openssl.conf -out apiserver-kubelet-client.crt -days 3560

（7）、创建front-proxy-client

openssl genrsa -out front-proxy-client.key 2048
openssl req -new -key front-proxy-client.key \
  -subj "/CN=front-proxy-client" \
  -out front-proxy-client.csr
openssl x509 -req -in front-proxy-client.csr -CA front-proxy-ca.crt -CAkey front-proxy-ca.key \
  -CAcreateserial -extensions v3_req_client \
  -extfile openssl.conf -out front-proxy-client.crt -days 3560

（8）、创建kube-scheduler

openssl genrsa -out kube-scheduler.key 2048
openssl req -new -key kube-scheduler.key \
  -subj "/CN=system:kube-scheduler" \
  -out kube-scheduler.csr
openssl x509 -req -in kube-scheduler.csr -CA ca.crt -CAkey ca.key \
  -CAcreateserial -extensions v3_req_client \
  -extfile openssl.conf -out kube-scheduler.crt -days 3560

（9）、创建sa(kube-controller-manager)

openssl genrsa -out sa.key 2048
openssl rsa -in sa.key -pubout -out sa.pub
openssl req -new -key sa.key \
  -subj "/CN=system:kube-controller-manager" \
  -out kube-controller-manager.csr
openssl x509 -req -in kube-controller-manager.csr -CA ca.crt -CAkey ca.key \
  -CAcreateserial -extensions v3_req_client \
  -extfile openssl.conf -out kube-controller-manager.crt -days 3560

（10）、创建admin(kubectl)

openssl genrsa -out admin.key 2048
openssl req -new -key admin.key \
  -subj "/CN=kubernetes-admin/O=system:masters" \
  -out admin.csr
openssl x509 -req -in admin.csr -CA ca.crt -CAkey ca.key \
  -CAcreateserial -extensions v3_req_client \
  -extfile openssl.conf -out admin.crt -days 3560

（11）、创建kubelet

openssl genrsa -out kubelet.key 2048
# 此处为 master 节点 nodeName，每个 master 生成对应的证书
openssl req -new -key kubelet.key \
  -subj "/CN=system:node:k8s-master/O=system:nodes" \
  -out kubelet.csr
openssl x509 -req -CA ca.crt -CAkey ca.key \
  -CAcreateserial -extensions v3_req_client \
  -extfile openssl.conf -days 3560 -in kubelet.csr -out kubelet.crt

生成kubernetes各组件配置文件并应用

需要生成的配置文件列表

配置文件名称	组件证书文件名称	组件秘钥文件名称	根证书文件名称
admin.conf(kubectl)	admin.crt	admin.key	ca.crt
kubelet.conf	kubelet.crt	kubelet.key	ca.crt
scheduler.conf	kube-scheduler.crt	kube-scheduler.key	ca.crt
controller-manager.conf	kube-controller-manager.crt	sa.key	ca.crt

注意：

操作前请先备份原有配置文件
除了kubelet.conf文件需注意配置为对应节点的nodeName，其余配置文件可通用
以下操作请先在一台 master 节点上操作确认没有问题后再进行配置其他节点
–certificate-authority：指定根证书
–client-certificate、–client-key：指定组件证书及秘钥
–embed-certs=true：将组件证书内容嵌入到生成的配置文件中(不加时，写入的是证书文件路径)

（1）、admin.conf(kubectl)

KUBE_APISERVER="https://10.1.10.128:6443"
CLUSTER_NAME="kubernetes"
KUBE_USER="kubernetes-admin"
KUBE_CERT="admin"
KUBE_CONFIG="admin.conf"
# 设置集群参数
kubectl config set-cluster ${CLUSTER_NAME} \
  --certificate-authority=ca.crt \
  --embed-certs=true \
  --server=${KUBE_APISERVER} \
  --kubeconfig=${KUBE_CONFIG}
# 设置客户端认证参数
kubectl config set-credentials ${KUBE_USER} \
  --client-certificate=${KUBE_CERT}.crt \
  --client-key=${KUBE_CERT}.key \
  --embed-certs=true \
  --kubeconfig=${KUBE_CONFIG}
# 设置上下文参数
kubectl config set-context ${KUBE_USER}@${CLUSTER_NAME} \
  --cluster=${CLUSTER_NAME} \
  --user=${KUBE_USER} \
  --kubeconfig=${KUBE_CONFIG}
# 设置当前使用的上下文
kubectl config use-context ${KUBE_USER}@${CLUSTER_NAME} --kubeconfig=${KUBE_CONFIG}
# 查看生成的配置文件
kubectl config view --kubeconfig=${KUBE_CONFIG}

（2）、kubelet.conf（注意配置对应的nodeName）

KUBE_APISERVER="https://10.1.10.128:6443"
CLUSTER_NAME="default-cluster"
# 此处为 master 节点 nodeName，每个 master 生成对应的 kubelet.conf
KUBE_USER="default-auth"
KUBE_CERT="kubelet"
KUBE_CONFIG="kubelet.conf"
# 设置集群参数
kubectl config set-cluster ${CLUSTER_NAME} \
  --certificate-authority=ca.crt \
  --embed-certs=true \
  --server=${KUBE_APISERVER} \
  --kubeconfig=${KUBE_CONFIG}
# 设置客户端认证参数
kubectl config set-credentials ${KUBE_USER} \
  --client-certificate=${KUBE_CERT}.crt \
  --client-key=kubelet.key \
  --embed-certs=true \
  --kubeconfig=${KUBE_CONFIG}
# 设置上下文参数
kubectl config set-context ${KUBE_USER}@${CLUSTER_NAME} \
  --cluster=${CLUSTER_NAME} \
  --user=${KUBE_USER} \
  --kubeconfig=${KUBE_CONFIG}
# 设置当前使用的上下文
kubectl config use-context ${KUBE_USER}@${CLUSTER_NAME} --kubeconfig=${KUBE_CONFIG}
# 查看生成的配置文件
kubectl config view --kubeconfig=${KUBE_CONFIG}

（3）、scheduler.conf

KUBE_APISERVER="https://10.1.10.128:6443"
CLUSTER_NAME="kubernetes"
KUBE_USER="system:kube-scheduler"
KUBE_CERT="kube-scheduler"
KUBE_CONFIG="scheduler.conf"
# 设置集群参数
kubectl config set-cluster ${CLUSTER_NAME} \
  --certificate-authority=ca.crt \
  --embed-certs=true \
  --server=${KUBE_APISERVER} \
  --kubeconfig=${KUBE_CONFIG}
# 设置客户端认证参数
kubectl config set-credentials ${KUBE_USER} \
  --client-certificate=${KUBE_CERT}.crt \
  --client-key=${KUBE_CERT}.key \
  --embed-certs=true \
  --kubeconfig=${KUBE_CONFIG}
# 设置上下文参数
kubectl config set-context ${KUBE_USER}@${CLUSTER_NAME} \
  --cluster=${CLUSTER_NAME} \
  --user=${KUBE_USER} \
  --kubeconfig=${KUBE_CONFIG}
# 设置当前使用的上下文
kubectl config use-context ${KUBE_USER}@${CLUSTER_NAME} --kubeconfig=${KUBE_CONFIG}
# 查看生成的配置文件
kubectl config view --kubeconfig=${KUBE_CONFIG}

（4）、controller-manager.conf

KUBE_APISERVER="https://10.1.10.128:6443"
CLUSTER_NAME="kubernetes"
KUBE_USER="system:kube-controller-manager"
KUBE_CERT="kube-controller-manager"
KUBE_CONFIG="controller-manager.conf"
# 设置集群参数
kubectl config set-cluster ${CLUSTER_NAME} \
  --certificate-authority=ca.crt \
  --embed-certs=true \
  --server=${KUBE_APISERVER} \
  --kubeconfig=${KUBE_CONFIG}
# 设置客户端认证参数
kubectl config set-credentials ${KUBE_USER} \
  --client-certificate=${KUBE_CERT}.crt \
  --client-key=sa.key \
  --embed-certs=true \
  --kubeconfig=${KUBE_CONFIG}
# 设置上下文参数
kubectl config set-context ${KUBE_USER}@${CLUSTER_NAME} \
  --cluster=${CLUSTER_NAME} \
  --user=${KUBE_USER} \
  --kubeconfig=${KUBE_CONFIG}
# 设置当前使用的上下文
kubectl config use-context ${KUBE_USER}@${CLUSTER_NAME} --kubeconfig=${KUBE_CONFIG}
# 查看生成的配置文件
kubectl config view --kubeconfig=${KUBE_CONFIG}

更新证书

Master节点

（1）、先备份原目录

tar zcvf ~/backup/kubernetes-cert.tar.gz /etc/kubernetes/

（2）、将新的证书文件拷贝到原目录

\cp -rf /root/k8s/newssl/* /etc/kubernetes/pki

（3）、将conf文件移动到上层目录

\mv /etc/kubernetes/pki/*.conf /etc/kubernetes

（4）、重启docker和kubelet

systemctl restart docker 
systemctl restart kubelet

（5）、更新准入kubeconfig

cp /etc/kubernetes/admin.conf ~/.kube/config

（6）、查看集群状态

# kubectl get node
NAME         STATUS     ROLES    AGE    VERSION
k8s-master   Ready      master   118d   v1.17.2
k8s-node01   NotReady   node01   118d   v1.17.2
k8s-node02   NotReady   node02   118d   v1.17.2

13.4、kubeadm集群更改证书