kubeadm部署k8s1.13高可用集群

概述

kubeadm已支持集群部署,且在1.13版本中GA,支持多master,多etcd集群化部署,它也是官方最为推荐的部署方式,一来是由它的sig组来推进的,二来kubeadm在很多方面确实很好的利用了kubernetes的许多特性,接下来几篇我们来实践并了解下它的魅力。

目标

  1. 通过kubeadm搭建高可用kubernetes集群,并新建管理用户
  2. 为后续做版本升级演示,此处使用1.13.1版本,到下一篇再升级到v1.14
  3. kubeadm的原理解读

本文主要介绍kubeadm对高可用集群的部署

kubeadm部署k8s v1.13高可用集群

方式有两种

  • Stacked etcd topology

    • 即每台etcd各自独立,分别部署在3台master上,互不通信,优点是简单,缺点是缺乏etcd高可用性
    • 需要至少4台机器(3master和etcd,1node)
  • External etcd topology

    • 即采用集群外etcd拓扑结构,这样的冗余性更好,但需要至少7台机器(3master,3etcd,1node)
    • 生产环境建议采用该方案
    • 本文也采用这个拓扑

步骤

  • 环境准备
  • 安装组件:docker,kubelet,kubeadm(所有节点)
  • 使用上述组件部署etcd高可用集群
  • 部署master
  • 加入node
  • 网络安装
  • 验证
  • 总结

机环境准备

  • 系统环境
    #操作系统版本(非必须,仅为此处案例)
    $cat /etc/redhat-release
    CentOS Linux release 7.2.1511 (Core)
    #内核版本(非必须,仅为此处案例)
    $uname -r
    4.17.8-1.el7.elrepo.x86_64
    
    #数据盘开启ftype(在每台节点上执行)
    umount /data
    mkfs.xfs -n ftype=1 -f /dev/vdb
    #禁用swap
    swapoff -a
    sed -i "s#^/swapfile#\#/swapfile#g" /etc/fstab
    mount -a
    

docker,kubelet,kubeadm的安装(所有节点)

  • 安装运行时(docker)

    • k8s1.13版本根据官方建议,暂不采用最新的18.09,这里我们采用18.06,安装时需指定版本
      • 来源:kubeadm now properly recognizes Docker 18.09.0 and newer, but still treats 18.06 as the default supported version.
    • 安装脚本如下(在每台节点上执行):
      yum install -y yum-utils device-mapper-persistent-data lvm2
      yum-config-manager \
          --add-repo \
          https://download.docker.com/linux/centos/docker-ce.repo
      
      yum makecache fast
      
      yum install -y --setopt=obsoletes=0 \
        docker-ce-18.06.1.ce-3.el7
      
      systemctl start docker
      systemctl enable docker
      
  • 安装kubeadm,kubelet,kubectl

    • 官方的Google yum源无法从国内服务器上直接下载,所以可先在其他渠道下载好,在上传到服务器上
    cat <<EOF > /etc/yum.repos.d/kubernetes.repo
    [kubernetes]
    name=Kubernetes
    baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
    enabled=1
    gpgcheck=1
    repo_gpgcheck=1
    gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
    exclude=kube*
    EOF
    $ yum -y install --downloadonly --downloaddir=k8s kubelet-1.13.1 kubeadm-1.13.1 kubectl-1.13.1
    $ ls k8s/
    25cd948f63fea40e81e43fbe2e5b635227cc5bbda6d5e15d42ab52decf09a5ac-kubelet-1.13.1-0.x86_64.rpm
    53edc739a0e51a4c17794de26b13ee5df939bd3161b37f503fe2af8980b41a89-cri-tools-1.12.0-0.x86_64.rpm
    5af5ecd0bc46fca6c51cc23280f0c0b1522719c282e23a2b1c39b8e720195763-kubeadm-1.13.1-0.x86_64.rpm
    7855313ff2b42ebcf499bc195f51d56b8372abee1a19bbf15bb4165941c0229d-kubectl-1.13.1-0.x86_64.rpm
    fe33057ffe95bfae65e2f269e1b05e99308853176e24a4d027bc082b471a07c0-kubernetes-cni-0.6.0-0.x86_64.rpm
    socat-1.7.3.2-2.el7.x86_64.rpm
    
    • 本地安装

      # 禁用selinux
      setenforce 0
      sed -i 's/^SELINUX=enforcing$/SELINUX=disabled/' /etc/selinux/config
      # 本地安装
      yum localinstall -y k8s/*.rpm
      systemctl enable --now kubelet
      
    • 网络修复,已知centos7会因iptables被绕过而将流量错误路由,因此需确保sysctl配置中的net.bridge.bridge-nf-call-iptables被设置为1

      cat <<EOF >  /etc/sysctl.d/k8s.conf
      net.bridge.bridge-nf-call-ip6tables = 1
      net.bridge.bridge-nf-call-iptables = 1
      EOF
      sysctl --system
      

使用上述组件部署etcd高可用集群

  1. 在etcd节点上,将etcd服务设置为由kubelet启动管理
cat << EOF > /etc/systemd/system/kubelet.service.d/20-etcd-service-manager.conf
[Service]
ExecStart=
ExecStart=/usr/bin/kubelet --address=127.0.0.1 --pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true
Restart=always
EOF

systemctl daemon-reload
systemctl restart kubelet
  1. 给每台etcd主机生成kubeadm配置文件,确保每台主机运行一个etcd实例:在etcd1(即上述的hosts0)上执行上述命令,可以在/tmp目录下看到几个主机名的目录
# Update HOST0, HOST1, and HOST2 with the IPs or resolvable names of your hosts
export HOST0=10.10.184.226
export HOST1=10.10.213.222
export HOST2=10.10.239.108

# Create temp directories to store files that will end up on other hosts.
mkdir -p /tmp/${HOST0}/ /tmp/${HOST1}/ /tmp/${HOST2}/

ETCDHOSTS=(${HOST0} ${HOST1} ${HOST2})
NAMES=("infra0" "infra1" "infra2")

for i in "${!ETCDHOSTS[@]}"; do
HOST=${ETCDHOSTS[$i]}
NAME=${NAMES[$i]}
cat << EOF > /tmp/${HOST}/kubeadmcfg.yaml
apiVersion: "kubeadm.k8s.io/v1beta1"
kind: ClusterConfiguration
etcd:
    local:
        serverCertSANs:
        - "${HOST}"
        peerCertSANs:
        - "${HOST}"
        extraArgs:
            initial-cluster: ${NAMES[0]}=https://${ETCDHOSTS[0]}:2380,${NAMES[1]}=https://${ETCDHOSTS[1]}:2380,${NAMES[2]}=https://${ETCDHOSTS[2]}:2380
            initial-cluster-state: new
            name: ${NAME}
            listen-peer-urls: https://${HOST}:2380
            listen-client-urls: https://${HOST}:2379
            advertise-client-urls: https://${HOST}:2379
            initial-advertise-peer-urls: https://${HOST}:2380
EOF
done
  1. 制作CA:在host0上执行命令生成证书,它将创建两个文件:/etc/kubernetes/pki/etcd/ca.crt /etc/kubernetes/pki/etcd/ca.key (这一步需要翻墙)
[[email protected] ~]# kubeadm init phase certs etcd-ca
[certs] Generating "etcd/ca" certificate and key
  1. 在host0上给每个etcd节点生成证书:
export HOST0=10.10.184.226
export HOST1=10.10.213.222
export HOST2=10.10.239.108
kubeadm init phase certs etcd-server --config=/tmp/${HOST2}/kubeadmcfg.yaml
kubeadm init phase certs etcd-peer --config=/tmp/${HOST2}/kubeadmcfg.yaml
kubeadm init phase certs etcd-healthcheck-client --config=/tmp/${HOST2}/kubeadmcfg.yaml
kubeadm init phase certs apiserver-etcd-client --config=/tmp/${HOST2}/kubeadmcfg.yaml
cp -R /etc/kubernetes/pki /tmp/${HOST2}/
# cleanup non-reusable certificates
find /etc/kubernetes/pki -not -name ca.crt -not -name ca.key -type f -delete

kubeadm init phase certs etcd-server --config=/tmp/${HOST1}/kubeadmcfg.yaml
kubeadm init phase certs etcd-peer --config=/tmp/${HOST1}/kubeadmcfg.yaml
kubeadm init phase certs etcd-healthcheck-client --config=/tmp/${HOST1}/kubeadmcfg.yaml
kubeadm init phase certs apiserver-etcd-client --config=/tmp/${HOST1}/kubeadmcfg.yaml
cp -R /etc/kubernetes/pki /tmp/${HOST1}/
find /etc/kubernetes/pki -not -name ca.crt -not -name ca.key -type f -delete

kubeadm init phase certs etcd-server --config=/tmp/${HOST0}/kubeadmcfg.yaml
kubeadm init phase certs etcd-peer --config=/tmp/${HOST0}/kubeadmcfg.yaml
kubeadm init phase certs etcd-healthcheck-client --config=/tmp/${HOST0}/kubeadmcfg.yaml
kubeadm init phase certs apiserver-etcd-client --config=/tmp/${HOST0}/kubeadmcfg.yaml
# No need to move the certs because they are for HOST0

# clean up certs that should not be copied off this host
find /tmp/${HOST2} -name ca.key -type f -delete
find /tmp/${HOST1} -name ca.key -type f -delete

  • 将证书和kubeadmcfg.yaml下发到各个etcd节点上效果为:
    /root/
    └── kubeadmcfg.yaml
    

/etc/kubernetes/pki ├── apiserver-etcd-client.crt ├── apiserver-etcd-client.key └── etcd ├── ca.crt ├── ca.key ├── healthcheck-client.crt ├── healthcheck-client.key ├── peer.crt ├── peer.key ├── server.crt └── server.key

5. 生成静态pod manifest ,在3台etcd节点上分别执行:(**需翻墙**)
```shell
$ kubeadm init phase etcd local --config=/root/kubeadmcfg.yaml
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
  1. 检查etcd集群状态,至此etcd集群搭建完成
docker run --rm -it --net host -v /etc/kubernetes:/etc/kubernetes k8s.gcr.io/etcd:3.2.24 etcdctl --cert-file /etc/kubernetes/pki/etcd/peer.crt --key-file /etc/kubernetes/pki/etcd/peer.key --ca-file /etc/kubernetes/pki/etcd/ca.crt --endpoints https://${HOST0}:2379 cluster-health
member 9969ee7ea515cbd2 is healthy: got healthy result from https://10.10.213.222:2379
member cad4b939d8dfb250 is healthy: got healthy result from https://10.10.239.108:2379
member e6e86b3b5b495dfb is healthy: got healthy result from https://10.10.184.226:2379
cluster is healthy

使用kubeadm部署master

  • 将任意一台etcd上的证书拷贝到master1节点的同样目录中 官方脚本如下(少了第二条ca.key的拷贝)
export CONTROL_PLANE="[email protected]"
+scp /etc/kubernetes/pki/etcd/ca.crt "${CONTROL_PLANE}":
+scp /etc/kubernetes/pki/etcd/ca.key "${CONTROL_PLANE}":
+scp /etc/kubernetes/pki/apiserver-etcd-client.crt "${CONTROL_PLANE}":
+scp /etc/kubernetes/pki/apiserver-etcd-client.key "${CONTROL_PLANE}":
  • 在第一台master上编写配置文件kubeadm-config.yaml并初始化
apiVersion: kubeadm.k8s.io/v1beta1
kind: ClusterConfiguration
kubernetesVersion: stable
#kubernetesVersion: 1.13.1
#注意这里我们想先安装1.13.1再后续升级1.13.4所以直接声明版本号
apiServer:
  certSANs:
  - "k8s.paas.test"
controlPlaneEndpoint: "k8s.paas.test:6443"
etcd:
    external:
        endpoints:
        - https://10.10.184.226:2379
        - https://10.10.213.222:2379
        - https://10.10.239.108:2379
        caFile: /etc/kubernetes/pki/etcd/ca.crt
        certFile: /etc/kubernetes/pki/apiserver-etcd-client.crt
        keyFile: /etc/kubernetes/pki/apiserver-etcd-client.key
networking:
  podSubnet: "10.244.0.0/16"

注意:这里的k8s.paas.test:6443是一个LB,如果没有可用虚拟IP来做

  • 使用私有仓库(自定义镜像功能) kubeadm支持通过修改配置文件中的参数来灵活定制集群初始化工作,如imageRepository可以设置镜像前缀,我们可以在将镜像传到自己内部私服上之后,编辑kubeadm-config.yaml中的该参数之后再执行init
  • 在master1 上执行:kubeadm init –config kubeadm-config.yaml
    Your Kubernetes master has initialized successfully!
    
    To start using your cluster, you need to run the following as a regular user:
    
      mkdir -p $HOME/.kube
      sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
      sudo chown $(id -u):$(id -g) $HOME/.kube/config
    
    You should now deploy a pod network to the cluster.
    Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
      https://kubernetes.io/docs/concepts/cluster-administration/addons/
    
    You can now join any number of machines by running the following on each node
    as root:
    
      kubeadm join k8s.paas.test:6443 --token f1oygc.3zlc31yjcut46prf --discovery-token-ca-cert-hash xx
    
      [[email protected] ~]
      $kubectl get node
      NAME     STATUS     ROLES    AGE     VERSION
      k8s-m1   NotReady   master   4m54s   v1.13.1
    
安装另外2台master
  • master1上的admin.conf配置文件和pki相关证书拷贝到另外2台master同样目录下如:
    /etc/kubernetes/pki/ca.crt
    /etc/kubernetes/pki/ca.key
    /etc/kubernetes/pki/sa.key
    /etc/kubernetes/pki/sa.pub
    /etc/kubernetes/pki/front-proxy-ca.crt
    /etc/kubernetes/pki/front-proxy-ca.key
    /etc/kubernetes/pki/etcd/ca.crt
    /etc/kubernetes/pki/etcd/ca.key
    /etc/kubernetes/admin.conf
    

    注:官网文档少了两个文件/etc/kubernetes/pki/apiserver-etcd-client.crt /etc/kubernetes/pki/apiserver-etcd-client.key,不加apiserver会启动失败并报错:

    Unable to create storage backend: config (&{ /registry [] /etc/kubernetes/pki/apiserver-etcd-client.key /etc/kubernetes/pki/apiserver-etcd-client.crt /etc/kubernetes/pki/etcd/ca.crt true 0xc000133c20 <nil> 5m0s 1m0s}), err (open /etc/kubernetes/pki/apiserver-etcd-client.crt: no such file or directory)
    
  • 在2和3master中执行加入:
  kubeadm join k8s.paas.test:6443 --token f1oygc.3zlc31yjcut46prf --discovery-token-ca-cert-hash sha256:078b63e29378fb6dcbedd80dd830b83e37521f294b4e3416cd77e854041d912f --experimental-control-plane

加入node节点

[[email protected] ~]
$  kubeadm join k8s.paas.test:6443 --token f1oygc.3zlc31yjcut46prf --discovery-token-ca-cert-hash sha256:078b63e29378fb6dcbedd80dd830b83e37521f294b4e3416cd77e854041d912f
[preflight] Running pre-flight checks
[discovery] Trying to connect to API Server "k8s.paas.test:6443"
[discovery] Created cluster-info discovery client, requesting info from "https://k8s.paas.test:6443"
[discovery] Requesting info from "https://k8s.paas.test:6443" again to validate TLS against the pinned public key
[discovery] Cluster info signature and contents are valid and TLS certificate validates against pinned roots, will use API Server "k8s.paas.test:6443"
[discovery] Successfully established connection with API Server "k8s.paas.test:6443"
[join] Reading configuration from the cluster...
[join] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[kubelet] Downloading configuration for the kubelet from the "kubelet-config-1.13" ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Activating the kubelet service
[tlsbootstrap] Waiting for the kubelet to perform the TLS Bootstrap...
[patchnode] Uploading the CRI Socket information "/var/run/dockershim.sock" to the Node API object "k8s-n1" as an annotation

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the master to see this node join the cluster.

网络安装

kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/a70459be0084506e4ec919aa1c114638878db11b/Documentation/kube-flannel.yml
kubectl get pod --all-namespaces -o wide
NAMESPACE     NAME                             READY   STATUS    RESTARTS   AGE     IP              NODE     NOMINATED NODE   READINESS GATES
kube-system   coredns-86c58d9df4-dc4t2         1/1     Running   0          14m     172.17.0.3      k8s-m1   <none>           <none>
kube-system   coredns-86c58d9df4-jxv6v         1/1     Running   0          14m     172.17.0.2      k8s-m1   <none>           <none>
kube-system   kube-apiserver-k8s-m1            1/1     Running   0          13m     10.10.119.128   k8s-m1   <none>           <none>
kube-system   kube-apiserver-k8s-m2            1/1     Running   0          5m      10.10.76.80     k8s-m2   <none>           <none>
kube-system   kube-apiserver-k8s-m3            1/1     Running   0          4m58s   10.10.56.27     k8s-m3   <none>           <none>
kube-system   kube-controller-manager-k8s-m1   1/1     Running   0          13m     10.10.119.128   k8s-m1   <none>           <none>
kube-system   kube-controller-manager-k8s-m2   1/1     Running   0          5m      10.10.76.80     k8s-m2   <none>           <none>
kube-system   kube-controller-manager-k8s-m3   1/1     Running   0          4m58s   10.10.56.27     k8s-m3   <none>           <none>
kube-system   kube-flannel-ds-amd64-nvmtk      1/1     Running   0          44s     10.10.56.27     k8s-m3   <none>           <none>
kube-system   kube-flannel-ds-amd64-pct2g      1/1     Running   0          44s     10.10.76.80     k8s-m2   <none>           <none>
kube-system   kube-flannel-ds-amd64-ptv9z      1/1     Running   0          44s     10.10.119.128   k8s-m1   <none>           <none>
kube-system   kube-flannel-ds-amd64-zcv49      1/1     Running   0          44s     10.10.175.146   k8s-n1   <none>           <none>
kube-system   kube-proxy-9cmg2                 1/1     Running   0          2m34s   10.10.175.146   k8s-n1   <none>           <none>
kube-system   kube-proxy-krlkf                 1/1     Running   0          4m58s   10.10.56.27     k8s-m3   <none>           <none>
kube-system   kube-proxy-p9v66                 1/1     Running   0          14m     10.10.119.128   k8s-m1   <none>           <none>
kube-system   kube-proxy-wcgg6                 1/1     Running   0          5m      10.10.76.80     k8s-m2   <none>           <none>
kube-system   kube-scheduler-k8s-m1            1/1     Running   0          13m     10.10.119.128   k8s-m1   <none>           <none>
kube-system   kube-scheduler-k8s-m2            1/1     Running   0          5m      10.10.76.80     k8s-m2   <none>           <none>
kube-system   kube-scheduler-k8s-m3            1/1     Running   0          4m58s   10.10.56.27     k8s-m3   <none>           <none>

安装完成

验证
  • 首先验证kube-apiserver, kube-controller-manager, kube-scheduler, pod network 是否正常:

    $kubectl create deployment nginx --image=nginx:alpine
    $kubectl get pods -l app=nginx -o wide
    NAME                     READY   STATUS    RESTARTS   AGE     IP           NODE     NOMINATED NODE   READINESS GATES
    nginx-54458cd494-r6hqm   1/1     Running   0          5m24s   10.244.4.2   k8s-n1   <none>           <none>
    
  • kube-proxy验证

    $kubectl expose deployment nginx --port=80 --type=NodePort
    service/nginx exposed
    
    [[email protected] ~]
    $kubectl get svc
    NAME         TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE
    kubernetes   ClusterIP   10.96.0.1        <none>        443/TCP        122m
    nginx        NodePort    10.108.192.221   <none>        80:30992/TCP   4s
    [[email protected] ~]
    $kubectl get pods -l app=nginx -o wide
    NAME                     READY   STATUS    RESTARTS   AGE     IP           NODE     NOMINATED NODE   READINESS GATES
    nginx-54458cd494-r6hqm   1/1     Running   0          6m53s   10.244.4.2   k8s-n1   <none>           <none>
    
    $curl -I k8s-n1:30992
    HTTP/1.1 200 OK
    
  • 验证dns,pod网络状态

    kubectl run --generator=run-pod/v1 -it curl --image=radial/busyboxplus:curl
    If you don't see a command prompt, try pressing enter.
    [ [email protected]:/ ]$ nslookup nginx
    Server:    10.96.0.10
    Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local
    
    Name:      nginx
    Address 1: 10.108.192.221 nginx.default.svc.cluster.local
    
  • 高可用

    • 关机master1,在一个随机pod中访问Nginx
    while true;do curl -I nginx && sleep 1 ;done
    

总结

  • 关于版本
    • 内核版本4.19更加稳定此处不建议4.17(再新就是5了)
    • docker最新稳定版是1.17.12,此处是1.18.06,虽k8s官方也确认兼容1.18.09了但生产上还是建议1.17.12
  • 关于网络,各家选择不同,flannel在中小公司较为普遍,但部署前要选好网络插件,在配置文件中提前设置好,(官方博客一开始的kubeadm配置中没写,后面在网络设置中又要求必须加)
  • 出错处理
    • 想重置环境的话,kubeadm reset是个很好的工具,但它并不会完全重置,在etcd中的的部分数据(如configmap secure等)是没有被清空的,所以如果有必要重置真个环境,记得在reset后将etcd也重置。
    • 重置etcd办法为清空etcd节点/var/lib/etcd,重启docker服务)
  • 翻墙
    • 镜像:kubeadm已支持自定义镜像前缀,kubeadm-config.yaml中设置imageRepository即可
    • yum,可提前下载导入,也可以设置http_proxy来访问
    • init,签发证书和init时需要连接google,也可以设置http_proxy来访问。
  • 更多
    • 证书、升级问题将在下一篇继续介绍。

参考: