Centos 7系统下Kubeadm方式搭建k8s集群
Kubernetes集群部署方式
Minikube方式
Minikube是一个工具，可以在本地快速运行一个单点的Kubernetes，适合尝试Kubernetes或日常开发的用户使用，但是不能用于生产环境。
kubeadm方式
Kubeadm也是一个工具，提供kubeadm init和kubeadm join，可用于快速部署Kubernetes集群。
- 二进制包方式
从官方下载发行版的二进制包，手动部署每个组件，组成Kubernetes集群，过程较为繁琐。
生产环境中部署Kubernetes集群，使用Kubeadm和二进制包部署两种方式。Kubeadm部署方式降低了部署门槛，但屏蔽了很多细节，遇到问题就很难排查。
实际生产环境中最好使用二进制包部署Kubernetes集群，虽然繁琐，但有利于了解其中很多工作原理，更有利于后期维护。
基础环境配置说明
主机角色安排
| 主机名 | IP | 功能 |
|---|---|---|
| k8s-master | 192.168.0.8 | master |
| k8s-node01 | 192.168.1.8 | node |
| k8s-node02 | 192.168.16.8 | node |
主机配置
| 配置 | 规格 |
|---|---|
| 内存配置 | 2G |
| CPU配置 | 2个 |
| 系统版本 | CentOS Linux release 7.6.1810 (Core) |
| kubelet版本 | 1.14.3 |
| docker版本 | docker-ce-18.09.6-3.el7.x86_64 |
系统版本说明
[root@k8s-master ~]# cat /etc/redhat-release
CentOS Linux release 7.6.1810 (Core)
[root@k8s-master ~]# uname -r
3.10.0-957.el7.x86_64
[root@k8s-master ~]# getenforce
Disabled
[root@k8s-master ~]# systemctl status firewalld.service
● firewalld.service - firewalld - dynamic firewall daemon
Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)
Active: inactive (dead)
Docs: man:firewalld(1)
[root@k8s-master ~]# 部署步骤
所有节点的基础配置（本节在master与node同时执行）
修改主机名
192.168.0.8服务器：
[root@k8s-master ~]# hostnamectl set-hostname k8s-master
[root@k8s-master ~]# hostname
k8s-master192.168.1.8服务器：
[root@k8s-node01 ~]# hostnamectl set-hostname k8s-node01
[root@k8s-node01 ~]# hostname
k8s-node01192.168.16.8服务器：
[root@k8s-node01 ~]# hostnamectl set-hostname k8s-node02
[root@k8s-node01 ~]# hostname
k8s-node02修改/etc/hosts文件，加入下面两行
192.168.0.8 k8s-master
192.168.1.8 k8s-node01
192.168.16.8 k8s-node02关掉swap分区
[root@k8s-master ~]# swapoff -a永久禁用swap分区：注释掉/etc/fstab文件中/dev/mapper/centos-swap这一行：
[root@k8s-master ~]# cat /etc/fstab
#
# /etc/fstab
# Created by anaconda on Tue Jul 31 23:03:49 2018
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
/dev/mapper/centos-root / xfs defaults 0 0
UUID=ad34d4f1-a758-4924-8ae9-99d0d36939aa /boot xfs defaults 0 0
#/dev/mapper/centos-swap swap swap defaults 0 0 #注释掉这一行
[root@k8s-master ~]# 关闭firewalld
[root@k8s-master ~]# systemctl stop firewalld
[root@k8s-master ~]# systemctl disable firewalld关闭selinux
[root@k8s-master selinux]# cat /etc/selinux/config|grep "^SELINUX="
SELINUX=disabled
[root@k8s-master selinux]# 修改sysctl内核参数
创建k8s.conf文件，写入下面的的内容：
[root@k8s-master ~]# cat /etc/sysctl.d/k8s.conf #k8s.conf文件原来不存在，需要自己创建的
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
vm.swappiness=0
[root@k8s-master ~]# sysctl --system #让参数修改生效配置kubernetes yum源
创建kubernetes.repo文件并写入下面的内容：
# vim /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg安装docker18版本
Centos7安装的docker一般是docker13版本，这里是将docker版本升级到最新版。步骤如下：
（1）保证内核版本在3.10及以上：uname -a
（2）删除旧版本：yum remove -y docker docker-common docker-selinux docker-engine #这一步骤在初次安装docker也最好执行一次，否则后面安装docker可能会报错
（3）安装需要的软件包：yum install -y yum-utils device-mapper-persistent-data lvm2
（4）设置Docker yum源：yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
（5）查看所有仓库中所有docker版本：yum list docker-ce --showduplicates | sort -r
（6）安装docker：yum install docker-ce -y #由于repo中默认只开启stable仓库，故这里一般会默认安装最新版本。如果要安装特定版本：yum install docker-ce-18.06.3.ce-3.el7 -y
（7）设置为开机启动：systemctl enable docker
（8）启动：systemctl start docker
（9）查看启动状态：systemctl status docker
（10）查看版本：docker version
安装kubeadm, kubelet and kubectl
Kubelet负责与其他节点集群通信，并进行本节点Pod和容器生命周期的管理
# yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes
# kubelet --version
# kubeadm version
# kubectl version检查所有服务版本：
# rpm -qa docker-ce kubelet kubeadm kubectl kubernetes-cni
docker-ce-18.09.6-3.el7.x86_64
kubernetes-cni-0.7.5-0.x86_64
kubeadm-1.15.0-0.x86_64
kubelet-1.15.0-0.x86_64
kubectl-1.15.0-0.x86_64启动docker和kubelet并设置为开机自启动
systemctl enable docker
systemctl enable kubelet
systemctl start docker
systemctl start kubelet下载相关镜像
（1）获取镜像列表
[root@k8s-master ~]# kubeadm config images list
W0622 14:33:02.478513 3581 version.go:98] could not fetch a Kubernetes version from the internet: unable to get URL "https://dl.k8s.io/release/stable-1.txt": Get https://dl.k8s.io/release/stable-1.txt: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
W0622 14:33:02.478589 3581 version.go:99] falling back to the local client version: v1.15.0
k8s.gcr.io/kube-apiserver:v1.15.0
k8s.gcr.io/kube-controller-manager:v1.15.0
k8s.gcr.io/kube-scheduler:v1.15.0
k8s.gcr.io/kube-proxy:v1.15.0
k8s.gcr.io/pause:3.1
k8s.gcr.io/etcd:3.3.10
k8s.gcr.io/coredns:1.3.1若报如下错误，则修改主机名
could not convert cfg to an internal cfg: name: Invalid value: "vm_0_8_centos": a DNS-1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*')（2）生成默认kubeadm.conf文件
# kubeadm config print init-defaults > kubeadm.conf（3）修改kubeadm.conf文件的镜像地址
默认为google的镜像仓库地址k8s.gcr.io，国内无法访问，需要把地址修改为国内的地址，这里使用阿里云的镜像仓库地址。
编辑kubeadm.conf，将imageRepository修改为registry.aliyuncs.com/google_containers。并确认Kubernetes版本是v1.15.0，和前文中的镜像列表的版本保持一致
# vim kubeadm.conf
apiVersion: kubeadm.k8s.io/v1beta1
bootstrapTokens:
- groups:
- system:bootstrappers:kubeadm:default-node-token
token: abcdef.0123456789abcdef
ttl: 24h0m0s
usages:
- signing
- authentication
kind: InitConfiguration
localAPIEndpoint:
advertiseAddress: 1.2.3.4
bindPort: 6443
nodeRegistration:
criSocket: /var/run/dockershim.sock
name: k8s-master
taints:
- effect: NoSchedule
key: node-role.kubernetes.io/master
---
apiServer:
timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta1
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controlPlaneEndpoint: ""
controllerManager: {}
dns:
type: CoreDNS
etcd:
local:
dataDir: /var/lib/etcd
#imageRepository: k8s.gcr.io
imageRepository: registry.aliyuncs.com/google_containers
kind: ClusterConfiguration
kubernetesVersion: v1.15.0
networking:
dnsDomain: cluster.local
podSubnet: ""
serviceSubnet: 10.96.0.0/12
scheduler: {}（4）下载镜像：
[root@k8s-master ~]# kubeadm config images pull --config kubeadm.conf
[config/images] Pulled registry.aliyuncs.com/google_containers/kube-apiserver:v1.15.0
[config/images] Pulled registry.aliyuncs.com/google_containers/kube-controller-manager:v1.15.0
[config/images] Pulled registry.aliyuncs.com/google_containers/kube-scheduler:v1.15.0
[config/images] Pulled registry.aliyuncs.com/google_containers/kube-proxy:v1.15.0
[config/images] Pulled registry.aliyuncs.com/google_containers/pause:3.1
[config/images] Pulled registry.aliyuncs.com/google_containers/etcd:3.3.10
[config/images] Pulled registry.aliyuncs.com/google_containers/coredns:1.3.1（5）修改tag
docker tag registry.aliyuncs.com/google_containers/kube-apiserver:v1.15.0 k8s.gcr.io/kube-apiserver:v1.15.0
docker tag registry.aliyuncs.com/google_containers/kube-controller-manager:v1.15.0 k8s.gcr.io/kube-controller-manager:v1.15.0
docker tag registry.aliyuncs.com/google_containers/kube-scheduler:v1.15.0 k8s.gcr.io/kube-scheduler:v1.15.0
docker tag registry.aliyuncs.com/google_containers/kube-proxy:v1.15.0 k8s.gcr.io/kube-proxy:v1.15.0
docker tag registry.aliyuncs.com/google_containers/pause:3.1 k8s.gcr.io/pause:3.1
docker tag registry.aliyuncs.com/google_containers/etcd:3.3.10 k8s.gcr.io/etcd:3.3.10
docker tag registry.aliyuncs.com/google_containers/coredns:1.3.1 k8s.gcr.io/coredns:1.3.1（6）再删除阿里云镜像：
docker rmi registry.aliyuncs.com/google_containers/kube-apiserver:v1.15.0
docker rmi registry.aliyuncs.com/google_containers/kube-controller-manager:v1.15.0
docker rmi registry.aliyuncs.com/google_containers/kube-scheduler:v1.15.0
docker rmi registry.aliyuncs.com/google_containers/kube-proxy:v1.15.0
docker rmi registry.aliyuncs.com/google_containers/pause:3.1
docker rmi registry.aliyuncs.com/google_containers/etcd:3.3.10
docker rmi registry.aliyuncs.com/google_containers/coredns:1.3.1（7）或者使用脚本解决：
[root@k8s-master ~]# cat image.sh
#!/bin/bash
images=(kube-proxy:v1.15.0 kube-scheduler:v1.15.0 kube-controller-manager:v1.15.0 kube-apiserver:v1.15.0 etcd:3.3.10 coredns:1.3.1 pause:3.1 )
for imageName in ${images[@]} ; do
docker pull registry.aliyuncs.com/google_containers/$imageName
docker tag registry.aliyuncs.com/google_containers/$imageName k8s.gcr.io/$imageName
docker rmi registry.aliyuncs.com/google_containers/$imageName
done（8）最后下载留下的镜像是这些：
[root@k8s-master ~]# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
k8s.gcr.io/kube-proxy v1.15.0 d235b23c3570 2 days ago 82.4MB
k8s.gcr.io/kube-apiserver v1.15.0 201c7a840312 2 days ago 207MB
k8s.gcr.io/kube-controller-manager v1.15.0 8328bb49b652 2 days ago 159MB
k8s.gcr.io/kube-scheduler v1.15.0 2d3813851e87 2 days ago 81.1MB
k8s.gcr.io/coredns 1.3.1 eb516548c180 5 months ago 40.3MB
k8s.gcr.io/etcd 3.3.10 2c4adeb21b4f 6 months ago 258MB
k8s.gcr.io/pause 3.1 da86e6ba6ca1 18 months ago 742kB
[root@k8s-master ~]# 忽略swap错误
kubernetes集群不允许开启swap，所以我们需要忽略这个错误
编辑文件 /etc/sysconfig/kubelet，将文件里的KUBELET_EXTRA_ARGS=改成：KUBELET_EXTRA_ARGS="--fail-swap-on=false"
修改之后的文件：
[root@k8s-master ~]# cat /etc/sysconfig/kubelet
KUBELET_EXTRA_ARGS="--fail-swap-on=false"
[root@k8s-master ~]# master节点部署（本节在master节点执行）
初始化Kubernetes Master
根据后面要安装的Calico网络组件：https://v1-12.docs.kubernetes.io/docs/setup/independent/create-cluster-kubeadm/#pod-network
这里定义先POD的网段为: 172.16.0.0/16，API Server地址为Master节点的IP地址。命令：
kubeadm init --kubernetes-version=v1.15.0 --pod-network-cidr=172.16.0.0/16 --apiserver-advertise-address=192.168.0.8执行结果：
[root@k8s-master ~]# kubeadm init --kubernetes-version=v1.15.0 --pod-network-cidr=172.16.0.0/16 --apiserver-advertise-address=192.168.0.8
[init] Using Kubernetes version: v1.15.0
[preflight] Running pre-flight checks
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [k8s-master localhost] and IPs [192.168.0.8 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [k8s-master localhost] and IPs [192.168.0.8 127.0.0.1 ::1]
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [k8s-master kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.0.8]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
[apiclient] All control plane components are healthy after 40.013221 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.15" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node k8s-master as control-plane by adding the label "node-role.kubernetes.io/master=''"
[mark-control-plane] Marking the node k8s-master as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: aakoqt.zq79agckuzc5wt29
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join 192.168.0.8:6443 --token aakoqt.zq79agckuzc5wt29 \
--discovery-token-ca-cert-hash sha256:fa781d59e6e69d1a8abb836f66e3d36fd2d3e1765a1afdb71b1a18af807585c2
[root@k8s-master ~]# 初始化成功后，将最后两行内容记录下来，这个命令用来加入Worker节点时使用。
kubeadm join 192.168.0.8:6443 --token aakoqt.zq79agckuzc5wt29 \
--discovery-token-ca-cert-hash sha256:fa781d59e6e69d1a8abb836f66e3d36fd2d3e1765a1afdb71b1a18af807585c2若报如下错误：
[ERROR FileContent--proc-sys-net-bridge-bridge-nf-call-iptables]: /proc/sys/net/bridge/bridge-nf-call-iptables contents are not set to 1解决：
# echo "1" >/proc/sys/net/bridge/bridge-nf-call-iptables配置kubectl，作为普通用户管理集群并在集群上工作
上一步骤初始化的要求：“To start using your cluster, you need to run the following as a regular user”，需要执行以下命令：
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config获取pods列表
kubectl get pods --all-namespaces命令查看相关状态，可以看到coredns pod处于pending状态，这是因为还没有部署pod网络：
[root@k8s-master ~]# kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-fb8b8dccf-twknr 0/1 Pending 0 3m25s
kube-system coredns-fb8b8dccf-wc4pd 0/1 Pending 0 3m25s
kube-system etcd-k8s-master 1/1 Running 0 2m25s
kube-system kube-apiserver-k8s-master 1/1 Running 0 2m30s
kube-system kube-controller-manager-k8s-master 1/1 Running 0 2m27s
kube-system kube-proxy-b298g 1/1 Running 0 3m25s
kube-system kube-scheduler-k8s-master 1/1 Running 0 2m37s查看集群的健康状态
kubectl get cs命令查看健康状态：
[root@k8s-master ~]# kubectl get cs
NAME STATUS MESSAGE ERROR
scheduler Healthy ok
controller-manager Healthy ok
etcd-0 Healthy {"health":"true"} 部署Pod网络
根据Pod Network Add-on介绍：
You must install a pod network add-on so that your pods can communicate with each other.
您必须安装一个pod网络附加组件，以便您的pod可以彼此通信。
The network must be deployed before any applications. Also, CoreDNS will not start up before a network is installed. kubeadm only supports Container Network Interface (CNI) based networks (and does not support kubenet).
pod网络附加组件是必须安装的，这样pod能够彼此通信，而且网络必须在任何应用程序之前部署。另外，CoreDNS在安装网络之前不会启动。kubeadm只支持基于容器网络接口(CNI)的网络。
如下图支持的Pod网络有JuniperContrail/TungstenFabric、Calico、Canal、Cilium、Flannel、Kube-router、Romana、Wave Net等：
这里我们部署Calico网络，Calico是一个纯三层的方案，其好处是它整合了各种云原生平台(Docker、Mesos 与 OpenStack 等)，每个 Kubernetes 节点上通过 Linux Kernel 现有的 L3 forwarding 功能来实现 vRouter 功能。
根据Pod Network Add-on提示，安装Calico网络就两个步骤：
https://v1-12.docs.kubernetes.io/docs/setup/independent/create-cluster-kubeadm/#pod-network
wget https://docs.projectcalico.org/v3.3/getting-started/kubernetes/installation/hosted/rbac-kdd.yaml
wget https://docs.projectcalico.org/v3.3/getting-started/kubernetes/installation/hosted/kubernetes-datastore/calico-networking/1.7/calico.yaml
curl https://docs.projectcalico.org/v3.4/getting-started/kubernetes/installation/hosted/calico.yaml -O如果pod CIDR即pod-network-cidr使用的是 192.168.0.0/16，可以跳过，否则执行下面命令，更新一下pod CIDR的配置信息：
POD_CIDR="<your-pod-cidr>" \
sed -i -e "s?192.168.0.0/16?$POD_CIDR?g" calico.yaml或直接编辑calico.yaml
# The default IPv4 pool to create on startup if none exists. Pod IPs will be
# chosen from this range. Changing this value after installation will have
# no effect. This should fall within `--cluster-cidr`.
- name: CALICO_IPV4POOL_CIDR
value: "172.16.0.0/16"kubectl apply -f rbac-kdd.yaml
kubectl apply -f calico.yaml这两个步骤的执行结果如下：
[root@k8s-master ~]# kubectl apply -f rbac-kdd.yaml
clusterrole.rbac.authorization.k8s.io/calico-node created
clusterrolebinding.rbac.authorization.k8s.io/calico-node created
[root@k8s-master ~]# kubectl apply -f calico.yaml
configmap/calico-config created
service/calico-typha created
deployment.apps/calico-typha created
poddisruptionbudget.policy/calico-typha created
daemonset.extensions/calico-node created
serviceaccount/calico-node created
customresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org created
[root@k8s-master ~]# 使用kubectl get pods --all-namespaces命令查看运行状态
还没部署好的时候，状态是这样的，需要等个几分钟：
[root@k8s-master ~]# kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system calico-node-nzqzl 0/2 ContainerCreating 0 82s
kube-system coredns-fb8b8dccf-twknr 0/1 Pending 0 83m
kube-system coredns-fb8b8dccf-wc4pd 0/1 Pending 0 83m
kube-system etcd-k8s-master 1/1 Running 0 82m
kube-system kube-apiserver-k8s-master 1/1 Running 0 82m
kube-system kube-controller-manager-k8s-master 1/1 Running 0 82m
kube-system kube-proxy-b298g 1/1 Running 0 83m
kube-system kube-scheduler-k8s-master 1/1 Running 0 82m几分钟之后，所有容器就变成了running状态，可以进行下一步了：
[root@k8s-master ~]# kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system calico-node-nzqzl 2/2 Running 0 13m
kube-system coredns-fb8b8dccf-twknr 1/1 Running 0 95m
kube-system coredns-fb8b8dccf-wc4pd 1/1 Running 0 95m
kube-system etcd-k8s-master 1/1 Running 0 94m
kube-system kube-apiserver-k8s-master 1/1 Running 0 94m
kube-system kube-controller-manager-k8s-master 1/1 Running 0 94m
kube-system kube-proxy-b298g 1/1 Running 0 95m
kube-system kube-scheduler-k8s-master 1/1 Running 0 94mworker节点加入（本节在worker节点操作）
在master节点上查看当前集群的所有节点，只有master
kubectl get nodes[root@k8s-master ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8s-master Ready master 3m22s v1.15.0在worker节点上将Worker节点加入集群
在Worker节点上运行Kubernetes Master初始化时生成的令牌：
[root@k8s-node01 ~]# kubeadm join 192.168.0.8:6443 --token aakoqt.zq79agckuzc5wt29 \
--discovery-token-ca-cert-hash sha256:fa781d59e6e69d1a8abb836f66e3d36fd2d3e1765a1afdb71b1a18af807585c2[root@k8s-node01 ~]# kubeadm join 192.168.0.8:6443 --token aakoqt.zq79agckuzc5wt29 \
> --discovery-token-ca-cert-hash sha256:fa781d59e6e69d1a8abb836f66e3d36fd2d3e1765a1afdb71b1a18af807585c2
[preflight] Running pre-flight checks
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.15" ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Activating the kubelet service
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...
This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.
Run 'kubectl get nodes' on the control-plane to see this node join the cluster.
[root@k8s-node01 ~]# 在master节点上检查加入结果
再回到master节点上去查看所有node，发现就多了一个节点：
kubectl get nodes[root@k8s-master ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8s-master Ready master 7h7m v1.15.0
k8s-node01 Ready <none> 3m41s v1.15.0
k8s-node02 Ready <none> 7h1m v1.15.0补充：当在worker节点上刚刚执行完加入集群的令牌之后，中间有出现ContainerCreating的状态，等几分钟再看，就已经OK了
[root@k8s-master ~]# kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system calico-node-d4ksg 0/2 ContainerCreating 0 3m13s
kube-system calico-node-l6gsm 2/2 Running 0 31m
kube-system calico-node-lrszl 2/2 Running 0 31m
kube-system coredns-5c98db65d4-5tmc4 1/1 Running 0 113m
kube-system coredns-5c98db65d4-9srmt 1/1 Running 0 113m
kube-system etcd-k8s-master 1/1 Running 0 112m
kube-system kube-apiserver-k8s-master 1/1 Running 0 112m
kube-system kube-controller-manager-k8s-master 1/1 Running 0 112m
kube-system kube-proxy-dkpmj 1/1 Running 0 113m
kube-system kube-proxy-gm9nr 0/1 ContainerCreating 0 3m13s
kube-system kube-proxy-vswzq 0/1 Running 0 3m13s
kube-system kube-scheduler-k8s-master 1/1 Running 0 112m等几分钟查看pod状态，就OK了
[root@k8s-master ~]# kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system calico-node-d4ksg 2/2 Running 0 4m27s
kube-system calico-node-l6gsm 2/2 Running 2 7h5m
kube-system calico-node-lrszl 1/2 Running 0 7h2m
kube-system coredns-5c98db65d4-5tmc4 1/1 Running 1 7h7m
kube-system coredns-5c98db65d4-9srmt 1/1 Running 1 7h7m
kube-system etcd-k8s-master 1/1 Running 1 7h6m
kube-system kube-apiserver-k8s-master 1/1 Running 1 7h7m
kube-system kube-controller-manager-k8s-master 1/1 Running 1 7h7m
kube-system kube-proxy-dkpmj 1/1 Running 0 4m27s
kube-system kube-proxy-gm9nr 1/1 Running 0 7h2m
kube-system kube-proxy-vswzq 1/1 Running 1 7h7m
kube-system kube-scheduler-k8s-master 1/1 Running 1 7h7m再查看节点，node节点已经就是ready状态了：
[root@k8s-master ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8s-master Ready master 7h7m v1.15.0
k8s-node01 Ready <none> 3m41s v1.15.0
k8s-node02 Ready <none> 7h1m v1.15.0如节点一直处于NotReady状态，pod一直处于ContainerCreating状态，则使用kubectl describe pod检查，即可查询pod错误信息
错误修复完成后，需要在Node节点重启Docker
systemctl restart docker[root@k8s-master ~]# kubectl describe pod calico-node-2kw7x --namespace=kube-system
Name: calico-node-2kw7x
Namespace: kube-system
Priority: 0
PriorityClassName: <none>
Node: k8s-node01/192.168.0.10
Start Time: Tue, 11 Jun 2019 22:18:35 +0800
Labels: controller-revision-hash=7d4b8c9897
k8s-app=calico-node
pod-template-generation=1
Annotations: scheduler.alpha.kubernetes.io/critical-pod:
Status: Pending
IP: 192.168.0.10
Controlled By: DaemonSet/calico-node
Containers:
calico-node:
Container ID:
Image: calico/node:v3.3.6
Image ID:
Port: <none>
Host Port: <none>
State: Waiting
Reason: ContainerCreating
Ready: False
Restart Count: 0
Requests:
cpu: 250m
Liveness: http-get http://localhost:9099/liveness delay=10s timeout=1s period=10s #success=1 #failure=6
Readiness: exec [/bin/calico-node -bird-ready -felix-ready] delay=0s timeout=1s period=10s #success=1 #failure=3
Environment:
DATASTORE_TYPE: kubernetes
FELIX_TYPHAK8SSERVICENAME: <set to the key 'typha_service_name' of config map 'calico-config'> Optional: false
WAIT_FOR_DATASTORE: true
NODENAME: (v1:spec.nodeName)
CALICO_NETWORKING_BACKEND: <set to the key 'calico_backend' of config map 'calico-config'> Optional: false
CLUSTER_TYPE: k8s,bgp
IP: autodetect
CALICO_IPV4POOL_IPIP: Always
FELIX_IPINIPMTU: <set to the key 'veth_mtu' of config map 'calico-config'> Optional: false
CALICO_IPV4POOL_CIDR: 192.168.0.0/16
CALICO_DISABLE_FILE_LOGGING: true
FELIX_DEFAULTENDPOINTTOHOSTACTION: ACCEPT
FELIX_IPV6SUPPORT: false
FELIX_LOGSEVERITYSCREEN: info
FELIX_HEALTHENABLED: true
Mounts:
/lib/modules from lib-modules (ro)
/run/xtables.lock from xtables-lock (rw)
/var/lib/calico from var-lib-calico (rw)
/var/run/calico from var-run-calico (rw)
/var/run/secrets/kubernetes.io/serviceaccount from calico-node-token-8bjw5 (ro)
install-cni:
Container ID:
Image: calico/cni:v3.3.6
Image ID:
Port: <none>
Host Port: <none>
Command:
/install-cni.sh
State: Waiting
Reason: ContainerCreating
Ready: False
Restart Count: 0
Environment:
CNI_CONF_NAME: 10-calico.conflist
KUBERNETES_NODE_NAME: (v1:spec.nodeName)
CNI_NETWORK_CONFIG: <set to the key 'cni_network_config' of config map 'calico-config'> Optional: false
CNI_MTU: <set to the key 'veth_mtu' of config map 'calico-config'> Optional: false
Mounts:
/host/etc/cni/net.d from cni-net-dir (rw)
/host/opt/cni/bin from cni-bin-dir (rw)
/var/run/secrets/kubernetes.io/serviceaccount from calico-node-token-8bjw5 (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
lib-modules:
Type: HostPath (bare host directory volume)
Path: /lib/modules
HostPathType:
var-run-calico:
Type: HostPath (bare host directory volume)
Path: /var/run/calico
HostPathType:
var-lib-calico:
Type: HostPath (bare host directory volume)
Path: /var/lib/calico
HostPathType:
xtables-lock:
Type: HostPath (bare host directory volume)
Path: /run/xtables.lock
HostPathType: FileOrCreate
cni-bin-dir:
Type: HostPath (bare host directory volume)
Path: /opt/cni/bin
HostPathType:
cni-net-dir:
Type: HostPath (bare host directory volume)
Path: /etc/cni/net.d
HostPathType:
calico-node-token-8bjw5:
Type: Secret (a volume populated by a Secret)
SecretName: calico-node-token-8bjw5
Optional: false
QoS Class: Burstable
Node-Selectors: beta.kubernetes.io/os=linux
Tolerations: :NoSchedule
:NoExecute
CriticalAddonsOnly
node.kubernetes.io/disk-pressure:NoSchedule
node.kubernetes.io/memory-pressure:NoSchedule
node.kubernetes.io/network-unavailable:NoSchedule
node.kubernetes.io/not-ready:NoExecute
node.kubernetes.io/pid-pressure:NoSchedule
node.kubernetes.io/unreachable:NoExecute
node.kubernetes.io/unschedulable:NoSchedule
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedCreatePodSandBox 3m24s (x1445 over 11h) kubelet, k8s-node01 Failed create pod sandbox: rpc error: code = Unknown desc = failed pulling image "k8s.gcr.io/pause:3.1": Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
[root@k8s-master ~]#k8s的pod默认不会调度到master节点，如果部署的是单节点的集群，就需要按照下面的方式修改pod的调度策略（https://v1-12.docs.kubernetes.io/docs/setup/independent/create-cluster-kubeadm/#pod-network）：
部署dashboard
关于dashboard的介绍和部署方式可参考：https://kubernetes.io/docs/tasks/access-application-cluster/web-ui-dashboard/#accessing-the-dashboard-ui
这里部署的是v1.10.1版本
注意：dashboard务必部署在master节点
下载部署dashboard的yaml文件到本地并修改拉取镜像地址
由于yaml配置文件中指定镜像从google拉取，先下载yaml文件到本地，修改配置从阿里云仓库拉取镜像。
（1）https模式
注意必须提供证书！
[root@k8smaster ~]# wget https://raw.githubusercontent.com/kubernetes/dashboard/master/aio/deploy/recommended/kubernetes-dashboard.yaml（2）http模式
[root@k8smaster ~]# wget https://raw.githubusercontent.com/kubernetes/dashboard/master/aio/deploy/alternative/kubernetes-dashboard.yaml（3）v1.10.1版本
# wget https://raw.githubusercontent.com/kubernetes/dashboard/v1.10.1/src/deploy/alternative/kubernetes-dashboard.yaml修改114行拉取镜像地址为阿里云的地址：
image: registry.cn-hangzhou.aliyuncs.com/google_containers/kubernetes-dashboard-amd64:v1.10.1template:
metadata:
labels:
k8s-app: kubernetes-dashboard
spec:
containers:
- name: kubernetes-dashboard
#image: k8s.gcr.io/kubernetes-dashboard-amd64:v2.0.0-alpha0
image: kubernetesdashboarddev/kubernetes-dashboard-amd64:v2.0.0-alpha0
ports:
- containerPort: 9090
protocol: TCP部署dashboard
kubectl create -f kubernetes-dashboard.yaml[root@k8s-master ~]# kubectl create -f kubernetes-dashboard.yaml
secret/kubernetes-dashboard-csrf created
serviceaccount/kubernetes-dashboard created
role.rbac.authorization.k8s.io/kubernetes-dashboard-minimal created
rolebinding.rbac.authorization.k8s.io/kubernetes-dashboard-minimal created
deployment.apps/kubernetes-dashboard created
service/kubernetes-dashboard created查看Pod 的状态为running说明dashboard已经部署成功
kubectl get pods --all-namespaces
kubectl get pod --namespace=kube-system -o wide | grep dashboard[root@k8s-master ~]# kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system calico-node-2kw7x 1/2 Running 0 12h
kube-system calico-node-nzqzl 1/2 Running 2 13h
kube-system coredns-fb8b8dccf-twknr 1/1 Running 1 14h
kube-system coredns-fb8b8dccf-wc4pd 1/1 Running 1 14h
kube-system etcd-k8s-master 1/1 Running 1 14h
kube-system kube-apiserver-k8s-master 1/1 Running 1 14h
kube-system kube-controller-manager-k8s-master 1/1 Running 1 14h
kube-system kube-proxy-b298g 1/1 Running 1 14h
kube-system kube-proxy-srcdt 1/1 Running 0 12h
kube-system kube-scheduler-k8s-master 1/1 Running 1 14h
kube-system kubernetes-dashboard-5d9599dc98-h4gpr 1/1 Running 0 74s
[root@k8s-master ~]# kubectl get pod --namespace=kube-system -o wide | grep dashboard
kubernetes-dashboard-5d9599dc98-h4gpr 1/1 Running 0 54s 172.16.0.6 k8s-master <none> <none>同时，Dashboard会在kube-system namespace中创建自己的Deployment和Service：
kubectl get deployment kubernetes-dashboard --namespace=kube-system
kubectl get service kubernetes-dashboard --namespace=kube-system[root@k8s-master ~]# kubectl get deployment kubernetes-dashboard --namespace=kube-system
NAME READY UP-TO-DATE AVAILABLE AGE
kubernetes-dashboard 1/1 1 1 2m26s
[root@k8s-master ~]# kubectl get service kubernetes-dashboard --namespace=kube-system
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes-dashboard ClusterIP 10.104.31.113 <none> 443/TCP 2m35s配置使用nodeport方式访问dashport
访问dashboard的方式有很多，这里使用的是配置nodeport的方式来访问。
修改配置文件
修改文件kubernetes-dashboard.yaml，将service type和nodeport添加进去，注意k8s只支持30000以上的端口
[root@k8s-master ~]# vim kubernetes-dashboard.yaml #加入下面两行配置# ------------------- Dashboard Service ------------------- #
kind: Service
apiVersion: v1
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard
namespace: kube-system
spec:
type: NodePort #添加Service的type为NodePort
ports:
- port: 80
targetPort: 9090
nodePort: 30006 # 添加映射到虚拟机的端口，k8s只支持30000以上的端口，端口自定义
selector:
k8s-app: kubernetes-dashboard修改后，重新应用配置文件
kubectl apply -f kubernetes-dashboard.yaml[root@k8s-master ~]# kubectl apply -f kubernetes-dashboard.yaml
Warning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply
secret/kubernetes-dashboard-certs configured
Warning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply
secret/kubernetes-dashboard-csrf configured
Warning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply
serviceaccount/kubernetes-dashboard configured
Warning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply
role.rbac.authorization.k8s.io/kubernetes-dashboard-minimal configured
Warning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply
rolebinding.rbac.authorization.k8s.io/kubernetes-dashboard-minimal configured
Warning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply
deployment.apps/kubernetes-dashboard configured
Warning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply
service/kubernetes-dashboard configured端口已经变成30006
kubectl get service -n kube-system | grep dashboard[root@k8s-master ~]# kubectl get service -n kube-system | grep dashboard
kubernetes-dashboard NodePort 10.104.31.113 <none> 443:30006/TCP 9m20s获取登录dashboard的token
kubectl -n kube-system describe $(kubectl -n kube-system get secret -n kube-system -o name |grep namespace) | grep token[root@k8s-master ~]# kubectl get service -n kube-system | grep dashboard
kubernetes-dashboard NodePort 10.110.148.107 <none> 80:30006/TCP 3m28s
[root@k8s-master ~]# kubectl -n kube-system describe $(kubectl -n kube-system get secret -n kube-system -o name |grep namespace) | grep token
Name: namespace-controller-token-hd6z8
Type: kubernetes.io/service-account-token
token: eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJuYW1lc3BhY2UtY29udHJvbGxlci10b2tlbi1oZDZ6OCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJuYW1lc3BhY2UtY29udHJvbGxlciIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6ImRiNzY4YTFhLTE5NmQtNDNmNC1hYjVhLWRjYWJiZTMzMmU4NyIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDprdWJlLXN5c3RlbTpuYW1lc3BhY2UtY29udHJvbGxlciJ9.iW30zjU4md6AVQFsI7AEpIpkQRe9tWw8EH5oilYo4l7CZIQ24CaWe4OvqrssyLDttP4aCqv3-DE3YPbLo8bZIsp73cKiBytfFNBHVPiO0S4eR-IQFvJNfxwTj_xDK4ksBbxS0BDVIOU6TyYWDCbXuKXn35szPwcSDylE3H2FE_L8ZnvIMQHGi-gtTSNYDmyILqgSfYpb7tA-rGcbcvdsA44BC0xCH7ELFRFFKezi_rTq4JoW5HB5Z9SDSstU18h7UuPj8NXTS1j1IFtL8Xt1CEZrKf7yEZEKFdNtvwK7t1vDTMQDn8Bhi7mN2qd3WRf9QXRTN73sTfPJlWPdzK_LxA使用nodeport方式访问dashboard
访问地址：http://nodeportIP:nodeport
登录地址：http://119.29.172.118:30006/#!/login
版权声明：
作者：Joe.Ye
链接：https://www.appblog.cn/index.php/2023/03/23/build-k8s-cluster-using-kubeadm-method-in-centos-7-system/
来源：APP全栈技术分享
文章版权归作者所有，未经允许请勿转载。
THE END
0
二维码
打赏
海报
Centos 7系统下Kubeadm方式搭建k8s集群
Kubernetes集群部署方式
Minikube方式
Minikube是一个工具，可以在本地快速运行一个单点的Kubernetes，适合尝试Kubernetes或日常开发的用户使用，但是不能用……
文章目录
关闭
共有 0 条评论