本文共 7538 字,大约阅读时间需要 25 分钟。
本次实验选择5台主机,3台作为master主机,2台作为node节点
节点ip | OS版本 | hostname -f | 安装软件 |
---|---|---|---|
192.168.0.1 | RHEL7.4 | k8s-master01 | docker,etcd,flanneld,kube-apiserver,kube-controller-manager,kube-scheduler |
192.168.0.2 | RHEL7.4 | k8s-master02 | docker,etcd,flanneld,kube-apiserver,kube-controller-manager,kube-scheduler |
192.168.0.3 | RHEL7.4 | k8s-master03 | docker,etcd,flanneld,kube-apiserver,kube-controller-manager,kube-scheduler |
192.168.0.4 | RHEL7.4 | k8s-node01 | docker,flanneld,kubelet,kube-proxy |
192.168.0.5 | RHEL7.4 | k8s-node02 | docker,flanneld,kubelet,kube-proxy |
kubernetes使用Flannel实现集群内各节点能通过Pod 网段互联互通
etcd 集群启用了双向TLS 认证,所以需要为flanneld 指定与etcd 集群通信的CA 和密钥。
创建flanneld 证书签名请求:cat > flanneld-csr.json <<EOF
{ "CN": "flanneld", "hosts": [], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "BeiJing", "L": "BeiJing", "O": "k8s", "OU": "System" } ] } EOF
# cfssl gencert -ca=/k8s/kubernetes/ssl/ca.pem -ca-key=/k8s/kubernetes/ssl/ca-key.pem -config=/k8s/kubernetes/ssl/ca-config.json -profile=kubernetes flanneld-csr.json | cfssljson -bare flanneld
# ls flanneld*
flanneld.csr flanneld-csr.json flanneld-key.pem flanneld.pem
# ETCDCTL_API=2 /k8s/etcd/bin/etcdctl --endpoints="https://192.168.0.3:2379,https://192.168.0.2:2379,https://192.168.0.1:2379" --ca-file=/k8s/kubernetes/ssl/ca.pem --cert-file=/k8s/flanneld/ssl/flanneld.pem --key-file=/k8s/flanneld/ssl/flanneld-key.pem set /kubernetes/network/config '{ "Network": "100.100.0.0/16", "Backend": {"Type": "vxlan"}}'
输出信息:
{ "Network": "100.100.0.0/16", "Backend": {"Type": "vxlan"}}
提示:
wget https://github.com/coreos/flannel/releases/download/v0.11.0/flannel-v0.11.0-linux-amd64.tar.gz
tar xf flannel-v0.10.0-linux-amd64.tar.gz mv flanneld mk-docker-opts.sh /k8s/kubernetes/bin/
cat << EOF > /lib/systemd/system/flanneld.service
[Unit] Description=Flanneld overlay address etcd agent After=network.target After=network-online.target Wants=network-online.target After=etcd.service Before=docker.service[Service]
Type=notify ExecStart=/k8s/kubernetes/bin/flanneld \ --etcd-cafile=/k8s/kubernetes/ssl/ca.pem \ --etcd-certfile=/k8s/flanneld/ssl/flanneld.pem \ --etcd-keyfile=/k8s/flanneld/ssl/flanneld-key.pem \ --etcd-endpoints=https://192.168.0.3:2379,https://192.168.0.2:2379,https://192.168.0.1:2379 \ --etcd-prefix=/kubernetes/network ExecStartPost=/k8s/kubernetes/bin/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/docker Restart=on-failure[Install]
WantedBy=multi-user.target RequiredBy=docker.service EOF
mk-docker-opts.sh脚本将分配给flanneld 的Pod 子网网段信息写入到/run/flannel/docker 文件中,后续docker 启动时使用这个文件中的参数值为 docker0 网桥
flanneld 使用系统缺省路由所在的接口和其他节点通信,对于有多个网络接口的机器(内网和公网),可以用 --iface 选项值指定通信接口(上面的 systemd unit 文件没指定这个选项)cat << EOF > /lib/systemd/system/docker.service
[Unit] Description=Docker Application Container Engine Documentation=https://docs.docker.com BindsTo=containerd.service After=network-online.target firewalld.service Wants=network-online.target Requires=docker.socket[Service]
Type=notify # the default is not to use systemd for cgroups because the delegate issues still # exists and systemd currently does not support the cgroup feature set required # for containers run by docker EnvironmentFile=/run/flannel/docker ExecStart=/usr/bin/dockerd -H fd:// ExecReload=/bin/kill -s HUP $MAINPID TimeoutSec=0 RestartSec=2 Restart=always# Note that StartLimit* options were moved from "Service" to "Unit" in systemd 229.
# Both the old, and new location are accepted by systemd 229 and up, so using the old location # to make them work for either version of systemd. StartLimitBurst=3# Note that StartLimitInterval was renamed to StartLimitIntervalSec in systemd 230.
# Both the old, and new name are accepted by systemd 230 and up, so using the old name to make # this option work for either version of systemd. StartLimitInterval=60s# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting. LimitNOFILE=infinity LimitNPROC=infinity LimitCORE=infinity# Comment TasksMax if your systemd version does not supports it.
# Only systemd 226 and above support this option. TasksMax=infinity# set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes# kill only the docker process, not all processes in the cgroup
KillMode=process[Install]
WantedBy=multi-user.target EOF
systemctl daemon-reload
systemctl enable flanneld systemctl start flanneld systemctl restart docker
# 查看flannel.1
ifconfig flannel.1 flannel.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1450 inet 100.100.31.0 netmask 255.255.255.255 broadcast 0.0.0.0 inet6 fe80::9c42:ecff:fe23:8885 prefixlen 64 scopeid 0x20<link> ether 9e:42:ec:23:88:85 txqueuelen 0 (Ethernet) RX packets 0 bytes 0 (0.0 B) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 0 bytes 0 (0.0 B) TX errors 0 dropped 55 overruns 0 carrier 0 collisions 0# 查看集群 Pod 网段(/16)
# ETCDCTL_API=2 /k8s/etcd/bin/etcdctl --endpoints="https://192.168.0.3:2379,https://192.168.0.2:2379,https://192.168.0.1:2379" --ca-file=/k8s/kubernetes/ssl/ca.pem --cert-file=/k8s/flanneld/ssl/flanneld.pem --key-file=/k8s/flanneld/ssl/flanneld-key.pem get /kubernetes/network/config { "Network": "100.100.0.0/16", "Backend": {"Type": "vxlan"}}# 查看已分配的 Pod 子网段列表(/24)
# ETCDCTL_API=2 /k8s/etcd/bin/etcdctl --endpoints="https://192.168.0.3:2379,https://192.168.0.2:2379,https://192.168.0.1:2379" --ca-file=/k8s/kubernetes/ssl/ca.pem --cert-file=/k8s/flanneld/ssl/flanneld.pem --key-file=/k8s/flanneld/ssl/flanneld-key.pem ls /kubernetes/network/subnets /kubernetes/network/subnets/100.100.31.0-24# 查看某一 Pod 网段对应的 flanneld 进程监听的 IP 和网络参数
# ETCDCTL_API=2 /k8s/etcd/bin/etcdctl --endpoints="https://192.168.0.3:2379,https://192.168.0.2:2379,https://192.168.0.1:2379" --ca-file=/k8s/kubernetes/ssl/ca.pem --cert-file=/k8s/flanneld/ssl/flanneld.pem --key-file=/k8s/flanneld/ssl/flanneld-key.pem get /kubernetes/network/subnets/100.100.31.0-24 {"PublicIP":"192.168.0.2","BackendType":"vxlan","BackendData":{"VtepMAC":"9e:42:ec:23:88:85"}}
scp -r /k8s/kubernetes/bin/flanneld 192.168.0.2:/k8s/kubernetes/bin/
scp -r /k8s/kubernetes/bin/flanneld 192.168.0.3:/k8s/kubernetes/bin/ scp -r /k8s/kubernetes/bin/flanneld 192.168.0.4:/k8s/kubernetes/bin/ scp -r /k8s/kubernetes/bin/flanneld 192.168.0.5:/k8s/kubernetes/bin/ scp -r /k8s/flanneld/ssl/* 192.168.0.2:/k8s/flanneld/ssl/ scp -r /k8s/flanneld/ssl/* 192.168.0.3:/k8s/flanneld/ssl/ scp -r /k8s/flanneld/ssl/* 192.168.0.4:/k8s/flanneld/ssl/ scp -r /k8s/flanneld/ssl/* 192.168.0.5:/k8s/flanneld/ssl/ scp /lib/systemd/system/flanneld.service 192.168.0.2:/lib/systemd/system/flanneld.service scp /lib/systemd/system/flanneld.service 192.168.0.3:/lib/systemd/system/flanneld.service scp /lib/systemd/system/flanneld.service 192.168.0.4:/lib/systemd/system/flanneld.service scp /lib/systemd/system/flanneld.service 192.168.0.5:/lib/systemd/system/flanneld.service scp /lib/systemd/system/docker.service 192.168.0.2:/lib/systemd/system/docker.service scp /lib/systemd/system/docker.service 192.168.0.3:/lib/systemd/system/docker.service scp /lib/systemd/system/docker.service 192.168.0.4:/lib/systemd/system/docker.service scp /lib/systemd/system/docker.service 192.168.0.5:/lib/systemd/system/docker.service
转载地址:http://takpi.baihongyu.com/