环境说明#

部署一套 单Master + 单Worker 集群。

开始想的是用minikube，但是局限性有太多，网上的教程有点乱，还是想着自己折腾一下。

正好有个核心网+K8S的项目是采用多节点方式部署，这里就使用kubeadm方式来部署了，采用为了熟悉命令，不使用dashboard。

必须安装containerd、kubelet、kubeadm、kubectl
crictl查看、拉取、删除镜像，containerd的操作工具

所有节点虚拟机参数如下：

系统：ubuntu-24.04.4-live-server-amd64；
VMware部署，单处理器8核，16G运存，80G磁盘；
单网卡做NAT

Master单节点部署#

k8s安装，这里安装的是 1.28 版本，apt源使用阿里源：

1
hostnamectl set-hostname k8s-master1
2
echo “192.168.24.130 k8s-master1” >> /etc/hosts
3

4
swapoff -a
5
sed -i '/swap/s/^/#/' /etc/fstab
6

7
ufw disable
8

9
# 调整网络参数
10
cat <<EOF | tee /etc/modules-load.d/k8s.conf
11
overlay
12
br_netfilter
13
net.bridge.bridge-nf-call-iptables  = 1
14
net.bridge.bridge-nf-call-ip6tables = 1
15
net.ipv4.ip_forward                 = 1
16
EOF
17

18
# 安装配置containerd（替代docker）
19
apt install -y containerd
20
mkdir -p /etc/containerd
21
containerd config default > /etc/containerd/config.toml
22
sed -i 's/SystemdCgroup = false/SystemdCgroup = true/g' /etc/containerd/config.toml
23
systemctl restart containerd
24
systemctl enable containerd
25
systemctl status containerd
26

27
# 安装k8s https://developer.aliyun.com/mirror/kubernetes
28
apt-get update && apt-get install -y apt-transport-https
29
curl -fsSL https://mirrors.aliyun.com/kubernetes-new/core/stable/v1.28/deb/Release.key |
30
    gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
31
echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://mirrors.aliyun.com/kubernetes-new/core/stable/v1.28/deb/ /" |
32
    tee /etc/apt/sources.list.d/kubernetes.list
33
apt-get update
34
apt-get install -y kubelet kubeadm kubectl

到这里container和k8s组件已经基本部署完成，接下来开始初始化k8s：

踩坑记录
这里使用的阿里云的镜像仓库，因此还需要修改container的配置文件。

1
sed -i 's|sandbox_image = .*|sandbox_image = "registry.aliyuncs.com/google_containers/pause:3.9"|' /etc/containerd/config.toml
2
systemctl restart containerd
3

4
kubeadm init \
5
  --apiserver-advertise-address=192.168.24.130 \
6
  --image-repository registry.aliyuncs.com/google_containers \
7
  --kubernetes-version v1.28.4 \
8
  --service-cidr=10.96.0.0/12 \
9
  --pod-network-cidr=172.30.0.0/16

这时候通过kubectl get nodes看主节点是连接失败的，需要配置其他组件：

1
kubectl get nodes
2
# E0405 15:48:47.825576   45541 memcache.go:265] couldn't get current server API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp 127.0.0.1:8080: connect: connection refused
3
# The connection to the server localhost:8080 was refused - did you specify the right host or port?
4

5

6
# 给权限
7
mkdir -p $HOME/.kube
8
cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
9
chown $(id -u):$(id -g) $HOME/.kube/config
10
kubectl get nodes
11
#NAME          STATUS     ROLES           AGE   VERSION
12
#k8s-master1   NotReady   control-plane   17m   v1.28.15
13

14
# 网络插件，可以理解为k8s集群中负责通信的网络设备，这里装calico
15
kubectl apply -f https://raw.githubusercontent.com/projectcalico/calico/v3.31.4/manifests/calico.yaml
16

17
# 查看整体
18
kubectl get pods -n kube-system  &&  kubectl get nodes
19
# NAME                                       READY   STATUS    RESTARTS   AGE
20
# calico-kube-controllers-66f8b6cf45-h7cmw   1/1     Running   0          5m21s
21
# calico-node-fzrsr                          1/1     Running   0          5m21s
22
# coredns-66f779496c-4j4w7                   1/1     Running   0          25m
23
# coredns-66f779496c-njdsz                   1/1     Running   0          25m
24
# etcd-k8s-master1                           1/1     Running   1          25m
25
# kube-apiserver-k8s-master1                 1/1     Running   1          25m
26
# kube-controller-manager-k8s-master1        1/1     Running   1          25m
27
# kube-proxy-bl7zf                           1/1     Running   0          25m
28
# kube-scheduler-k8s-master1                 1/1     Running   1          25m
29
# NAME          STATUS   ROLES           AGE   VERSION
30
# k8s-master1   Ready    control-plane   25m   v1.28.15

到这里 Master 节点已经部署完成了，现在处理工作（Worker）节点。

Worker单节点部署#

首先在master1节点上生成worker加入的密钥：

1
kubeadm token create --print-join-command
2
# kubeadm join 192.168.24.130:6443 --token 199087.5d6po9e0znrwbwkr --discovery-token-ca-cert-hash sha256:90f8f15114f42eebc9625b34a70383b4fb80eb979495aaba22300ebdd95c6c2a

worker节点使用之前部署好containerd和K8s组件的快照，链接克隆就行，修改主机名为k8s-worker1：

1
# worker1
2
hostnamectl set-hostname k8s-worker1
3
# /etc/hosts同步修改
4

5
# 重置一下
6
kubeadm reset -f
7
rm -rf $HOME/.kube
8
rm -rf /var/lib/kubelet
9
rm -rf /etc/kubernetes
10

11
# 贴上生成的密钥
12
kubeadm join 192.168.24.130:6443 --token 199087.5d6po9e0znrwbwkr --discovery-token-ca-cert-hash sha256:90f8f15114f42eebc9625b34a70383b4fb80eb979495aaba22300ebdd95c6c2a
13
# This node has joined the cluster:
14
# ...

TIP
containerd配置文件和master1相同，不然纳管的时候抓取calico和proxy镜像时可能一直处于init状态

出现以上字段就说明加入没问题，这时候回到master1上查看节点状态：

1
kubectl get pods -n kube-system -o wide  | grep -i worker1
2
# NAME                                       READY   STATUS    RESTARTS      AGE   IP                NODE          NOMINATED NODE   READINESS GATES
3
# calico-node-5ddj4                          1/1     Running   0             12m   192.168.24.131    k8s-worker1   <none>           <none>
4
# kube-proxy-5qh7f                           1/1     Running   0             12m   192.168.24.131    k8s-worker1   <none>           <none>
5
kubectl get nodes
6
# NAME          STATUS   ROLES           AGE   VERSION
7
# k8s-master1   Ready    control-plane   17h   v1.28.15
8
# k8s-worker1   Ready    <none>          11m   v1.28.15

worker1的calico和proxy网络组件处于running，且worker1 node状态为ready，代表worker1已经加入到集群了。

tips 一些其他踩坑记录：#

crictl images显示有报错#

crictl images显示有报错，没有指定用containerd，解决：

1
# WARN[0000] image connect using default endpoints...ERRO[0000] validate service connection: unix:///var/run/dockershim.sock: no such file or directory
2

3
# crictl连接containerd
4
cat > /etc/crictl.yaml <<'EOF'
5
runtime-endpoint: unix:///run/containerd/containerd.sock
6
image-endpoint: unix:///run/containerd/containerd.sock
7
timeout: 10
8
debug: false
9
EOF
10

11
crictl images
12
# IMAGE                                                             TAG                 IMAGE ID            SIZE
13
# quay.io/calico/cni                                                v3.31.4             c433a27dd94ce       72.2MB
14
# quay.io/calico/kube-controllers                                   v3.31.4             ff033cc89dab5       54MB
15
# quay.io/calico/node                                               v3.31.4             e6536b93706ed       160MB
16
# registry.aliyuncs.com/google_containers/coredns                   v1.10.1             ead0a4a53df89       16.2MB
17
# registry.aliyuncs.com/google_containers/etcd                      3.5.15-0            2e96e5913fc06       56.9MB
18
# registry.aliyuncs.com/google_containers/kube-apiserver            v1.28.15            9dc6939e7c573       34.4MB
19
# registry.aliyuncs.com/google_containers/kube-apiserver            v1.28.4             7fe0e6f37db33       34.7MB
20
# registry.aliyuncs.com/google_containers/kube-controller-manager   v1.28.15            10541d8af03f4       33.3MB
21
# registry.aliyuncs.com/google_containers/kube-controller-manager   v1.28.4             d058aa5ab969c       33.4MB
22
# registry.aliyuncs.com/google_containers/kube-proxy                v1.28.15            ba6d7f8bc25be       28.3MB
23
# registry.aliyuncs.com/google_containers/kube-proxy                v1.28.4             83f6cc407eed8       24.6MB
24
# registry.aliyuncs.com/google_containers/kube-scheduler            v1.28.15            9d3465f8477c6       18.5MB
25
# registry.aliyuncs.com/google_containers/kube-scheduler            v1.28.4             e3db313c6dbc0       18.8MB
26
# registry.aliyuncs.com/google_containers/pause                     3.9                 e6f1816883972       322kB

crictl镜像拉取问题#

crictl在拉取镜像时出现镜像拉取超时，同时报dns解析失败，如图所示： dns

这里首先得配置container镜像加速地址，/etc/containerd/config.toml 修改镜像加速配置：

1
      [plugins."io.containerd.grpc.v1.cri".registry.mirrors]
2
        [plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
3
          endpoint = ["https://docker.m.daocloud.io"]

1
systemctl restart containerd

会发现解析就是上面的报错（之前用的阿里云加速，半天找不到能用的），这里是使用的/etc/resolv.conf的配置，这个解析很怪，修改后重启还是127.0.0.53，修改了/etc/systemd/resolved.conf也是不行，就很迷惑ubuntu这个操作。

直接把/etc/resolv.conf删了重新加个：

1
systemctl stop systemd-resolved
2
systemctl disable systemd-resolved
3
rm -f /etc/resolv.conf
4
echo "nameserver 223.5.5.5" > /etc/resolv.conf
5
systemctl restart containerd
6

7
# 看下还用没用127.0.0.53，没用就正常了
8
dig www.baidu.com
9

10
# 再pull一下
11
crictl pull nginx
12
# Image is up to date for sha256:0cf1d6af5ca72e2ca196afdbdbe26d96f141bd3dc14d70210707cf89032ea217
13
crictl images
14
# IMAGE                                                             TAG                 IMAGE ID            SIZE
15
# docker.io/library/nginx                                           latest              0cf1d6af5ca72       63MB

这下才拉下来-=，有点恶心