文章/答案/技术大牛

发布

社区首页 >专栏 >鲲鹏+欧拉部署KubeSphere3.4

鲲鹏+欧拉部署KubeSphere3.4

编码如写诗

发布于 2024-08-30 03:29:53

27700

代码可运行

文章被收录于专栏：编码如写诗编码如写诗

运行总次数：0

代码可运行

1.安装docker

1.1 下载docker离线包

地址：https://download.docker.com/linux/static/stable/

进入aarch64目录，下载需要的docker版本

下载后上传至服务器解压

tar -xvf docker-20.10.7.tgz
# 将 docker 文件移动到 /usr/bin 目录下
cp -p docker/* /usr/bin

1.2 编写docker.service

vim docker.service

内容如下：


[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network-online.target firewalld.service
Wants=network-online.target

[Service]
Type=notify
# the default is not to use systemd for cgroups because the delegate issues still
# exists and systemd currently does not support the cgroup feature set required
# for containers run by docker
ExecStart=/usr/bin/dockerd
ExecReload=/bin/kill -s HUP $MAINPID
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
# Uncomment TasksMax if your systemd version supports it.
# Only systemd 226 and above support this version.
#TasksMax=infinity
TimeoutStartSec=0
# set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes
# kill only the docker process, not all processes in the cgroup
KillMode=process
# restart the docker process if it exits prematurely
Restart=on-failure
StartLimitBurst=3
StartLimitInterval=60s

[Install]
WantedBy=multi-user.target

添加到系统服务

# 将 docker.service 移到 /etc/systemd/system/ 目录
cp docker.service /etc/systemd/system/
# 设置 docker.service 文件权限
chmod +x /etc/systemd/system/docker.service

# 重新加载配置文件
systemctl daemon-reload

# 启动docker
systemctl start docker

# 设置 docker 开机自启
systemctl enable docker.service

1.3 验证docker的安装

2.安装KS

服务器配置 (华为云服务器)

主机名	IP	CPU	内存
master-1	192.168.0.115	4	8
master-2	192.168.0.150	4	8
master-3	192.168.0.58	4	8

2.1下载KK

在master节点操作(后续没有特殊说明都是在此节点)下载最新版的 KubeKey

export KKZONE=cn

curl -sfL https://get-kk.kubesphere.io | sh -

下载后查看kk版本,可以看到版本信息以及架构

[root@node1 kubekey]# ./kk version
kk version: &version.Info{Major:"3", Minor:"0", GitVersion:"v3.0.13-dirty", GitCommit:"ac75d3ef3c22e6a9d999dcea201234d6651b3e72", GitTreeState:"dirty", BuildDate:"2023-11-07T08:43:42Z", GoVersion:"go1.19.2", Compiler:"gc", Platform:"linux/arm64"}
[root@node1 kubekey]#

查看 KubeKey 支持的 Kubernetes 版本列表

./kk version --show-supported-k8s

k8s 1.24后不再直接使用docker管理容器，我们为方便继续使用docker减少研发和运维工作量，使用1.22.12版本k8s

2.2创建k8s配置文件

./kk create config -f kubesphere-v340-v12212.yaml --with-kubernetes v1.22.12 --with-kubesphere v3.4.0

创建后编辑配置文件 kubesphere-v340-v12212.yaml，修改节点信息,hosts节点信息中添加arch为arm64

修改k8s中的containerManager为docker

apiVersion: kubekey.kubesphere.io/v1alpha2
kind: Cluster
metadata:
  name: sample
spec:
  hosts:
  - {name: node1, address: 192.168.0.115, internalAddress: 192.168.0.115, user: root, password: "123456", arch: arm64}
  - {name: node2, address: 192.168.0.150, internalAddress: 192.168.0.150, user: root, password: "123456", arch: arm64}
  - {name: node3, address: 192.168.0.58, internalAddress: 192.168.0.58, user: root, password: "123456", arch: arm64}
  roleGroups:
    etcd:
    - node1
    - node2
    - node3
    control-plane:
    - node1
    - node2
    - node3
    worker:
    - node1
    - node2
    - node3
  controlPlaneEndpoint:
    ## Internal loadbalancer for apiservers 
    internalLoadbalancer: haproxy

    domain: lb.kubesphere.local
    address: ""
    port: 6443
  kubernetes:
    version: v1.22.12
    clusterName: cluster.local
    autoRenewCerts: true
    containerManager: docker
  etcd:
    type: kubekey
  network:
    plugin: calico
    kubePodsCIDR: 10.233.64.0/18
    kubeServiceCIDR: 10.233.0.0/18
    ## multus support. https://github.com/k8snetworkplumbingwg/multus-cni
    multusCNI:
      enabled: false
  storage:
    openebs:
      basePath: /data/openebs/local # base path of the local PV provisioner
  registry:
    privateRegistry: ""
    namespaceOverride: ""
    registryMirrors: []
    insecureRegistries: []
  addons: []

2.3服务器基础配置和初始化

将kk拷贝到各节点服务器，执行

./kk init os

2.4部署

export KKZONE=cn
./kk create cluster -f kubesphere-v340-v12212.yaml

输入yes后开始执行安装部署

安装过程日志输出比较多，本文只展示重要的一点，一定要观察下载的二进制包，格式为 arm64（篇幅所限，只展示了部分日志输出）

Continue this installation? [yes/no]: yes
16:26:32 CST success: [LocalHost]
16:26:32 CST [NodeBinariesModule] Download installation binaries
16:26:32 CST message: [localhost]
downloading arm64 kubeadm v1.26.5 ...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 43.3M  100 43.3M    0     0  1016k      0  0:00:43  0:00:43 --:--:-- 1065k

实际部署时如果不手工干预，必然会出现下面的情况 :

clusterconfiguration.installer.kubesphere.io/ks-installer created
18:13:51 CST skipped: [ksp-master-3]
18:13:51 CST skipped: [ksp-master-2]
18:13:51 CST success: [ksp-master-1]
Please wait for the installation to complete:  <---<<
20:13:51 CST skipped: [ksp-master-3]
20:13:51 CST skipped: [ksp-master-2]
20:13:51 CST failed: [ksp-master-1]
error: Pipeline[CreateClusterPipeline] execute failed: Module[CheckResultModule] exec failed:
failed: [ksp-master-1] execute task timeout, Timeout=2h

当出现等待安装完成的滚动条后，应该立即新开终端，使用 kubectl get pod -A 关注所有 Pod 的部署情况。

根据报错信息，参考下文的「异常组件解决方案」及时排查解决创建和启动异常的组件。

3. 异常组件及解决方案

此处需要使用kubectl get pods -A查看服务部署在哪个节点，并到该节点重新拉取镜像。为高可用实现，建议每个节点都操作一遍。

3.1 解决 metrics-server 异常

这个异常是第一个要处理的，在 kk 部署任务的最后，出现等待部署完成的任务滚动条时就要关注，如果该 Pod 出现异常需要立即处理。

获取适配的 ARM 版镜像（相同版本 KubeSphere 官方 ARM 版镜像）

# 拉取 arm64 镜像

docker pull registry.cn-beijing.aliyuncs.com/kubesphereio/metrics-server:v0.4.2-arm64  --platform arm64

镜像重新打 tag（可不做 ）

# 删除 amd64 镜像
#docker rmi registry.cn-beijing.aliyuncs.com/kubesphereio/metrics-server:v0.4.2

# 重新打 tag
# docker tag registry.cn-beijing.aliyuncs.com/kubesphereio/metrics-server:v0.4.2-arm64 registry.cn-beijing.aliyuncs.com/kubesphereio/metrics-server:v0.4.2

重新部署组件

# 修改 Deployment 使用的镜像，并重启
kubectl set image deployment/metrics-server metrics-server=registry.cn-beijing.aliyuncs.com/kubesphereio/metrics-server:v0.4.2-arm64 -n kube-system
kubectl rollout restart deployment/metrics-server -n kube-system

3.2 解决 Argo CD 异常

获取适配的 ARM 版镜像（相同版本 KubeSphere 官方 ARM 版镜像）

# 找个相同版本的 ARM 架构的镜像

docker pull kubespheredev/argocd-applicationset-arm64:v0.4.1

镜像重新打 tag（为了保持镜像名称风格一致）

docker tag docker.io/kubespheredev/argocd-applicationset-arm64:v0.4.1 registry.cn-beijing.aliyuncs.com/kubesphereio/argocd-applicationset-arm64:v0.4.1

重新部署组件

# 修改 Deployment 使用的镜像，并重启
kubectl set image deployment/devops-argocd-applicationset-controller applicationset-controller=registry.cn-beijing.aliyuncs.com/kubesphereio/argocd-applicationset-arm64:v0.4.1 -n argocd
kubectl rollout restart deployment/devops-argocd-applicationset-controller -n argocd

验证新的 Pod 创建并启动成功

kubectl get pods -o wide -n argocd | grep applicationset-controller

3.3解决 Minio 异常

注意节点分配

获取适配的 ARM 版镜像（相近版本 Minio 官方 ARM 版镜像）

# 找个相近版本的 ARM 架构的镜像
# minio
docker pull minio/minio:RELEASE.2020-11-25T22-36-25Z-arm64

# mc
docker pull minio/mc:RELEASE.2020-11-25T23-04-07Z-arm64

镜像重新打 tag

# minio
docker rmi registry.cn-beijing.aliyuncs.com/kubesphereio/minio:RELEASE.2019-08-07T01-59-21Z
docker tag docker.io/minio/minio:RELEASE.2020-11-25T22-36-25Z-arm64 registry.cn-beijing.aliyuncs.com/kubesphereio/minio:RELEASE.2019-08-07T01-59-21Z

# mc
docker rmi registry.cn-beijing.aliyuncs.com/kubesphereio/mc:RELEASE.2019-08-07T23-14-43Z
docker tag docker.io/minio/mc:RELEASE.2020-11-25T23-04-07Z-arm64 registry.cn-beijing.aliyuncs.com/kubesphereio/mc:RELEASE.2019-08-07T23-14-43Z

重新部署组件

# 重新部署,删除旧的 Pod，系统会自动重建（此步的操作也可以使用修改 minio 对应的 deployment 使用的镜像名称的方式）
kubectl delete pod minio-757c8bc7f-tlnts -n kubesphere-system
kubectl delete pod minio-make-bucket-job-fzz95 -n kubesphere-system