Kubelet从入门到放弃系列:GPU加持

Events:
  Type     Reason         Age                   From               Message
  ----     ------         ----                  ----               -------
  Normal   Scheduled      10m                   default-scheduler  Successfully assigned gpu-operator-resources/nvidia-device-plugin-daemonset-f99md to cl-gpu-md-0-f4gm6
  Warning  InspectFailed  10m (x3 over 10m)     kubelet            Failed to inspect image "nvcr.io/nvidia/k8s/cuda-sample:vectoradd-cuda10.2": rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = "transport: Error while dialing dial unix /run/containerd/containerd.sock: connect: connection refused"

# kubectl logs ‐f nvidia‐device‐plugin‐daemonset‐q9svq ‐nkube‐system
2021/02/11 01:32:29 Loading NVML
2021/02/11 01:32:29 Failed to initialize NVML: could not load NVML library.
2021/02/11 01:32:29 If this is a GPU node, did you set the docker default runtime to `nvidia`?
2021/02/11 01:32:29 You can check the prerequisites at: https://github.com/NVIDIA/k8s-device-plugin#prerequisites
2021/02/11 01:32:29 You can learn how to set the runtime at: https://github.com/NVIDIA/k8s-device-plugin#quick-start
2021/02/11 01:32:29 If this is not a GPU node, you should set up a toleration or nodeSelector to only deploy this plugin on GPU nodes

containerd[76114]: time="2020-12-04T08:52:13.029072066Z" level=error msg="StartContainer for "7a1453c6e7ab8af7395ccc8dac5efcffa94a0834aa7b252e1dcd5b51f92bf13e" failed" error="failed to create containerd task: OCI runtime create failed: container_linux.go:370: starting container process caused: process_linux.go:459: container init caused: Running hook #0:: error running hook: exit status 1, stdout: , stderr: nvidia-container-cli: mount error: open failed: /sys/fs/cgroup/devices/system.slice/containerd.service/kubepods-pod80540e95304d8cece2ae2afafd8b8976.slice/devices.allow: no such file or directory: unknown"

$ tar ‐zxvf gpu.tar.gz 
## 安装依赖 
$ cd gpu/runtime 
$ tar ‐zxvf dependency.tar.gz 
$ cd dependency 
## 查看是否支持CUDA的Nvidia的GPU 
$ cd ./lspci/ 
$ yum localinstall ‐y *.rpm 
$ lspci | grep ‐i nvidia ## 安装devel 
$ cd ../devel 
$ yum localinstall ‐y *.rpm 
## 安装gcc $ cd ../gcc 
$ yum localinstall ‐y *.rpm 
## 卸载nouveau驱动 
$ lsmod | grep nouveau 
$ rmmod nouveau 
## 安装驱动，过程见下面附的图片。如果要更新驱动，从https://developer.nvidia.com/cuda‐75‐downloads‐ archive下载 
$ cd ../../../driver 
$ sh cuda_10.1.105_418.39_linux.run ## 测试驱动，有如下输出则正常安装

$ nvidia‐smi

## 更新runc，下载地址https://github.com/opencontainers/runc/releases 
$ cd ../runtime 
$ cp runc /usr/bin/ 
## 更新containerd，下载地址 https://github.com/containerd/containerd/releases 
$ tar ‐zxvf containerd‐1.4.3‐linux‐amd64.tar.gz 
$ cp bin/* /usr/bin/ 
## 安装nvidia‐container‐runtime，yum源https://nvidia.github.io/nvidia‐docker/centos7/nvidia‐ docker.repo，yum安装：yum install ‐y nvidia‐container‐runtime $ tar ‐zxvf nvidia‐container‐runtime.tar.gz 
$ cd nvidia‐container‐runtime 
$ yum localinstall ‐y *.rpm

# 配置containerd的参数 
$ mkdir /etc/containerd/ 
$ vi /etc/containerd/config.toml 
# 配置containerd.service 
$ vi /usr/lib/systemd/system/containerd.service 
$ systemctl daemon‐reload 
$ systemctl restart containerd 
# 配置crictl 
$ tar ‐zxvf crictl‐v1.18.0‐linux‐amd64.tar.gz 
$ mv crictl /usr/bin/ 
$ vi /etc/profile alias crictl='crictl ‐‐runtime‐endpoint unix:///run/containerd/containerd.sock' 
$ source /etc/profile 
# 测试containerd和nvidia‐container‐runtime安装是否成功 
$ cd test‐image 
$ ctr images import cuda‐vector‐add_v0.1.tar 
$ ctr images push ‐‐plain‐http registry.paas/cmss/cuda‐vector‐add:v0.1

ctr run ‐t ‐‐gpus 0 registry.paas/cmss/cuda‐vector‐add:v0.1 cp nvidia‐smi

ctr c rm cp

[Unit] 
Description=containerd container runtime 
Documentation=https://containerd.io After=network.target

[Service] 
ExecStartPre=‐/sbin/modprobe overlay 
ExecStart=/usr/bin/containerd 
KillMode=process 
Delegate=yes 
LimitNOFILE=1048576 
# Having non‐zero Limit*s causes performance problems due to accounting overhead # in the kernel. We recommend using cgroups to do container‐local accounting. LimitNPROC=infinity 
LimitCORE=infinity 
TasksMax=infinity 
[Install] 
WantedBy=multi‐user.target    

$ vi /apps/conf/kubernetes/kubelet.env ‐‐container‐runtime=remote ‐‐container‐runtime‐endpoint=unix:///run/containerd/containerd.sock

$ cd device‐plugin 
$ docker load ‐i k8s‐device‐plugin_v0.7.3.tar 
$ docker push
// https://github.com/NVIDIA/k8s-device-plugin/tree/master/deployments/static
$ kubectl apply ‐f nvidia‐device‐plugin.yml 
$ kubectl logs ‐f nvidia‐device‐plugin‐daemonset‐q9svq ‐nkube‐system 
2021/02/08 06:32:36 Loading NVML 2021/02/08 06:32:42 Starting FS watcher. 2021/02/08 06:32:42 Starting OS watcher. 2021/02/08 06:32:42 Retreiving plugins. 2021/02/08 06:32:42 Starting GRPC server for 'nvidia.com/gpu' 2021/02/08 06:32:42 Starting to serve 'nvidia.com/gpu' on /var/lib/kubelet/device‐ plugins/nvidia‐gpu.sock 2021/02/08 06:32:42 Registered device plugin for 'nvidia.com/gpu' with Kubelet

$ cd test‐image 
# 启动测试pod 
$ kubectl apply ‐f demo.yml
// https://github.com/NVIDIA/gpu-operator/blob/master/tests/gpu-pod.yaml
$ kubectl logs ‐f cuda‐vector‐add 
[Vector addition of 50000 elements] 
Copy input data from the host memory to the CUDA device 
CUDA kernel launch with 196 blocks of 256 threads 
Copy output data from the CUDA device to the host memory 
Test PASSED 
Done