功能概述
基础设施的不稳定性、环境的不确定性经常会引发不同维度的系统故障。为了将工作人员从繁重的运维事务中解放出来,腾讯云容器服务团队自研故障自愈功能来帮助运维人员快速定位问题,并通过预置平台运维经验,针对不同检测项提供最小化的自愈动作。该能力在 NPD Plus 组件的基础上进一步扩展,具体包含如下特性:
系统实时检测需要人为干预解决的持续性故障。
故障范围涵盖操作系统、K8s 环境、运行时等数十种检测项。
通过预置专家经验(执行修复脚本、重启组件)来对故障进行快速响应。
检测项介绍
检测项 | 描述 | 风险等级 | 自愈动作 |
FDPressure | Too many open files(查看主机的文件描述符数量是否达到最大值的 90%) | low | - |
RuntimeUnhealthy | List containerd task failed | low | RestartRuntime |
KubeletUnhealthy | Call kubelet healthz failed | low | RestartKubelet |
ReadonlyFilesystem | Filesystem is readonly | high | - |
OOMKilling | Process has been oom-killed | high | - |
TaskHung | Task blocked more then beyond the threshold | high | - |
UnregisterNetDevice | Net device unregister | high | - |
KernelOopsDivideError | Kernel oops with divide error | high | - |
KernelOopsNULLPointer | Kernel oops with NULL pointer | high | - |
Ext4Error | Ext4 filesystem error | high | - |
Ext4Warning | Ext4 filesystem warning | high | - |
IOError | IOError | high | - |
MemoryError | MemoryError | high | - |
DockerHung | Task blocked more then beyond the threshold | high | - |
KubeletRestart | Kubelet restart | low | - |
为节点开启故障自愈功能
通过控制台操作
1. 登录 容器服务控制台,选择左侧导航栏中的集群。
2. 在集群列表页中,单击集群 ID,进入该集群详情页。
3. 选择左侧菜单栏中的节点管理 > 节点配置 > 故障自愈,进入故障自愈规则页面。
4. 单击新建故障自愈规则,创建新的故障自愈规则。如下图所示:
![](https://qcloudimg.tencent-cloud.cn/image/document/a735f98806c3cdf867e99cda954acdb7.png)
![](https://qcloudimg.tencent-cloud.cn/image/document/a735f98806c3cdf867e99cda954acdb7.png)
5. 创建完成后,返回节点池列表页。
6. 单击节点池 ID,进入节点池详情页。
7. 在节点池详情页的“运维信息”模块,单击编辑为节点池开启故障自愈能力。如下图所示:
![](https://qcloudimg.tencent-cloud.cn/image/document/80bbe5a8bf3c048b3bc83f5a007250e8.png)
![](https://qcloudimg.tencent-cloud.cn/image/document/80bbe5a8bf3c048b3bc83f5a007250e8.png)
8. 开启后,可以在“运维记录”中查看实时故障检测详情,状态为“失败”则代表该检测项未通过。
![](https://qcloudimg.tencent-cloud.cn/image/document/2b7fcfccad0efddd6443cf3112a27c8e.png)
![](https://qcloudimg.tencent-cloud.cn/image/document/2b7fcfccad0efddd6443cf3112a27c8e.png)
通过 YAML 操作
1. 新建故障自愈规则
根据命令
kubectl create -f demo-HealthCheckPolicy.yaml
集群中创建自愈规则,YAML 配置如下:apiVersion: config.tke.cloud.tencent.com/v1kind: HealthCheckPolicymetadata:name: test-allnamespace: cls-xxxxxxxx(集群 id)spec:machineSetSelector:matchLabels:key: fake-labelrules:- action: RestartKubeletenabled: truename: FDPressure- action: RestartKubeletautoRepairEnabled: trueenabled: truename: RuntimeUnhealthy- action: RestartKubeletautoRepairEnabled: trueenabled: truename: KubeletUnhealthy- action: RestartKubeletenabled: truename: ReadonlyFilesystem- action: RestartKubeletenabled: truename: OOMKilling- action: RestartKubeletenabled: truename: TaskHung- action: RestartKubeletenabled: truename: UnregisterNetDevice- action: RestartKubeletenabled: truename: KernelOopsDivideError- action: RestartKubeletenabled: truename: KernelOopsNULLPointer- action: RestartKubeletenabled: truename: Ext4Error- action: RestartKubeletenabled: truename: Ext4Warning- action: RestartKubeletenabled: truename: IOError- action: RestartKubeletenabled: truename: MemoryError- action: RestartKubeletenabled: truename: DockerHung- action: RestartKubeletenabled: truename: KubeletRestart
2. 开启自愈开关
在 MachineSet 中指定字段
healthCheckPolicyName: test-all
,YAML 配置如下:apiVersion: node.tke.cloud.tencent.com/v1beta1kind: MachineSetspec:type: HosteddisplayName: demo-machinesetreplicas: 2autoRepair: truedeletePolicy: RandomhealthCheckPolicyName: test-allinstanceTypes:- C3.LARGE8subnetIDs:- subnet-xxxxxxxx- subnet-yyyyyyyy......