cAdvisor+InfluxDB+Grafana
cAdvisor:是谷歌开源的一个容器监控工具,采集主机上容器相关的性能指标数据。比如CPU、内存、网络、文件系统等。
Heapster是谷歌开源的集群监控数据收集工具,会所有节点监控数据,Heapster作为一个pod在集群中运行,通过API获得集群中的所有节点,然后从节点kubelet暴露的10255汇总数据。目前社区用Metrics Server 取代 Heapster。
InfluxDB:时序数据库,存储监控数据。
Grafana:可视化展示。Grafana提供一个易于配置的仪表盘UI,可以轻松定制和扩展。
不足
l 功能单一,无法做到对K8S资源对象监控
l 告警单一,利用Grafana基本告警
l 依赖Influxdb,而Influxdb单点(高可用商业才支持),缺少开源精神
l heapster在1.8+弃用,metrics-server取代
[root@localhost monitor]# cat heapster.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: heapster
namespace: kube-system
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: heapster
roleRef:
kind: ClusterRole
name: cluster-admin
apiGroup: rbac.authorization.k8s.io
subjects:
- kind: ServiceAccount
name: heapster
namespace: kube-system
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: heapster
namespace: kube-system
spec:
replicas: 1
template:
metadata:
labels:
task: monitoring
k8s-app: heapster
spec:
serviceAccountName: heapster
containers:
- name: heapster
image: registry.cn-hangzhou.aliyuncs.com/google-containers/heapster-amd64:v1.5.1
imagePullPolicy: IfNotPresent
command:
- /heapster
- --source=kubernetes:https://kubernetes.default
- --sink=influxdb:http://monitoring-influxdb:8086
---
apiVersion: v1
kind: Service
metadata:
labels:
task: monitoring
kubernetes.io/cluster-service: 'true'
kubernetes.io/name: Heapster
name: heapster
namespace: kube-system
spec:
ports:
- port: 80
targetPort: 8082
selector:
k8s-app: heapster
使用top命令
[root@localhost monitor]# kubectl top nodes
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
192.168.0.220 208m 10% 1978Mi 53%
192.168.0.221 137m 6% 1068Mi 28%
192.168.0.222 135m 6% 1009Mi 27%
[root@localhost monitor]# kubectl top pods
NAME CPU(cores) MEMORY(bytes)
nginx-7db9fccd9b-qkb7t 0m 2Mi
nginx-7db9fccd9b-wtvk8 0m 3Mi
nginx-7db9fccd9b-xxkzb 0m 2Mi
登录Dashboard即可查看到对应的监控图表
部署grafana和influxdb
[root@localhost monitor]# cat grafana.yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: monitoring-grafana
namespace: kube-system
spec:
replicas: 1
template:
metadata:
labels:
task: monitoring
k8s-app: grafana
spec:
containers:
- name: grafana
image: registry.cn-hangzhou.aliyuncs.com/google-containers/heapster-grafana-amd64:v4.4.1
ports:
- containerPort: 3000
protocol: TCP
volumeMounts:
- mountPath: /var
name: grafana-storage
env:
- name: INFLUXDB_HOST
value: monitoring-influxdb
- name: GF_AUTH_BASIC_ENABLED
value: "false"
- name: GF_AUTH_ANONYMOUS_ENABLED
value: "true"
- name: GF_AUTH_ANONYMOUS_ORG_ROLE
value: Admin
- name: GF_SERVER_ROOT_URL
value: /
volumes:
- name: grafana-storage
emptyDir: {}
---
apiVersion: v1
kind: Service
metadata:
labels:
kubernetes.io/cluster-service: 'true'
kubernetes.io/name: monitoring-grafana
name: monitoring-grafana
namespace: kube-system
spec:
type: NodePort
ports:
- port : 80
targetPort: 3000
selector:
k8s-app: grafana
[root@localhost monitor]# cat influxdb.yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: monitoring-influxdb
namespace: kube-system
spec:
replicas: 1
template:
metadata:
labels:
task: monitoring
k8s-app: influxdb
spec:
containers:
- name: influxdb
image: registry.cn-hangzhou.aliyuncs.com/google-containers/heapster-influxdb-amd64:v1.1.1
volumeMounts:
- mountPath: /data
name: influxdb-storage
volumes:
- name: influxdb-storage
emptyDir: {}
---
apiVersion: v1
kind: Service
metadata:
labels:
task: monitoring
kubernetes.io/cluster-service: 'true'
kubernetes.io/name: monitoring-influxdb
name: monitoring-influxdb
namespace: kube-system
spec:
ports:
- port: 8086
targetPort: 8086
selector:
k8s-app: influxdb
[root@localhost monitor]# kubectl apply -f .
deployment.extensions/monitoring-grafana unchanged
service/monitoring-grafana unchanged
serviceaccount/heapster unchanged
clusterrolebinding.rbac.authorization.k8s.io/heapster unchanged
deployment.extensions/heapster unchanged
service/heapster unchanged
deployment.extensions/monitoring-influxdb unchanged
service/monitoring-influxdb unchanged
查看
[root@localhost monitor]# kubectl get pod,svc,deploy -n kube-system -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
pod/coredns-5b8c57999b-z9jh8 1/1 Running 0 22h 172.17.52.3 192.168.0.220 <none> <none>
pod/heapster-77f96f6f6c-qcr98 1/1 Running 0 14m 172.17.52.2 192.168.0.220 <none> <none>
pod/kubernetes-dashboard-644c96f9c6-bvw8w 1/1 Running 0 23h 172.17.100.2 192.168.0.222 <none> <none>
pod/monitoring-grafana-8646976767-cb86b 1/1 Running 0 8m55s 172.17.100.5 192.168.0.222 <none> <none>
pod/monitoring-influxdb-58b849b9f8-t26p8 1/1 Running 0 8m52s 172.17.50.4 192.168.0.221 <none> <none>
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
service/heapster ClusterIP 10.0.0.208 <none> 80/TCP 14m k8s-app=heapster
service/kube-dns ClusterIP 10.0.0.2 <none> 53/UDP,53/TCP 23h k8s-app=kube-dns
service/kubernetes-dashboard NodePort 10.0.0.84 <none> 443:30001/TCP 23h k8s-app=kubernetes-dashboard
service/monitoring-grafana NodePort 10.0.0.252 <none> 80:35811/TCP 14m k8s-app=grafana
service/monitoring-influxdb ClusterIP 10.0.0.85 <none> 8086/TCP 14m k8s-app=influxdb
访问grafana:http://192.168.0.222:35811
导入图表