Kubernetes 部署 Kafka 集群「Helm 部署案例」

原创

sir5kong

修改于 2023-08-29 17:14:24

5.6K0

修改于 2023-08-29 17:14:24

文章被收录于专栏：企业容器化之路

本项目完全兼容 KRaft, 不依赖 ZooKeeper，最小化的 Kafka 集群仅启动一个 Pod 即可。也可以单独用 Docker 启动，便于本地开发调试，详情请参考 Docker 启动 Kafka

Prerequisites

Kubernetes 1.18+
Helm 3.3+

添加 helm 仓库

helm repo add kafka-repo https://helm-charts.itboon.top/kafka
helm repo update kafka-repo

helm 部署

部署单节点集群

下面这个案例关闭了持久化存储，仅演示部署效果

helm upgrade --install kafka \
  --namespace kafka-demo \
  --create-namespace \
  --set broker.combinedMode.enabled="true" \
  --set broker.persistence.enabled="false" \
  kafka-repo/kafka

Controller 与 Broker 分离部署

helm upgrade --install kafka \
  --namespace kafka-demo \
  --create-namespace \
  --set broker.persistence.size="20Gi" \
  kafka-repo/kafka

默认已开启持久化存储。

部署高可用集群

helm upgrade --install kafka \
  --namespace kafka-demo \
  --create-namespace \
  --set controller.replicaCount="3" \
  --set broker.replicaCount="3" \
  --set broker.heapOpts="-Xms4096m -Xmx4096m" \
  --set broker.resources.requests.memory="8Gi" \
  --set broker.resources.limits.memory="16Gi" \
  kafka-repo/kafka

更多 values 请参考 examples/values-production.yml

LoadBalancer 外部暴露

开启 Kubernetes 集群外访问：

helm upgrade --install kafka \
  --namespace kafka-demo \
  --create-namespace \
  --set broker.external.enabled="true" \
  --set broker.external.service.type="LoadBalancer" \
  --set broker.external.domainSuffix="kafka.example.com" \
  kafka-repo/kafka

上面部署成功后请完成域名解析配置。

Chart Values

Key	Type	Default	Description
broker.combinedMode.enabled	bool	`false`	Whether to enable the combined mode

broker:
  combinedMode:
    enabled: true
  replicaCount: 1
  heapOpts: "-Xms1024m -Xmx1024m"
  persistence:
    enabled: true
    size: 20Gi

集群外访问

In order to connect to the Kafka server outside the cluster, each Broker must be exposed and advertised.listeners must be correctly configured.

There are two ways to expose, NodePort and LoadBalancer, each broker node needs a NodePort or LoadBalancer.

Chart Values

Key	Type	默认值	描述
broker.external.enabled	bool	`false`	是否开启集群外访问
broker.external.service.type	string	`NodePort`	`NodePort` or `LoadBalancer`
broker.external.service.annotations	object	`{}`	External serivce annotations
broker.external.nodePorts	list	`[]`	NodePort 模式，至少提供一个端口号，如果端口数量少于 broker 数量，则自增
broker.external.domainSuffix	string	`kafka.example.com`	If you use `LoadBalancer` for external access, you must use a domain name. The external domain name corresponding to the broker is `POD_NAME` + `domain name suffix`, such as `kafka-broker-0.kafka.example.com`. After the deployment, you need to complete the domain name resolution operation

## NodePort example
broker:
  replicaCount: 3
  external:
    enabled: true
    service:
      type: "NodePort"
      annotations: {}
    nodePorts:
      - 31050
      - 31051
      - 31052

## LoadBalancer example
broker:
  replicaCount: 3
  external:
    enabled: true
    service:
      type: "LoadBalancer"
      annotations: {}
    domainSuffix: "kafka.example.com"

Docker Image

Docker Compose 启动 Kafka

version: "3"

volumes:
  kafka-data: {}

services:
  kafka:
    image: kafkace/kafka:v3.5
    # restart: always
    ports:
      - "29092:29092"
    volumes:
      - kafka-data:/opt/kafka/data
    environment:
      - KAFKA_HEAP_OPTS=-Xmx512m -Xms512m
      - KAFKA_BROKER_EXTERNAL_HOST=kafka.example.com   ## 对外暴露的主机名，可以是域名或IP地址
      - KAFKA_BROKER_EXTERNAL_PORT=29092

  ## kafka web 管理 (可选)
  kafka-ui:
    image: provectuslabs/kafka-ui:v0.7.1
    # restart: always
    ports:
      - "18080:8080"
    environment:
      - KAFKA_CLUSTERS_0_NAME=demo-kafka-server
      - KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS=kafka:9092

### 内部网络
## broker 默认内部端口 9092
## bootstrap-server: kafka:9092

### 外部网络
## broker 默认外部端口 29092
## bootstrap-server: ${KAFKA_BROKER_EXTERNAL_HOST}:29092

Environment Variables

变量	默认值	描述
`KAFKA_CLUSTER_ID`	随机生成	Cluster ID
`KAFKA_BROKER_LISTENER_PORT`	`9092`	broker 端口号，如果配置了 `KAFKA_CFG_LISTENERS` 则此项失效
`KAFKA_CONTROLLER_LISTENER_PORT`	`19091`	controller 端口号，如果配置了 `KAFKA_CFG_LISTENERS` 则此项失效
`KAFKA_BROKER_EXTERNAL_HOST`	null	对外暴露的主机名，可以是域名或IP地址，如果配置了 `KAFKA_CFG_ADVERTISED_LISTENERS` 则此项失效
`KAFKA_BROKER_EXTERNAL_PORT`	`29092`	对外暴露的端口号，不能跟内部端口重复，如果配置了 `KAFKA_CFG_ADVERTISED_LISTENERS` 则此项失效
`KAFKA_HEAP_OPTS`	`null`	Kafka Java Heap size. 例如: `-Xmx512m -Xms512m`

https://github.com/itboon/kafka-docker

Kafka Configurations

所有以 KAFKA_CFG_ 开头的环境变量都将映射到其相应的 Apache Kafka 配置项。

例如 KAFKA_CFG_LISTENERS 对应配置参数 listeners，KAFKA_CFG_ADVERTISED_LISTENERS 对应配置参数 advertised.listeners

Variable examples:

变量	配置项
`KAFKA_CFG_PROCESS_ROLES`	`process.roles`
`KAFKA_CFG_LISTENERS`	`listeners`
`KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP`	`listener.security.protocol.map`
`KAFKA_CFG_ADVERTISED_LISTENERS`	`advertised.listeners`
`KAFKA_CFG_CONTROLLER_QUORUM_VOTERS`	`controller.quorum.voters`
`KAFKA_CFG_LOG_RETENTION_HOURS`	`log.retention.hours`