单机单实例部署Kafka及测试

方亮

发布于 2024-05-24 19:22:11

670

发布于 2024-05-24 19:22:11

文章被收录于专栏：方亮方亮

文章目录

部署
- Docker
测试
- 启动
- 测试
- - 创建Topic
  - 生产者
  - 消费者
集成Manager的部署
- dockerfile
- 效果
参考资料

在我们做和kafka开发相关的工作时，往往希望独立部署一套kafka测试环境。而kafka部署时，不能只是简单安装kafka自身组件，还要安装zookeeper、JDK之类的辅助软件。这让其部署变得不是那么方便。本文将使用kafka和zookeeper的官方编译包来做部署。

部署

Docker

# dockerfile
FROM ubuntu:22.04
WORKDIR /home/kafka-alone/
RUN apt-get update && apt-get upgrade -y
RUN apt-get install default-jdk -y
RUN apt-get install wget -y
RUN wget https://downloads.apache.org/kafka/3.7.0/kafka_2.13-3.7.0.tgz
RUN tar -xvzf kafka_2.13-3.7.0.tgz
RUN wget https://archive.apache.org/dist/zookeeper/zookeeper-3.9.2/apache-zookeeper-3.9.2-bin.tar.gz
RUN tar -xvzf apache-zookeeper-3.9.2-bin.tar.gz
RUN mv apache-zookeeper-3.9.2-bin zookeeper
RUN mv kafka_2.13-3.7.0 kafka
RUN rm -rf kafka_2.13-3.7.0.tgz
RUN rm -rf apache-zookeeper-3.9.2-bin.tar.gz
RUN mv zookeeper/conf/zoo_sample.cfg zookeeper/conf/zoo.cfg
EXPOSE 9092
RUN echo "./kafka/bin/zookeeper-server-start.sh ./kafka/config/zookeeper.properties &" >> start.sh
RUN echo "./kafka/bin/kafka-server-start.sh ./kafka/config/server.properties" >> start.sh
CMD ["/bin/bash", "-ce", "sh start.sh"]

然后使用下面指令编译成镜像

docker build --pull --rm -f "dockerfile" -t kafka-all-in-one:latest "."

测试

启动

docker container run -d -p 9092:9092 --name kafka-all-in-one-container kafka-all-in-one:latest

容器内的执行日志如下

测试

创建Topic

docker exec -it kafka-all-in-one-container /bin/bash kafka/bin/kafka-topics.sh --create --topic test-topic --bootstrap-server localhost:9092

生产者

docker exec -it kafka-all-in-one-container /bin/bash kafka/bin/kafka-console-producer.sh --topic test-topic --bootstrap-server localhost:9092

然后在终端中输入 hello world两个单词。

消费者

docker exec -it kafka-all-in-one-container /bin/bash kafka/bin/kafka-console-consumer.sh --topic test-topic --from-beginning --bootstrap-server localhost:9092

消费者就会收到生产者发布的两个消息。

集成Manager的部署

我们集成Kafka Manager到docker中，以可视化的方式管理Kafka。

dockerfile

FROM ubuntu:22.04
WORKDIR /home/kafka-alone/
RUN apt-get update && apt-get upgrade -y
RUN apt-get install default-jdk -y
RUN apt-get install wget -y
RUN wget https://downloads.apache.org/kafka/3.7.0/kafka_2.13-3.7.0.tgz
RUN tar -xvzf kafka_2.13-3.7.0.tgz
RUN wget https://archive.apache.org/dist/zookeeper/zookeeper-3.9.2/apache-zookeeper-3.9.2-bin.tar.gz
RUN tar -xvzf apache-zookeeper-3.9.2-bin.tar.gz
RUN mv apache-zookeeper-3.9.2-bin zookeeper
RUN mv kafka_2.13-3.7.0 kafka
RUN rm -rf kafka_2.13-3.7.0.tgz
RUN rm -rf apache-zookeeper-3.9.2-bin.tar.gz
RUN mv zookeeper/conf/zoo_sample.cfg zookeeper/conf/zoo.cfg
EXPOSE 9092
RUN echo "./kafka/bin/zookeeper-server-start.sh ./kafka/config/zookeeper.properties &" >> start.sh
RUN echo "./kafka/bin/kafka-server-start.sh ./kafka/config/server.properties &" >> start.sh

RUN apt-get install unzip -y
RUN wget https://github.com/yahoo/CMAK/releases/download/3.0.0.6/cmak-3.0.0.6.zip
RUN unzip cmak-3.0.0.6.zip
RUN mv cmak-3.0.0.6 cmak
RUN rm -rf cmak-3.0.0.6.zip
RUN sed -i 's/kafka-manager-zookeeper:2181/localhost:2181/g' cmak/conf/application.conf
RUN sed -i 's/cmak.zkhosts/#cmak.zkhosts/g' cmak/conf/application.conf
RUN echo "./cmak/bin/cmak -Dconfig.file=./cmak/conf/application.conf -Dhttp.port=8080" >> start.sh
EXPOSE 8080
CMD ["/bin/bash", "-ce", "sh start.sh"]

效果

使用下面命令部署后

docker build --pull --rm -f "dockerfile" -t kafka-all-in-one:latest "." 
docker container run -d -p 9092:9092 -p 8080:8080 --name kafka-all-in-one-container kafka-all-in-one:latest