Hyperledger fabric 1.4 kafka排序服务原理与安装

1.Kafka排序服务原理

官方文档在google doc上

参考翻译 https://www.jianshu.com/p/db006359133d

2. kafka 排序服务安装

所有的代码已分享在

https://github.com/zealzeng/kafka-orderer-demo

2.1 安装环境

官方文档有一些简单的描述

https://hyperledger-fabric.readthedocs.io/en/release-1.4/kafka.html

kafka节点需要3f+1, 最少需要4个节点;

zookeeper节点2f+1, 最少需要3个节点;

资源有限, orderer和peer最好分开,kafka和zookeeper一台问题不大。

Hyperledger fabric 1.4年前已正式发布, 是一个长期维护的版本, 适合用于生产环境。orderer开发可以设为solo, 生产就最好用kafka了。

2.2 生成密钥,证书,docker compose文件

(1) 配置crypto-config.yaml

以first-network的模板做调整, 一些关键配置

OrdererOrgs:

# ---------------------------------------------------------------------------

# Orderer

# ---------------------------------------------------------------------------

- Name: Orderer

Domain: example.com

# ---------------------------------------------------------------------------

# "Specs" - See PeerOrgs below for complete description

# ---------------------------------------------------------------------------

Specs:

- Hostname: orderer1

- Hostname: orderer2

- Hostname: orderer3

orderer组织配置三个节点. peer的组织默认还是Org1和Org2.

(2) 配置configtx.yaml

修改ordererType为kafka, 增加三个orderer节点地址, 四个kafka broker节点, 这里统一使用域名或机器名, 需要修改docker container的/etc/hosts做ip映射

Orderer: &OrdererDefaults

# Orderer Type: The orderer implementation to start

# Available types are "solo" and "kafka"

OrdererType: kafka

Addresses:

- orderer1.example.com:7050

- orderer2.example.com:7050

- orderer3.example.com:7050

# Batch Timeout: The amount of time to wait before creating a batch

BatchTimeout: 2s

# Batch Size: Controls the number of messages batched into a block

BatchSize:

# Max Message Count: The maximum number of messages to permit in a batch

MaxMessageCount: 10

# Absolute Max Bytes: The absolute maximum number of bytes allowed for

# the serialized messages in a batch.

AbsoluteMaxBytes: 99 MB

# Preferred Max Bytes: The preferred maximum number of bytes allowed for

# the serialized messages in a batch. A message larger than the preferred

# max bytes will result in a batch larger than preferred max bytes.

PreferredMaxBytes: 512 KB

Kafka:

# Brokers: A list of Kafka brokers to which the orderer connects

# NOTE: Use IP:port notation

Brokers:

- kafka1:9092

- kafka2:9092

- kafka3:9092

- kafka4:9092

(3) 生成密钥,证书,创始块, 通道, 锚节点更新文件等

cryptogen generate --config=./crypto-config.yaml

mkdir channel-artifacts

configtxgen -profile TwoOrgsOrdererGenesis -outputBlock ./channel-artifacts/genesis.block

configtxgen -profile TwoOrgsChannel -outputCreateChannelTx ./channel-artifacts/channel.tx -channelID mychannel

configtxgen -profile TwoOrgsChannel -outputAnchorPeersUpdate ./channel-artifacts/Org1MSPanchors.tx -channelID mychannel -asOrg Org1MSP

configtxgen -profile TwoOrgsChannel -outputAnchorPeersUpdate ./channel-artifacts/Org2MSPanchors.tx -channelID mychannel -asOrg Org2MSP

从86的机器拷贝到其它机器

scp -r channel-artifacts crypto-config root@192.168.31.168:/mnt/sda3/fabric-samples/kafka-orderer

scp -r channel-artifacts crypto-config root@192.168.31.121:/mnt/sda3/fabric-samples/kafka-orderer

scp -r channel-artifacts crypto-config root@192.168.31.231:/mnt/sda3/fabric-samples/kafka-orderer

(4) 编辑zookeeper节点对应文件

以zk1节点为例, 编辑docker-compose-zk1.yaml, 名字ID等都配置成唯一, 2181端口用于客户端连接, 2888和3888应该是内部leader选举和通信使用, 具体参考zookeeper文档。

extra_hosts在容器环境配置上域名和ip映射, 会写到容器的/etc/hosts

version: '2'

services:

zk1:

container_name: zk1

hostname: zk1

image: hyperledger/fabric-zookeeper:0.4.14

restart: always

environment:

- ZOO_MY_ID=1

- ZOO_SERVERS=server.1=zk1:2888:3888 server.2=zk2:2888:3888 server.3=zk3:2888:3888

ports:

- 2181:2181

- 2888:2888

- 3888:3888

extra_hosts:

- "zk1:192.168.31.86"

- "zk2:192.168.31.168"

- "zk3:192.168.31.121"

- "kafka1:192.168.31.86"

- "kafka2:192.168.31.168"

- "kafka3:192.168.31.121"

- "kafka4:192.168.31.231"

编辑step1-start-zk1.sh脚本用于方便启动。

docker-compose -f docker-compose-zk1.yaml up -d

其它zookeeper节点类似配置。

(5) 编辑kafka节点对应文件

以kafka1节点为例, 编辑docker-compose-kafka1.yaml

services:

kafka1:

container_name: kafka1

hostname: kafka1

image: hyperledger/fabric-kafka:0.4.14

restart: always

environment:

- KAFKA_MESSAGE_MAX_BYTES=103809024 # 99 * 1024 * 1024 B

- KAFKA_REPLICA_FETCH_MAX_BYTES=103809024 # 99 * 1024 * 1024 B

- KAFKA_UNCLEAN_LEADER_ELECTION_ENABLE=false

environment:

- KAFKA_BROKER_ID=1

- KAFKA_MIN_INSYNC_REPLICAS=2

- KAFKA_DEFAULT_REPLICATION_FACTOR=3

- KAFKA_ZOOKEEPER_CONNECT=zk1:2181,zk2:2181,zk3:2181

ports:

- 9092:9092

extra_hosts:

- "zk1:192.168.31.86"

- "zk2:192.168.31.168"

- "zk3:192.168.31.121"

- "kafka1:192.168.31.86"

- "kafka2:192.168.31.168"

- "kafka3:192.168.31.121"

- "kafka4:192.168.31.231"

编辑启动脚本step2-start-kafka1.sh

docker-compose -f docker-compose-kafka1.yaml up -d

其它节点配置类似.

(6) 配置orderer节点对应文件

以orderer1.example.com节点为例, 编辑docker-compose-orderer1.yaml

这里需要指定networks为byfn, peer节点也需要归到该网络, 因为这里前三组的orderer和peer是运行在对应一台机器。 如果不这样做, peer节点实例化链码的时候会报一些lscc timeout的错误, 可能是同一台的机器不同container的peer和orderer内部通信有问题， 或者是chaincode新起container实例通信有问题。

OSN orderer节点要和kafka节点通信, 设置好域名ip映射。

version: '2'

networks:

byfn:

volumes:

orderer.example.com:

services:

orderer1.example.com:

container_name: orderer1.example.com

image: hyperledger/fabric-orderer:1.4.0

environment:

- FABRIC_LOGGING_SPEC=INFO

- ORDERER_GENERAL_LISTENADDRESS=0.0.0.0

- ORDERER_GENERAL_GENESISMETHOD=file

- ORDERER_GENERAL_GENESISFILE=/var/hyperledger/orderer/orderer.genesis.block

- ORDERER_GENERAL_LOCALMSPID=OrdererMSP

- ORDERER_GENERAL_LOCALMSPDIR=/var/hyperledger/orderer/msp

# enabled TLS

- ORDERER_GENERAL_TLS_ENABLED=true

- ORDERER_GENERAL_TLS_PRIVATEKEY=/var/hyperledger/orderer/tls/server.key

- ORDERER_GENERAL_TLS_CERTIFICATE=/var/hyperledger/orderer/tls/server.crt

- ORDERER_GENERAL_TLS_ROOTCAS=[/var/hyperledger/orderer/tls/ca.crt]

- ORDERER_KAFKA_TOPIC_REPLICATIONFACTOR=1

- ORDERER_KAFKA_RETRY_LONGINTERVAL=10s

- ORDERER_KAFKA_RETRY_LONGTOTAL=100s

- ORDERER_KAFKA_RETRY_SHORTINTERVAL=1s

- ORDERER_KAFKA_RETRY_SHORTTOTAL=30s

- ORDERER_KAFKA_VERBOSE=true

working_dir: /opt/gopath/src/github.com/hyperledger/fabric

command: orderer

volumes:

- ./channel-artifacts/genesis.block:/var/hyperledger/orderer/orderer.genesis.block

- ./crypto-config/ordererOrganizations/example.com/orderers/orderer1.example.com/msp:/var/hyperledger/orderer/msp

- ./crypto-config/ordererOrganizations/example.com/orderers/orderer1.example.com/tls/:/var/hyperledger/orderer/tls

- orderer.example.com:/var/hyperledger/production/orderer

ports:

- 7050:7050

extra_hosts:

- "kafka1:192.168.31.86"

- "kafka2:192.168.31.168"

- "kafka3:192.168.31.121"

- "kafka4:192.168.31.231"

networks:

- byfn

对应启动脚本step3-start-orderer1.sh, 其它节点类似.

(7) 配置peer节点对应文件

peer节点最为关键, 编辑docker-compose-peer0-org1.yaml

Peer0不配置extra_hosts的时候, 按照first-network的例子照搬, peer节点是可以跑起来, 但是peer节点之间的通信有一些报警或错误, 按理说extra_hosts是要配置的, 不然peer0无法识别peer1和orderer节点。

但加入extra_hosts之后, peer节点启动一阵就挂了, 查看日志chaincodeListenerAddress无法绑定到192.168.31.86:7052(可能是extra_hosts映射到了ip), 有自动检测容器ip的一些日志。

配置CORE_PEER_CHAINCODELISTENADDRESS=peer0.org1.example.com:7052是不行的, 依旧域名命中转为31.86:7052端口绑定失败。 容器的ip自然不是86, 类似orderer监听的配置方法设置为0.0.0.0:7052就好了。 不过奇怪的是实际查看docker ps, 容器的7052是没暴露到宿主机器的。 老实说配置是有点多了, 官方给的例子7052都没配置过, 这些只能去跟源码才可能了解到了。

一个比较重要的配置项是CORE_VM_DOCKER_HOSTCONFIG_NETWORKMODE, 这里注意到的${COMPOSE_PROJECT_NAME}是在.env文件中配置了值为net, 漏了会出问题, 而_后跟着的byfn是对应的networks配置的值。这个配置项从注释上来看好像是让chaincode实例化时新起的容器与所在的peer节点在同一个桥接网络中。笔者的环境不这样配就无法实例化链码，也需要设置peer, orderer,cli在同一个byfn网络内。

也有可能peer节点和orderer节点单独在不同机器就不需要这样配置， 请大家尝试。

networks:

byfn:

volumes:

peer0.org1.example.com:

peer1.org1.example.com:

peer0.org2.example.com:

peer1.org2.example.com:

services:

peer0.org1.example.com:

container_name: peer0.org1.example.com

#hostname: peer0.org1.example.com

image: hyperledger/fabric-peer:1.4.0

environment:

- CORE_PEER_ID=peer0.org1.example.com

- CORE_PEER_ADDRESS=peer0.org1.example.com:7051

- CORE_PEER_GOSSIP_BOOTSTRAP=peer1.org1.example.com:7051

- CORE_PEER_GOSSIP_EXTERNALENDPOINT=peer0.org1.example.com:7051

#- CORE_PEER_CHAINCODELISTENADDRESS=peer0.org1.example.com:7052

- CORE_PEER_CHAINCODELISTENADDRESS=0.0.0.0:7052

- CORE_PEER_LOCALMSPID=Org1MSP

- CORE_VM_ENDPOINT=unix:///host/var/run/docker.sock

- CORE_VM_DOCKER_HOSTCONFIG_NETWORKMODE=${COMPOSE_PROJECT_NAME}_byfn

# the following setting starts chaincode containers on the same

# bridge network as the peers

# https://docs.docker.com/compose/networking/

- CORE_LOGGING_LEVEL=INFO

- CORE_PEER_GOSSIP_USELEADERELECTION=true

- CORE_PEER_GOSSIP_ORGLEADER=false

- CORE_PEER_PROFILE_ENABLED=true

- CORE_PEER_TLS_ENABLED=true

- CORE_PEER_TLS_CERT_FILE=/etc/hyperledger/fabric/tls/server.crt

- CORE_PEER_TLS_KEY_FILE=/etc/hyperledger/fabric/tls/server.key

- CORE_PEER_TLS_ROOTCERT_FILE=/etc/hyperledger/fabric/tls/ca.crt

working_dir: /opt/gopath/src/github.com/hyperledger/fabric/peer

command: peer node start

volumes:

- /var/run/:/host/var/run/

- ./crypto-config/peerOrganizations/org1.example.com/peers/peer0.org1.example.com/msp:/etc/hyperledger/fabric/msp

- ./crypto-config/peerOrganizations/org1.example.com/peers/peer0.org1.example.com/tls:/etc/hyperledger/fabric/tls

- peer0.org1.example.com:/var/hyperledger/production

ports:

- 7051:7051

- 7052:7052

- 7053:7053

extra_hosts:

- "orderer1.example.com:192.168.31.86"

- "orderer2.example.com:192.168.31.168"

- "orderer3.example.com:192.168.31.121"

- "peer0.org1.example.com:192.168.31.86"

- "peer1.org1.example.com:192.168.31.168"

- "peer0.org2.example.com:192.168.31.121"

- "peer1.org2.example.com:192.168.31.231"

networks:

- byfn

cli:

container_name: cli

image: hyperledger/fabric-tools:1.4.0

tty: true

stdin_open: true

environment:

- GOPATH=/opt/gopath

- CORE_VM_ENDPOINT=unix:///host/var/run/docker.sock

- CORE_LOGGING_LEVEL=INFO

- CORE_PEER_ID=cli

- CORE_PEER_ADDRESS=peer0.org1.example.com:7051

- CORE_PEER_LOCALMSPID=Org1MSP

- CORE_PEER_TLS_ENABLED=true

- CORE_PEER_TLS_CERT_FILE=/opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/peerOrganizations/org1.example.com/peers/peer0.org1.example.com/tls/server.crt

- CORE_PEER_TLS_KEY_FILE=/opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/peerOrganizations/org1.example.com/peers/peer0.org1.example.com/tls/server.key

- CORE_PEER_TLS_ROOTCERT_FILE=/opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/peerOrganizations/org1.example.com/peers/peer0.org1.example.com/tls/ca.crt

- CORE_PEER_MSPCONFIGPATH=/opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/peerOrganizations/org1.example.com/users/Admin@org1.example.com/msp

working_dir: /opt/gopath/src/github.com/hyperledger/fabric/peer

volumes:

- /var/run/:/host/var/run/

- ./chaincode/:/opt/gopath/src/github.com/chaincode

- ./crypto-config:/opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/

- ./scripts:/opt/gopath/src/github.com/hyperledger/fabric/peer/scripts/

- ./channel-artifacts:/opt/gopath/src/github.com/hyperledger/fabric/peer/channel-artifacts

extra_hosts:

- "orderer1.example.com:192.168.31.86"

- "orderer2.example.com:192.168.31.168"

- "orderer3.example.com:192.168.31.121"

- "peer0.org1.example.com:192.168.31.86"

- "peer1.org1.example.com:192.168.31.168"

- "peer0.org2.example.com:192.168.31.121"

- "peer1.org2.example.com:192.168.31.231"

networks:

- byfn

对应启动脚本step4-start-peer0-org1.sh, 其它节点类似。

(8) 拷贝yaml到各节点, 其次启动节点.

scp -r *.yaml root@192.168.31.168:/mnt/sda3/fabric-samples/kafka-orderer

scp -r *.yaml root@192.168.31.121:/mnt/sda3/fabric-samples/kafka-orderer

scp -r *.yaml root@192.168.31.231:/mnt/sda3/fabric-samples/kafka-orderer

先启动三个zookeeper

192.168.31.86 ./step1-start-zk1.sh

192.168.31.168 ./step1-start-zk2.sh

192.168.31.121 ./step1-start-zk3.sh

启动四个kafka

192.168.31.86 ./step2-start-kafka1.sh

192.168.31.168 ./step2-start-kafka2.sh

192.168.31.121 ./step2-start-kafka3.sh

192.168.31.231 ./step2-start-kafka3.sh

启动三个orderer

192.168.31.86 ./step3-start-orderer1.sh

192.168.31.168 ./step3-start-orderer2.sh

192.168.31.121 ./step3-start-orderer3.sh

启动四个peer

192.168.31.86 ./step4-start-peer0-org1.sh

192.168.31.168 ./step4-start-peer1-org1.sh

192.168.31.121 ./step4-start-peer0-org2.sh

192.168.31.231 ./step4-start-peer1-org2.sh

每一步骤多看下docker进程日志.

(9) 创建通道, 加入通道, 安装和调用链码

很多时候在这步才能判断出安装的问题, 没搞好很可能全部重来。

创建通道, 加入通道

192.168.31.86

#登录到容器cli

docker exec -it -e LINES=$(tput lines) -e COLUMNS=$(tput cols) cli bash

#创建通道,可能会提示SERVICE UNAVIALBLE, 但不影响

peer channel create -o orderer1.example.com:7050 -c mychannel -f ./channel-artifacts/channel.tx --tls --cafile /opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/ordererOrganizations/example.com/orderers/orderer1.example.com/msp/tlscacerts/tlsca.example.com-cert.pem

#加入通道

peer channel join -b mychannel.block

#拷贝mychannel.block到其它peer

mv mychannel.block ./channel-artifacts

#退出cli

exit

scp -r channel-artifacts crypto-config root@192.168.31.168:/mnt/sda3/fabric-samples/kafka-orderer

scp -r channel-artifacts crypto-config root@192.168.31.121:/mnt/sda3/fabric-samples/kafka-orderer

scp -r channel-artifacts crypto-config root@192.168.31.231:/mnt/sda3/fabric-samples/kafka-orderer

=======================================

192.168.31.168, 192.168.121, 192.168.31.231

#进入cli, 加入通道

docker exec -it -e LINES=$(tput lines) -e COLUMNS=$(tput cols) cli bash

cd ./channel-artifacts

peer channel join -b mychannel.block

更新锚节点

192.168.31.86

#加入cli bash执行,一个组织一个节点执行即可

peer channel update -o orderer1.example.com:7050 -c mychannel -f ./channel-artifacts/Org1MSPanchors.tx --tls --cafile /opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/ordererOrganizations/example.com/orderers/orderer1.example.com/msp/tlscacerts/tlsca.example.com-cert.pem

==========================================

192.168.121

#加入cli bash执行,一个组织一个节点执行即可

peer channel update -o orderer2.example.com:7050 -c mychannel -f ./channel-artifacts/Org2MSPanchors.tx --tls --cafile /opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/ordererOrganizations/example.com/orderers/orderer2.example.com/msp/tlscacerts/tlsca.example.com-cert.pem

安装链码,实例化, 调用链码

192.168.31.86

#cli bash, install

peer chaincode install -n mycc -v 1.0 -p github.com/chaincode/chaincode_example02/go/

#实例化, 执行成功后到其它节点查询peer chaincode list --instantiated -C mychannel应该会有记录,但是无安装记录

peer chaincode instantiate -o orderer1.example.com:7050 --tls --cafile /opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/ordererOrganizations/example.com/orderers/orderer1.example.com/msp/tlscacerts/tlsca.example.com-cert.pem -C mychannel -n mycc -v 1.0 -c '{"Args":["init","a", "100", "b","200"]}' -P "AND ('Org1MSP.peer','Org2MSP.peer')"

#链码查询,应该返回100

peer chaincode query -C mychannel -n mycc -c '{"Args":["query","a"]}'

#转账调用，a应该剩下90

peer chaincode invoke -o orderer1.example.com:7050 --tls true --cafile /opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/ordererOrganizations/example.com/orderers/orderer1.example.com/msp/tlscacerts/tlsca.example.com-cert.pem -C mychannel -n mycc --peerAddresses peer0.org1.example.com:7051 --tlsRootCertFiles /opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/peerOrganizations/org1.example.com/peers/peer0.org1.example.com/tls/ca.crt --peerAddresses peer0.org2.example.com:7051 --tlsRootCertFiles /opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/peerOrganizations/org2.example.com/peers/peer0.org2.example.com/tls/ca.crt -c '{"Args":["invoke","a","b","10"]}'

==============================

192.168.31.168, 192.168.121, 192.168.31.231

#需要先安装链码代码

peer chaincode install -n mycc -v 1.0 -p github.com/chaincode/chaincode_example02/go/

#执行查询, 返回90

peer chaincode query -C mychannel -n mycc -c '{"Args":["query","a"]}'

3. 小结

提供的代码只用于学习, 不适合用于生产。持久化配置, 记账本使用couchdb, kafka, zookeeper网络安全设置, 系统参数， 容器参数等都需要按照实际情况调整。

希望对大家有帮助