一、原生Docker命令
1. 删除所有dangling数据卷（即无用的Volume，僵尸文件）
2. 删除所有dangling镜像（即无tag的镜像）
?
1
docker rmi $(docker images | grep “^<none>” | awk “{print $3}”
3. 删除所有关闭的容器
?
1
docker ps -a | grep Exit | cut -d -f 1 | xargs docker rm
二、镜像选择
环境为M1版本的mbp:
Zookeeper采用zookeeper Kafka采用wurstmeister/kafka Kafka-Manager采用scjtqs/kafka-manager Mysql采用mysql/mysql-server三、集群规划
1. 新建docker网络
?
1
docker network create docker-net –subnet 172.20.10.0/16
?
1
docker network ls
2. 集群规划
hostname Ip addr port listener zook1 172.20.10.11 2184:2181 zook2 172.20.10.12 2185:2181 zook3 172.20.10.13 2186:2181 kafka1 172.20.10.14 内部9092:9092，外部9192:9192 kafka1 kafka2 172.20.10.15 内部9093:9093，外部9193:9193 kafka2 Kafka3 172.20.10.16 内部9094:9094，外部9194:9194 Kafka3 本机（宿主机Mbp） 172.20.10.2 kafka manager 172.20.10.10 9000:9000四、Zookeeper集群安装
1. 新建文件zk-docker-compose.yml
?
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
version: 3.4
services:
zook1:
image: zookeeper:latest
restart: always
hostname: zook1
container_name: zook1 #容器名称，方便在rancher中显示有意义的名称
ports:
– 2183:2181 #将本容器的zookeeper默认端口号映射出去
volumes: # 挂载数据卷
– “/Users/konsy/Development/volume/zkcluster/zook1/data:/data”
– “/Users/konsy/Development/volume/zkcluster/zook1/datalog:/datalog”
– “/Users/konsy/Development/volume/zkcluster/zook1/logs:/logs”
environment:
ZOO_MY_ID: 1 #即是zookeeper的节点值，也是kafka的brokerid值
ZOO_SERVERS: server.1=zook1:2888:3888;2181 server.2=zook2:2888:3888;2181 server.3=zook3:2888:3888;2181
networks:
docker-net:
ipv4_address: 172.20.10.11
zook2:
image: zookeeper:latest
restart: always
hostname: zook2
container_name: zook2 #容器名称，方便在rancher中显示有意义的名称
ports:
– 2184:2181 #将本容器的zookeeper默认端口号映射出去
volumes:
– “/Users/konsy/Development/volume/zkcluster/zook2/data:/data”
– “/Users/konsy/Development/volume/zkcluster/zook2/datalog:/datalog”
– “/Users/konsy/Development/volume/zkcluster/zook2/logs:/logs”
environment:
ZOO_MY_ID: 2 #即是zookeeper的节点值，也是kafka的brokerid值
ZOO_SERVERS: server.1=zook1:2888:3888;2181 server.2=zook2:2888:3888;2181 server.3=zook3:2888:3888;2181
networks:
docker-net:
ipv4_address: 172.20.10.12
zook3:
image: zookeeper:latest
restart: always
hostname: zook3
container_name: zook3 #容器名称，方便在rancher中显示有意义的名称
ports:
– 2185:2181 #将本容器的zookeeper默认端口号映射出去
volumes:
– “/Users/konsy/Development/volume/zkcluster/zook3/data:/data”
– “/Users/konsy/Development/volume/zkcluster/zook3/datalog:/datalog”
– “/Users/konsy/Development/volume/zkcluster/zook3/logs:/logs”
environment:
ZOO_MY_ID: 3 #即是zookeeper的节点值，也是kafka的brokerid值
ZOO_SERVERS: server.1=zook1:2888:3888;2181 server.2=zook2:2888:3888;2181 server.3=zook3:2888:3888;2181
networks:
docker-net:
ipv4_address: 172.20.10.13
networks:
docker-net:
external:
name: docker-net
2. 执行脚本部署zookeeper至Docker:
?
1
docker compose -f ./zk-docker-compose.yml up -d
五、Kafka集群安装
1. 新建文件kafka-docker-compose.yml
?
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
version: 2
services:
kafka1:
image: docker.io/wurstmeister/kafka
restart: always
hostname: kafka1
container_name: kafka1
ports:
– 9093:9093
– 9193:9193
environment:
KAFKA_BROKER_ID: 1
KAFKA_LISTENERS: INSIDE://:9093,OUTSIDE://:9193
#KAFKA_ADVERTISED_LISTENERS=INSIDE://<container>:9092,OUTSIDE://<host>:9094
SKAFKA_ADVERTISED_LISTENERS: INSIDE://kafka1:9093,OUTSIDE://localhost:9193
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT
KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE
KAFKA_ZOOKEEPER_CONNECT: zook1:2181,zook2:2181,zook3:2181
ALLOW_PLAINTEXT_LISTENER : yes
JMX_PORT: 9999 #开放JMX监控端口，来监测集群数据
volumes:
– /Users/konsy/Development/volume/kafka/kafka1/wurstmeister/kafka:/wurstmeister/kafka
– /Users/konsy/Development/volume/kafka/kafka1/kafka:/kafka
external_links:
– zook1
– zook2
– zook3
networks:
docker-net:
ipv4_address: 172.20.10.14
kafka2:
image: docker.io/wurstmeister/kafka
restart: always
hostname: kafka2
container_name: kafka2
ports:
– 9094:9094
– 9194:9194
environment:
KAFKA_BROKER_ID: 2
KAFKA_LISTENERS: INSIDE://:9094,OUTSIDE://:9194
#KAFKA_ADVERTISED_LISTENERS=INSIDE://<container>:9092,OUTSIDE://<host>:9094
KAFKA_ADVERTISED_LISTENERS: INSIDE://kafka2:9094,OUTSIDE://localhost:9194
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT
KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE
KAFKA_ZOOKEEPER_CONNECT: zook1:2181,zook2:2181,zook3:2181
ALLOW_PLAINTEXT_LISTENER : yes
JMX_PORT: 9999 #开放JMX监控端口，来监测集群数据
volumes:
– /Users/konsy/Development/volume/kafka/kafka2/wurstmeister/kafka:/wurstmeister/kafka
– /Users/konsy/Development/volume/kafka/kafka2/kafka:/kafka
external_links:
– zook1
– zook2
– zook3
networks:
docker-net:
ipv4_address: 172.20.10.15
kafka3:
image: docker.io/wurstmeister/kafka
restart: always
hostname: kafka3
container_name: kafka3
ports:
– 9095:9095
– 9195:9195
environment:
KAFKA_BROKER_ID: 3
KAFKA_LISTENERS: INSIDE://:9095,OUTSIDE://:9195
#KAFKA_ADVERTISED_LISTENERS=INSIDE://<container>:9092,OUTSIDE://<host>:9094
KAFKA_ADVERTISED_LISTENERS: INSIDE://kafka3:9095,OUTSIDE://localhost:9195
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT
KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE
KAFKA_ZOOKEEPER_CONNECT: zook1:2181,zook2:2181,zook3:2181
ALLOW_PLAINTEXT_LISTENER : yes
JMX_PORT: 9999 #开放JMX监控端口，来监测集群数据
volumes:
– /Users/konsy/Development/volume/kafka/kafka3/wurstmeister/kafka:/wurstmeister/kafka
– /Users/konsy/Development/volume/kafka/kafka3/kafka:/kafka
external_links:
– zook1
– zook2
– zook3
networks:
docker-net:
ipv4_address: 172.20.10.16
networks:
docker-net:
external:
name: docker-net
2. 执行脚本部署kafka至Docker:
?
1
docker compose -f ./kafka-docker-compose.yml up -d
3. listeners 和 advertised.listeners
listeners: 学名叫监听器，其实就是告诉外部连接者要通过什么协议访问指定主机名和端口开放的 Kafka 服务。 advertised.listeners：和 listeners 相比多了个 advertised。Advertised 的含义表示宣称的、公布的，就是说这组监听器是 Broker 用于对外发布的。比如说：
?
1
2
3
4
listeners: INSIDE://172.17.0.10:9092,OUTSIDE://172.17.0.10:9094
advertised_listeners: INSIDE://172.17.0.10:9092,OUTSIDE://<公网 ip>:端口
kafka_listener_security_protocol_map: “INSIDE:SASL_PLAINTEXT,OUTSIDE:SASL_PLAINTEXT”
kafka_inter_broker_listener_name: “INSIDE”
advertised_listeners 监听器会注册在 zookeeper 中；
当我们对 172.17.0.10:9092 请求建立连接，kafka 服务器会通过 zookeeper 中注册的监听器，找到 INSIDE 监听器，然后通过 listeners 中找到对应的 通讯 ip 和 端口；
同理，当我们对 <公网 ip>:端口 请求建立连接，kafka 服务器会通过 zookeeper 中注册的监听器，找到 OUTSIDE 监听器，然后通过 listeners 中找到对应的 通讯 ip 和 端口 172.17.0.10:9094；
总结：advertised_listeners 是对外暴露的服务端口，真正建立连接用的是 listeners。
4.安装kafka-manager
4.1 新建文件kafka-manager-docker-compose.yml
?
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
version: 2
services:
kafka-manager:
image: scjtqs/kafka-manager:latest
restart: always
hostname: kafka-manager
container_name: kafka-manager
ports:
– 9000:9000
external_links: # 连接本compose文件以外的container
– zook1
– zook2
– zook3
– kafka1
– kafka2
– kafka3
environment:
ZK_HOSTS: zook1:2181,zook2:2181,zook3:2181
KAFKA_BROKERS: kafka1:9093,kafka2:9094,kafka3:9095
APPLICATION_SECRET: letmein
KM_ARGS: -Djava.net.preferIPv4Stack=true
networks:
docker-net:
ipv4_address: 172.20.10.10
networks:
docker-net:
external:
name: docker-net
4.2 执行脚本部署kafka-manager至Docker:
?
1
docker compose -f ./kafka-manager-docker-compose.yml up -d
4.3 配置Cluster
5. 测试kafka
5.1连接容器
进入kafka容器的命令行：
?
1
docker exec -ti kafka /bin/bash
进入kafka所在目录：
?
1
cd opt/kafka_2.13-2.8.1/
5.2 创建Replication为2，Partition为2的topic
?
1
bin/kafka-topics.sh –create –zookeeper zook1:2181 –replication-factor 2 –partitions 2 –topic partopic
5.3 查看topic的状态
在kafka容器中的 opt/kafka_2.12-1.1.0/ 目录下输入
?
1
bin/kafka-topics.sh –describe –zookeeper zook1:2181 –topic partopic
到此这篇关于docker搭建kafka集群的方法实现的文章就介绍到这了,更多相关docker搭建kafka集群内容请搜索服务器之家以前的文章或继续浏览下面的相关文章希望大家以后多多支持服务器之家！
原文链接：https://blog.csdn.net/sinat_36053757/article/details/123724748
声明： 猿站网有关资源均来自网络搜集与网友提供，任何涉及商业盈利目的的均不得使用，否则产生的一切后果将由您自己承担！ 本平台资源仅供个人学习交流、测试使用 所有内容请在下载后24小时内删除，制止非法恶意传播，不对任何下载或转载者造成的危害负任何法律责任！也请大家支持、购置正版！ 。本站一律禁止以任何方式发布或转载任何违法的相关信息访客发现请向站长举报，会员发帖仅代表会员个人观点，并不代表本站赞同其观点和对其真实性负责。本网站的资源部分来源于网络，如有侵权烦请发送邮件至：2697268773@qq.com进行处理。