kafka-在centos7上集群部署

张

张建站

2026/7/1 3:05:54

10分钟阅读

目录1.环境准备1.1环境概述1.2配置hostskafka01、kafka02、kafka031.3关闭防火墙kafka01、kafka02、kafka031.4设置时钟同步kafka01、kafka02、kafka031.5配置免密登录kafka011.6安装JDKkafka01、kafka02、kafka032.zookeeper集群环境部署2.1下载zookeeperkafka01、kafka02、kafka032.2解压zookeeperkafka01、kafka02、kafka032.3配置环境变量kafka01、kafka02、kafka032.4修改配置kafka01、kafka02、kafka032.5标识节点kafka01、kafka02、kafka032.6启动集群kafka01、kafka02、kafka033.kafka集群环境部署3.1下载kafkakafka01、kafka02、kafka033.2解压kafkakafka01、kafka02、kafka033.3配置环境变量kafka01、kafka02、kafka033.4修改配置kafka01、kafka02、kafka033.5启动集群kafka01、kafka02、kafka033.6验证群集kafka013.6.1创建主题3.6.2查看主题3.6.3删除主题3.6.4生产消息3.6.5消费消息1.环境准备1.1环境概述序号主机名称IP地址角色配置1kafka0110.9.254.41zookeeper01、kafka014C4G60G2kafka0210.9.254.42zookeeper02、kafka024C4G60G3kafka0310.9.254.43zookeeper03、kafka034C4G60G版本信息序号信息备注1系统版本CentOS 7.62JDK版本8u2013zookeeper版本3.7.04kafka版本2.8.01.2配置hostskafka01、kafka02、kafka03cat /etc/hosts10.9.254.41 kafka01 zookeeper0110.9.254.42 kafka02 zookeeper0210.9.254.43 kafka03 zookeeper03EOF[rootlocalhost ~]# hostnamectl set-hostname kafka01[rootlocalhost ~]# su - root[rootkafka01 ~]# cat /etc/hosts 10.9.254.41 kafka01 zookeeper01 10.9.254.42 kafka02 zookeeper02 10.9.254.43 kafka03 zookeeper03 EOF[rootkafka01 ~]# cat /etc/hosts127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4::1 localhost localhost.localdomain localhost6 localhost6.localdomain610.9.254.41 kafka01 zookeeper0110.9.254.42 kafka02 zookeeper0210.9.254.43 kafka03 zookeeper031.3关闭防火墙kafka01、kafka02、kafka03[rootkafka01 ~]# systemctl disable --now firewalld[rootkafka01 ~]# setenforce 0setenforce: SELinux is disabled[rootkafka01 ~]# sed -i s#SELINUXenforcing#SELINUXdisabled#g /etc/sysconfig/selinux[rootkafka01 ~]# sed -i s#SELINUXenforcing#SELINUXdisabled#g /etc/selinux/config[rootkafka01 ~]# cat /etc/selinux/config# This file controls the state of SELinux on the system.# SELINUX can take one of these three values:# enforcing - SELinux security policy is enforced.# permissive - SELinux prints warnings instead of enforcing.# disabled - No SELinux policy is loaded.SELINUXdisabled# SELINUXTYPE can take one of three values:# targeted - Targeted processes are protected,# minimum - Modification of targeted policy. Only selected processes are protected.# mls - Multi Level Security protection.SELINUXTYPEtargeted1.4设置时钟同步kafka01、kafka02、kafka03[rootkafka01 ~]# dnf -y install chrony[rootkafka01 ~]# ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime[rootkafka01 ~]# echo Asia/Shanghai /etc/timezone[rootkafka01 ~]# systemctl start chronyd[rootkafka01 ~]# systemctl enable chronyd[rootkafka01 ~]# vi /etc/chrony.conf#pool 2.pool.ntp.org iburstserver ntp.aliyun.com iburstserver cn.ntp.org.cn iburst[rootkafka01 ~]# systemctl restart chronyd[rootkafka01 ~]# chronyc sources -v.-- Source mode ^ server, peer, # local clock./ .- Source state * current best, combined, - not combined,| / x may be in error, ~ too variable, ? unusable.|| .- xxxx [ yyyy ] /- zzzz|| Reachability register (octal) -. | xxxx adjusted offset,|| Log2(Polling interval) --. | | yyyy measured offset,|| \ | | zzzz estimated error.|| | | \MS Name/IP address Stratum Poll Reach LastRx Last sample^* 203.107.6.88 2 6 17 3 1563us[5585us] /- 23ms^? 36.154.179.82 0 7 0 - 0ns[ 0ns] /- 0ns1.5配置免密登录kafka01[rootkafka01 ~]# yum -y install openssh-clients[rootkafka01 ~]# ssh-keygen -t rsaGenerating public/private rsa key pair.Enter file in which to save the key (/root/.ssh/id_rsa):Enter passphrase (empty for no passphrase):Enter same passphrase again:Your identification has been saved in /root/.ssh/id_rsa.Your public key has been saved in /root/.ssh/id_rsa.pub.The key fingerprint is:SHA256:jxdTBGMRDsLZA0LAupLJF2MD0i7x8ac9qGlyY0SY rootkafka01The keys randomart image is:---[RSA 2048]----| o . .o .o ||o.o .... o. ||oE.. oo o.. ||. * . o . . || oo.. S o || ...o || o. . o || .*. . . || ..oo |----[SHA256]-----[rootkafka01 ~]# ssh-copy-id -p 43215 -i .ssh/id_rsa.pub kafka01[rootkafka01 ~]# ssh-copy-id -p 43215 -i .ssh/id_rsa.pub kafka02[rootkafka01 ~]# ssh-copy-id -p 43215 -i .ssh/id_rsa.pub kafka031.6安装JDKkafka01、kafka02、kafka03[rootkafka01 ~]# rpm -qa | grep java[rootkafka01 ~]# ll /usr/local/src/total 172080-rw-r--r-- 1 root root 176209195 Aug 11 22:01 jdk-8u201-linux-x64.rpm[rootkafka01 ~]# rpm -ivh /usr/local/src/jdk-8u201-linux-x64.rpmwarning: /usr/local/src/jdk-8u201-linux-x64.rpm: Header V3 RSA/SHA256 Signature, key ID ec551f03: NOKEYPreparing... ################################# [100%]Updating / installing...1:jdk1.8-2000:1.8.0_201-fcs ################################# [100%]Unpacking JAR files...tools.jar...plugin.jar...javaws.jar...deploy.jar...rt.jar...jsse.jar...charsets.jar...localedata.jar...[rootkafka01 ~]# java -versionjava version 1.8.0_201Java(TM) SE Runtime Environment (build 1.8.0_201-b09)Java HotSpot(TM) 64-Bit Server VM (build 25.201-b09, mixed mode)[rootkafka01 ~]# vim /etc/profileJAVA_HOME/usr/java/jdk1.8.0_201-amd64JRE_HOME/usr/java/jdk1.8.0_201-amd64/jrePATH$PATH:$JAVA_HOME/bin:$JRE_HOME/binCLASSPATH.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:$JRE_HOME/libexport JAVA_HOME JRE_HOME PATH CLASSPATH[rootkafka01 ~]# source /etc/profile[rootkafka01 ~]# echo $PATH/usr/local/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/root/bin:/usr/java/jdk1.8.0_201-amd64/bin:/usr/java/jdk1.8.0_201-amd64/jre/bin2.zookeeper集群环境部署2.1下载zookeeperkafka01、kafka02、kafka03https://archive.apache.org/dist/zookeeper/https://archive.apache.org/dist/zookeeper/zookeeper-3.7.0/apache-zookeeper-3.7.0-bin.tar.gz[rootkafka01 ~]# wget -P /usr/local/src/ https://archive.apache.org/dist/zookeeper/zookeeper-3.7.0/apache-zookeeper-3.7.0-bin.tar.gz[rootkafka01 ~]# ll /usr/local/src/total 184180-rw-r--r-- 1 root root 12387614 Mar 27 18:12 apache-zookeeper-3.7.0-bin.tar.gz-rw-r--r-- 1 root root 176209195 Aug 11 22:01 jdk-8u201-linux-x64.rpm2.2解压zookeeperkafka01、kafka02、kafka03[rootkafka01 ~]# mkdir -p /opt/software[rootkafka01 ~]# tar -zxvf /usr/local/src/apache-zookeeper-3.7.0-bin.tar.gz -C /opt/software/[rootkafka01 ~]# mv /opt/software/apache-zookeeper-3.7.0-bin/ /opt/software/zookeeper-3.7.0[rootkafka01 ~]# ll /opt/software/zookeeper-3.7.0/total 36drwxr-xr-x 2 kaifa kaifa 4096 Mar 17 17:45 bindrwxr-xr-x 2 kaifa kaifa 77 Mar 17 17:45 confdrwxr-xr-x 5 kaifa kaifa 4096 Mar 17 17:45 docsdrwxr-xr-x 2 root root 4096 Aug 11 22:31 lib-rw-r--r-- 1 kaifa kaifa 11358 Mar 17 17:45 LICENSE.txt-rw-r--r-- 1 kaifa kaifa 432 Mar 17 17:45 NOTICE.txt-rw-r--r-- 1 kaifa kaifa 2214 Mar 17 17:45 README.md-rw-r--r-- 1 kaifa kaifa 3570 Mar 17 17:45 README_packaging.md2.3配置环境变量kafka01、kafka02、kafka03[rootkafka01 ~]# vim /etc/profileexport ZOOKEEPER_HOME/opt/software/zookeeper-3.7.0export PATH${ZOOKEEPER_HOME}/bin:$PATH[rootkafka01 ~]# source /etc/profile[rootkafka01 ~]# echo $ZOOKEEPER_HOME/opt/software/zookeeper-3.7.02.4修改配置kafka01、kafka02、kafka03[rootkafka01 ~]# cd /opt/software/zookeeper-3.7.0/conf/[rootkafka01 conf]# cp zoo_sample.cfg zoo.cfg[rootkafka01 conf]# vim zoo.cfgtickTime2000initLimit10syncLimit5dataDir/opt/software/zookeeper-3.7.0/zoo_datadataLogDir/opt/software/zookeeper-3.7.0/zoo_logsclientPort2181server.1kafka01:2888:3888server.2kafka02:2888:3888server.3kafka03:2888:3888[rootkafka01 conf]# mkdir /opt/software/zookeeper-3.7.0/zoo_data[rootkafka01 conf]# mkdir /opt/software/zookeeper-3.7.0/zoo_logs2.5标识节点kafka01、kafka02、kafka03[rootkafka01 conf]# echo 1 /opt/software/zookeeper-3.7.0/zoo_data/myid[rootkafka01 conf]# cat /opt/software/zookeeper-3.7.0/zoo_data/myid1[rootkafka02 conf]# echo 2 /opt/software/zookeeper-3.7.0/zoo_data/myid[rootkafka02 conf]# cat /opt/software/zookeeper-3.7.0/zoo_data/myid2[rootkafka03 conf]# echo 3 /opt/software/zookeeper-3.7.0/zoo_data/myid[rootkafka03 conf]# cat /opt/software/zookeeper-3.7.0/zoo_data/myid32.6启动集群kafka01、kafka02、kafka03[rootkafka01 ~]# zkServer.sh startZooKeeper JMX enabled by defaultUsing config: /opt/software/zookeeper-3.7.0/bin/../conf/zoo.cfgStarting zookeeper ... STARTED[rootkafka01 ~]# jps27591 Jps27326 QuorumPeerMain[rootkafka01 ~]# zkServer.sh statusZooKeeper JMX enabled by defaultUsing config: /opt/software/zookeeper-3.7.0/bin/../conf/zoo.cfgClient port found: 2181. Client address: localhost. Client SSL: false.Mode: follower[rootkafka01 ~]# zkServer.sh stopZooKeeper JMX enabled by defaultUsing config: /opt/software/zookeeper-3.7.0/bin/../conf/zoo.cfgStopping zookeeper ... STOPPED编写群起脚本kafka01[rootkafka01 ~]# vim /opt/software/zookeeper-3.7.0/bin/zookeeper#!/bin/bashred\e[91mgreen\e[92myellow\e[93mmagenta\e[95mcyan\e[96mnone\e[0mzk_home$ZOOKEEPER_HOMEcluster_array(kafka01 kafka02 kafka03)# 1. 获取输入参数个数如果没有参数直接退出if (($# 0)); thenecho -e ------------------- ${yellow}no args${none} -------------------exitfi# 2. 根据指令循环操作集群case $1 instart) {for host in ${cluster_array[]}; doecho -e ------------------- $yellow$host$none -------------------ssh -p 43215 $host $zk_home/bin/zkServer.sh startdone} ;;stop) {for host in ${cluster_array[]}; doecho -e ------------------- $yellow$host$none -------------------ssh -p 43215 $host $zk_home/bin/zkServer.sh stopdone} ;;status) {for host in ${cluster_array[]}; doecho -e ------------------- $yellow$host$none -------------------ssh -p 43215 $host $zk_home/bin/zkServer.sh statusdone} ;;esac[rootkafka01 ~]# chmod 755 /opt/software/zookeeper-3.7.0/bin/zookeeper[rootkafka01 ~]# zookeeper start------------------- kafka01 -------------------/usr/bin/javaZooKeeper JMX enabled by defaultUsing config: /opt/software/zookeeper-3.7.0/bin/../conf/zoo.cfgStarting zookeeper ... STARTED------------------- kafka02 -------------------/usr/bin/javaZooKeeper JMX enabled by defaultUsing config: /opt/software/zookeeper-3.7.0/bin/../conf/zoo.cfgStarting zookeeper ... STARTED------------------- kafka03 -------------------/usr/bin/javaZooKeeper JMX enabled by defaultUsing config: /opt/software/zookeeper-3.7.0/bin/../conf/zoo.cfgStarting zookeeper ... STARTED[rootkafka01 ~]# zookeeper status------------------- kafka01 -------------------/usr/bin/javaZooKeeper JMX enabled by defaultUsing config: /opt/software/zookeeper-3.7.0/bin/../conf/zoo.cfgClient port found: 2181. Client address: localhost. Client SSL: false.Mode: follower------------------- kafka02 -------------------/usr/bin/javaZooKeeper JMX enabled by defaultUsing config: /opt/software/zookeeper-3.7.0/bin/../conf/zoo.cfgClient port found: 2181. Client address: localhost. Client SSL: false.Mode: leader------------------- kafka03 -------------------/usr/bin/javaZooKeeper JMX enabled by defaultUsing config: /opt/software/zookeeper-3.7.0/bin/../conf/zoo.cfgClient port found: 2181. Client address: localhost. Client SSL: false.Mode: follower[rootkafka01 ~]# zookeeper stop------------------- kafka01 -------------------/usr/bin/javaZooKeeper JMX enabled by defaultUsing config: /opt/software/zookeeper-3.7.0/bin/../conf/zoo.cfgStopping zookeeper ... STOPPED------------------- kafka02 -------------------/usr/bin/javaZooKeeper JMX enabled by defaultUsing config: /opt/software/zookeeper-3.7.0/bin/../conf/zoo.cfgStopping zookeeper ... STOPPED------------------- kafka03 -------------------/usr/bin/javaZooKeeper JMX enabled by defaultUsing config: /opt/software/zookeeper-3.7.0/bin/../conf/zoo.cfgStopping zookeeper ... STOPPED3.kafka集群环境部署3.1下载kafkakafka01、kafka02、kafka03http://kafka.apache.org/downloadshttps://mirrors.tuna.tsinghua.edu.cn/apache/kafka/2.8.0/kafka_2.13-2.8.0.tgz[rootkafka01 ~]# wget -P /usr/local/src/ https://mirrors.tuna.tsinghua.edu.cn/apache/kafka/2.8.0/kafka_2.13-2.8.0.tgz[rootkafka01 ~]# ll /usr/local/srctotal 253912-rw-r--r-- 1 root root 12387614 Mar 27 18:12 apache-zookeeper-3.7.0-bin.tar.gz-rw-r--r-- 1 root root 176209195 Aug 11 22:01 jdk-8u201-linux-x64.rpm-rw-r--r-- 1 root root 71403603 Apr 19 12:44 kafka_2.13-2.8.0.tgz3.2解压kafkakafka01、kafka02、kafka03[rootkafka01 ~]# tar -zxvf /usr/local/src/kafka_2.13-2.8.0.tgz -C /opt/software/[rootkafka01 ~]# mv /opt/software/kafka_2.13-2.8.0/ /opt/software/kafka-2.8.0[rootkafka01 ~]# ll /opt/software/kafka-2.8.0/total 40drwxr-xr-x 3 root root 4096 Apr 14 22:34 bindrwxr-xr-x 3 root root 4096 Apr 14 22:34 configdrwxr-xr-x 2 root root 8192 Aug 11 23:37 libs-rw-r--r-- 1 root root 14515 Apr 14 22:28 LICENSEdrwxr-xr-x 2 root root 236 Apr 14 22:34 licenses-rw-r--r-- 1 root root 953 Apr 14 22:28 NOTICEdrwxr-xr-x 2 root root 44 Apr 14 22:34 site-docs3.3配置环境变量kafka01、kafka02、kafka03[rootkafka01 ~]# vim /etc/profileexport KAFKA_HOME/opt/software/kafka-2.8.0export PATH${KAFKA_HOME}/bin:$PATH[rootkafka01 ~]# source /etc/profile[rootkafka01 ~]# echo $KAFKA_HOME/opt/software/kafka-2.8.03.4修改配置kafka01、kafka02、kafka03[rootkafka01 ~]# mkdir /opt/software/kafka-2.8.0/kafka-logs[rootkafka01 ~]# vim /opt/software/kafka-2.8.0/config/server.propertiesbroker.id1log.dirs/opt/software/kafka-2.8.0/kafka-logszookeeper.connectkafka01:2181,kafka02:2181,kafka03:2181[rootkafka02 ~]# vim /opt/software/kafka-2.8.0/config/server.propertiesbroker.id2log.dirs/opt/software/kafka-2.8.0/kafka-logszookeeper.connectkafka01:2181,kafka02:2181,kafka03:2181[rootkafka03 ~]# vim /opt/software/kafka-2.8.0/config/server.propertiesbroker.id3log.dirs/opt/software/kafka-2.8.0/kafka-logszookeeper.connectkafka01:2181,kafka02:2181,kafka03:2181[rootkafka01 ~]# vim /opt/software/kafka-2.8.0/config/producer.propertiesbootstrap.serverskafka01:9092,kafka02:9092,kafka03:9092[rootkafka01 ~]# vim /opt/software/kafka-2.8.0/config/consumer.propertiesbootstrap.serverskafka01:9092,kafka02:9092,kafka03:90923.5启动集群kafka01、kafka02、kafka03[rootkafka01 ~]# kafka-server-start.sh -daemon /opt/software/kafka-2.8.0/config/server.properties[rootkafka01 ~]# kafka-server-stop.sh编写群起脚本[rootkafka01 ~]# vim /opt/software/kafka-2.8.0/bin/kafka#! /bin/bashcase $1 instart){for i in kafka01 kafka02 kafka03doecho --------启动 $i Kafka-------ssh -p 43215 $i /opt/software/kafka-2.8.0/bin/kafka-server-start.sh -daemon /opt/software/kafka-2.8.0/config/server.properties done};;stop){for i in kafka01 kafka02 kafka03doecho --------停止 $i Kafka-------ssh -p 43215 $i /opt/software/kafka-2.8.0/bin/kafka-server-stop.sh stopdone};;esac[rootkafka01 ~]# chmod 755 /opt/software/kafka-2.8.0/bin/kafka[rootkafka01 ~]# kafka stop--------停止 kafka01 Kafka-------No kafka server to stop--------停止 kafka02 Kafka-------No kafka server to stop--------停止 kafka03 Kafka-------No kafka server to stop[rootkafka01 ~]# kafka start--------启动 kafka01 Kafka---------------启动 kafka02 Kafka---------------启动 kafka03 Kafka-------[rootkafka01 ~]# jps31122 Kafka31277 Jps27326 QuorumPeerMain3.6验证群集kafka013.6.1创建主题创建一个副本数为 1、分区数为 3、名为 test 的主题[rootkafka01 ~]# kafka-topics.sh --zookeeper kafka01:2181 --create --replication-factor 1 --partitions 3 --topic testCreated topic test.3.6.2查看主题列出所有主题[rootkafka01 ~]# kafka-topics.sh --zookeeper kafka01:2181 --listtest查看某个主题的详情[rootkafka01 ~]# kafka-topics.sh --zookeeper kafka01:2181 --describe --topic testTopic: test TopicId: 7_2bA04STq2UGGFMX30RQQ PartitionCount: 3 ReplicationFactor: 1 Configs:Topic: test Partition: 0 Leader: 3 Replicas: 3 Isr: 3Topic: test Partition: 1 Leader: 1 Replicas: 1 Isr: 1Topic: test Partition: 2 Leader: 2 Replicas: 2 Isr: 2[rootkafka01 ~]# ll /opt/software/kafka-2.8.0/kafka-logs/total 16-rw-r--r-- 1 root root 0 Aug 12 00:01 cleaner-offset-checkpoint-rw-r--r-- 1 root root 4 Aug 12 00:21 log-start-offset-checkpoint-rw-r--r-- 1 root root 88 Aug 12 00:12 meta.properties-rw-r--r-- 1 root root 13 Aug 12 00:21 recovery-point-offset-checkpoint-rw-r--r-- 1 root root 13 Aug 12 00:21 replication-offset-checkpointdrwxr-xr-x 2 root root 141 Aug 12 00:18 test-1[rootkafka02 ~]# ll /opt/software/kafka-2.8.0/kafka-logs/total 16-rw-r--r-- 1 root root 0 Aug 12 00:12 cleaner-offset-checkpoint-rw-r--r-- 1 root root 4 Aug 12 00:22 log-start-offset-checkpoint-rw-r--r-- 1 root root 88 Aug 12 00:12 meta.properties-rw-r--r-- 1 root root 13 Aug 12 00:22 recovery-point-offset-checkpoint-rw-r--r-- 1 root root 13 Aug 12 00:22 replication-offset-checkpointdrwxr-xr-x 2 root root 141 Aug 12 00:18 test-2[rootkafka03 ~]# ll /opt/software/kafka-2.8.0/kafka-logs/total 16-rw-r--r-- 1 root root 0 Aug 12 00:12 cleaner-offset-checkpoint-rw-r--r-- 1 root root 4 Aug 12 00:22 log-start-offset-checkpoint-rw-r--r-- 1 root root 88 Aug 12 00:12 meta.properties-rw-r--r-- 1 root root 13 Aug 12 00:22 recovery-point-offset-checkpoint-rw-r--r-- 1 root root 13 Aug 12 00:23 replication-offset-checkpointdrwxr-xr-x 2 root root 141 Aug 12 00:18 test-03.6.3删除主题[rootkafka01 ~]# kafka-topics.sh --zookeeper kafka01:2181 --delete --topic testTopic test is marked for deletion.Note: This will have no impact if delete.topic.enable is not set to true.3.6.4生产消息在kafka01中生产消息[rootkafka01 ~]# kafka-console-producer.sh --broker-list kafka01:9092 --topic testhelloworld3.6.5消费消息在kafka02中消费消息[rootkafka02 ~]# kafka-console-consumer.sh --bootstrap-server kafka02:9092 --topic testhelloworld^CProcessed a total of 2 messages