小趙同學
0%

Kafka

发表于 更新于

kafka

Kafka

概述

https://kafka.apache.org/documentation/#quickstart

安装

服务器信息

服务器IP 部署列表
172.20.3.31 zk + kafka
172.20.3.32 zk + kafka
172.20.3.33 zk + kafka

基础环境

1
2
3
mkdir -p /data/servers/kafka/
mkdir -p /data/servers/kafka/kafka_data
mkdir -p /data/servers/kafka/zookeeper_data

下载解压安装包

https://kafka.apache.org/downloads

https://dlcdn.apache.org/kafka/3.2.0/kafka_2.13-3.2.0.tgz

1
2
wget https://dlcdn.apache.org/kafka/3.2.0/kafka_2.13-3.2.0.tgz -N -P /tmp
tar zxf kafka_2.13-3.2.0.tgz -C /data/servers/

编辑配置文件

Zookeeper

config/zookeeper.properties

1
2
3
# 修改默认存储路径
# dataDir=/tmp/zookeeper
dataDir=/data/servers/kafka/zookeeper_data
1
2
3
4
5
# 完整配置
dataDir=/data/servers/kafka/zookeeper_data
clientPort=2181
maxClientCnxns=0
admin.enableServer=false
Kafka

config/server.properties

1
2
3
4
5
6
7
8
9
10
# 确保broker id唯一
broker.id=0

# 修改默认存储路径
# dataDir=/tmp/kafka-logs
log.dirs=/data/servers/kafka/kafka_data

# 修改zookeeper连接地址
#zookeeper.connect=localhost:2181
zookeeper.connect=172.20.3.31:2181,172.20.3.32:2181,172.20.3.33:2181
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
# 完整配置：
broker.id=0
listeners=PLAINTEXT://172.20.3.31:9092
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/data/servers/kafka/kafka_data
num.partitions=1
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=172.20.3.31:2181,172.20.3.32:2181,172.20.3.33:2181
zookeeper.connection.timeout.ms=18000
group.initial.rebalance.delay.ms=0

启动服务

zk

1
nohup bin/zookeeper-server-start.sh config/zookeeper.properties > zookeeper.log 2>&1 &

kafka

1
nohup bin/kafka-server-start.sh config/server.properties > kafka.log 2>&1 &

常用命令

查看topics列表

1
2
3
4
bin/kafka-topics.sh --bootstrap-server ${ip_addr}:9092 --list

# demo
bin/kafka-topics.sh --bootstrap-server 172.20.3.31:9092,172.20.3.32:9092,172.20.3.33:9092 --list

创建topics

1
2
3
4
bin/kafka-topics.sh --bootstrap-server ${ip_addr}:9092 --create --topic ${topic_name}

# demo
bin/kafka-topics.sh --bootstrap-server 172.20.3.31:9092,172.20.3.32:9092,172.20.3.33:9092 --create --topic quickstart-events

查看topic详情

1
2
3
4
bin/kafka-topics.sh --bootstrap-server ${ip_addr}:9092 --describe --topic ${topic_name}

# demo
bin/kafka-topics.sh --bootstrap-server 172.20.3.31:9092,172.20.3.32:9092,172.20.3.33:9092 --describe --topic quickstart-events

生产消息

1
2
3
4
bin/kafka-console-producer.sh --bootstrap-server ${ip_addr}:9092 --topic ${topic_name}

# demo
bin/kafka-console-producer.sh --bootstrap-server 172.20.3.31:9092,172.20.3.32:9092,172.20.3.33:9092 --topic quickstart-events

查看消息

1
2
3
4
bin/kafka-console-consumer.sh --bootstrap-server ${ip_addr}:9092 --from-beginning --topic ${topic_name}

# demo
bin/kafka-console-consumer.sh --bootstrap-server 172.20.3.31:9092,172.20.3.32:9092,172.20.3.33:9092 --from-beginning --topic quickstart-events

查看运行的消费组

1
2
3
4
bin/kafka-consumer-groups.sh --bootstrap-server ${ip_addr}:9092 --list

# demo
bin/kafka-consumer-groups.sh --bootstrap-server 172.20.3.31:9092,172.20.3.32:9092,172.20.3.33:9092 --list