做网站点,注册公司需要多久的时间,网络维护基础知识,wordpress 模板破解版大数据之kafka应用 2024启kafkakafka常见命令生产上重放信息jmxjmx的配置和开启jmx的使用jmx例子一 jmx例子二jmx例子三jmx例子四#xff08;special#xff09; 2024启kafka
kafka常见命令 指定jmx端口启动kafka lsof -i :9999 JMX_PORT9999 /opt/kafka_2.12-3.1.0/bin/kaf… 大数据之kafka应用 2024启kafkakafka常见命令生产上重放信息jmxjmx的配置和开启jmx的使用jmx例子一 jmx例子二jmx例子三jmx例子四special 2024启kafka
kafka常见命令 指定jmx端口启动kafka lsof -i :9999 JMX_PORT9999 /opt/kafka_2.12-3.1.0/bin/kafka-server-start.sh -daemon /opt/kafka_2.12-3.1.0/config/server.properties新建topic副本不能大于kafka_server数 /opt/kafka_2.12-3.1.0/bin/kafka-topics.sh --bootstrap-server only:9092 --partitions 3 --replication-factor 1 --create --topic knowScript查看topic的详情 /opt/kafka_2.12-3.1.0/bin/kafka-topics.sh --bootstrap-server only:9092 --describe --topic knowScript罗列所有的topic /opt/kafka_2.12-3.1.0/bin/kafka-topics.sh --bootstrap-server only:9092 --list给topic拓展分区:只增大不减小 /opt/kafka_2.12-3.1.0/bin/kafka-topics.sh --bootstrap-server only:9092 --alter --topic knowScript --partitions 4查看topic的磁盘大小单位为Byte /opt/kafka_2.12-3.1.0/bin/kafka-log-dirs.sh --bootstrap-server only:9092 --topic-list knowScript --describe | grep -oP (?size:)\d | awk { sum $1 } END { print sum }删除topic /opt/kafka_2.12-3.1.0/bin/kafka-topics.sh --bootstrap-server only:9092 --topic knowScript --delete查看消费者组列表 /opt/kafka_2.12-3.1.0/bin/kafka-consumer-groups.sh --bootstrap-server only:9092 --list查看特定消费者组的消费情况 /opt/kafka_2.12-3.1.0/bin/kafka-consumer-groups.sh --bootstrap-server only:9092 --describe --group consumeByOffsetId新建一个消费者组:消费一个topic即可 /opt/kafka_2.12-3.1.0/bin/kafka-console-consumer.sh --bootstrap-server only:9092 --topic consumerTopicByOffset --group makeGroup删除一个消费者组 /opt/kafka_2.12-3.1.0/bin/kafka-consumer-groups.sh --bootstrap-server only:9092 --delete --group makeGroup往特定topic生产message不指定key /opt/kafka_2.12-3.1.0/bin/kafka-console-producer.sh --bootstrap-server only:9092 --topic knowScript往特定topic生产message指定key /opt/kafka_2.12-3.1.0/bin/kafka-console-producer.sh --bootstrap-server only:9092 --topic knowScript --property parse.keytrue --property key.separator,消费指定topic的message–from-beginning,默认为最近的消息消费 /opt/kafka_2.12-3.1.0/bin/kafka-console-consumer.sh --bootstrap-server only:9092 --topic knowScript --from-beginning消费topic的message限制条数 /opt/kafka_2.12-3.1.0/bin/kafka-console-consumer.sh --bootstrap-server only:9092 --topic knowScript --offset earliest --partition 0 --max-messages 3消费指定topic的message:指定分区的offset /opt/kafka_2.12-3.1.0/bin/kafka-console-consumer.sh --bootstrap-server only:9092 --topic knowScript --offset latest --partition 0/opt/kafka_2.12-3.1.0/bin/kafka-console-consumer.sh --bootstrap-server only:9092 --topic knowScript --offset earliest --partition 0/opt/kafka_2.12-3.1.0/bin/kafka-console-consumer.sh --bootstrap-server only:9092 --topic knowScript --offset 10 --partition 0查看kafka的log文件明文内容 /opt/kafka_2.12-3.1.0/bin/kafka-dump-log.sh --files /opt/kafka_2.12-3.1.0/data/kafka_logs/knowScript-0/00000000000000000000.log -deep-iteration --print-data-log查看topic的message总数 每个分区的最小offset /opt/kafka_2.12-3.1.0/bin/kafka-run-class.sh kafka.tools.GetOffsetShell --bootstrap-server only:9092 -topic knowScript --time -2每个分区的最大offset /opt/kafka_2.12-3.1.0/bin/kafka-run-class.sh kafka.tools.GetOffsetShell --bootstrap-server only:9092 -topic knowScript --time -1查看指定消费者组的消费情况offset和lag /opt/kafka_2.12-3.1.0/bin/kafka-consumer-groups.sh --bootstrap-server only:9092 --describe --group consumeByOffsetId重设指定消费者组消费指定的offset每个分区 打印结果没有执行生产使用这个验证 /opt/kafka_2.12-3.1.0/bin/kafka-consumer-groups.sh --bootstrap-server only:9092 --group consumeByOffsetId --reset-offsets --to-offset 4 --topic consumerTopicByOffset --dry-run直接执行 /opt/kafka_2.12-3.1.0/bin/kafka-consumer-groups.sh --bootstrap-server only:9092 --group consumeByOffsetId --reset-offsets --to-offset 4 --topic consumerTopicByOffset --execute重设指定消费者组指定分区的指定offsettopic后使用 :$partitionNum来设定指定的partition /opt/kafka_2.12-3.1.0/bin/kafka-consumer-groups.sh --bootstrap-server only:9092 --group consumeByOffsetId --reset-offsets --to-offset 3 --topic consumerTopicByOffset:1 --execute指定消费者组从最新的offset进行消费没有指定group则全部都消费指定了会从消费者组最新的offset开始消费 /opt/kafka_2.12-3.1.0/bin/kafka-console-consumer.sh --bootstrap-server only:9092 --topic consumerTopicByOffset --group consumeByOffsetId --from-beginning生产message基准测试; num-records生产的message条数,through为-1表示不限制吞吐量,record-size表示每条record的大小为1024K, producer-props后面跟着kv的producer属性配置 /opt/kafka_2.12-3.1.0/bin/kafka-producer-perf-test.sh --topic test_producer_perf --num-records 10000000 --throughput -1 --record-size 1024 --producer-props bootstrap.serversonly:9092 acks-1 linger.ms2000 compression.typelz4消费message基准测试 /opt/kafka_2.12-3.1.0/bin/kafka-consumer-perf-test.sh --broker-list only:9092 --messages 10000000 --topic test_producer_perf生产上重放信息 查看指定消费者组的消费情况offset /opt/kafka_2.12-3.1.0/bin/kafka-consumer-groups.sh --bootstrap-server only:9092 --describe --group consumeByOffsetId重设指定消费者组消费指定的offset每个分区 /opt/kafka_2.12-3.1.0/bin/kafka-consumer-groups.sh --bootstrap-server only:9092 --group consumeByOffsetId --reset-offsets --to-offset 4 --topic consumerTopicByOffset --execute打印结果没有执行 /opt/kafka_2.12-3.1.0/bin/kafka-consumer-groups.sh --bootstrap-server only:9092 --group consumeByOffsetId --reset-offsets --to-offset 4 --topic consumerTopicByOffset --dry-runother:重设指定消费者组指定分区的指定offsettopic后使用 :partitionNum来设定指定的partition /opt/kafka_2.12-3.1.0/bin/kafka-consumer-groups.sh --bootstrap-server only:9092 --group consumeByOffsetId --reset-offsets --to-offset 3 --topic consumerTopicByOffset:1 --dry-run指定消费者组从最新的offset进行消费没有指定group则全部都消费指定了会从消费者组最新的offset开始消费 /opt/kafka_2.12-3.1.0/bin/kafka-console-consumer.sh --bootstrap-server only:9092 --topic consumerTopicByOffset --group consumeByOffsetId --from-beginning指定分区指定offset进行开始消费查看某消费者组在特定的分区丢失了哪些消息 /opt/kafka_2.12-3.1.0/bin/kafka-console-consumer.sh --bootstrap-server only:9092 --topic consumerTopicByOffset --offset 10 --partition 0jmx
jmx的配置和开启 配置jmx JMX_PORT9999 /opt/kafka_2.12-3.1.0/bin/kafka-server-start.sh -daemon /opt/kafka_2.12-3.1.0/config/server.properties配置文件配置更稳定
jmx的使用
jmx例子一 参考url的MBEAN https://docs.confluent.io/platform/current/kafka/monitoring.html#kafka-monitoring-metrics-broker 找到category类别下面的MBEAN: kafka.server:typeBrokerTopicMetrics,nameBytesInPerSec,topic{topicName}获取到对应的MBEAN的指标值 /opt/kafka_2.12-3.1.0/bin/kafka-run-class.sh kafka.tools.JmxTool --object-name kafka.server:typeBrokerTopicMetrics,nameBytesInPerSec --jmx-url service:jmx:rmi:///jndi/rmi://only:9999/jmxrmi输出列有 time,kafka.server:typeBrokerTopicMetrics,nameBytesInPerSec:Count,kafka.server:typeBrokerTopicMetrics,nameBytesInPerSec:EventType,kafka.server:typeBrokerTopicMetrics,nameBytesInPerSec:FifteenMinuteRate,kafka.server:typeBrokerTopicMetrics,nameBytesInPerSec:FiveMinuteRate,kafka.server:typeBrokerTopicMetrics,nameBytesInPerSec:MeanRate,kafka.server:typeBrokerTopicMetrics,nameBytesInPerSec:OneMinuteRate,kafka.server:typeBrokerTopicMetrics,nameBytesInPerSec:RateUnit寻找上面的name的值如 FifteenMinuteRate /opt/kafka_2.12-3.1.0/bin/kafka-run-class.sh kafka.tools.JmxTool --object-name kafka.server:typeBrokerTopicMetrics,nameBytesInPerSec --jmx-url service:jmx:rmi:///jndi/rmi://only:9999/jmxrmi --date-format YYYY-MM-dd HH:mm:ss --attributes FifteenMinuteRate --reporting-interval 5000–date-format格式化时间 、 --reporting-interval指定更新时间间隔
jmx例子二 找到category类别下面的MBEAN kafka.controller:typeControllerEventManager,nameEventQueueTimeMs获取到对应的MBEAN的指标值 /opt/kafka_2.12-3.1.0/bin/kafka-run-class.sh kafka.tools.JmxTool --object-name kafka.controller:typeControllerEventManager,nameEventQueueTimeMs --jmx-url service:jmx:rmi:///jndi/rmi://only:9999/jmxrmi输出列有 time,kafka.controller:typeControllerEventManager,nameEventQueueTimeMs:50thPercentile,kafka.controller:typeControllerEventManager,nameEventQueueTimeMs:75thPercentile,kafka.controller:typeControllerEventManager,nameEventQueueTimeMs:95thPercentile,kafka.controller:typeControllerEventManager,nameEventQueueTimeMs:98thPercentile,kafka.controller:typeControllerEventManager,nameEventQueueTimeMs:999thPercentile,kafka.controller:typeControllerEventManager,nameEventQueueTimeMs:99thPercentile,kafka.controller:typeControllerEventManager,nameEventQueueTimeMs:Count,kafka.controller:typeControllerEventManager,nameEventQueueTimeMs:Max,kafka.controller:typeControllerEventManager,nameEventQueueTimeMs:Mean,kafka.controller:typeControllerEventManager,nameEventQueueTimeMs:Min,kafka.controller:typeControllerEventManager,nameEventQueueTimeMs:StdDev寻找上面的name的值如 75thPercentile、 95thPercentile /opt/kafka_2.12-3.1.0/bin/kafka-run-class.sh kafka.tools.JmxTool --object-name kafka.controller:typeControllerEventManager,nameEventQueueTimeMs --jmx-url service:jmx:rmi:///jndi/rmi://only:9999/jmxrmi --attributes 75thPercentile/opt/kafka_2.12-3.1.0/bin/kafka-run-class.sh kafka.tools.JmxTool --object-name kafka.controller:typeControllerEventManager,nameEventQueueTimeMs --jmx-url service:jmx:rmi:///jndi/rmi://only:9999/jmxrmi --attributes 95thPercentilejmx例子三 找到category类别下面的MBEAN kafka.network:typeRequestChannel,nameResponseQueueSize获取到对应的MBEAN的指标值 /opt/kafka_2.12-3.1.0/bin/kafka-run-class.sh kafka.tools.JmxTool --object-name kafka.network:typeRequestChannel,nameResponseQueueSize --jmx-url service:jmx:rmi:///jndi/rmi://only:9999/jmxrmi输出列有 time,kafka.network:typeRequestChannel,nameResponseQueueSize:Value寻找上面的name的值如 Value /opt/kafka_2.12-3.1.0/bin/kafka-run-class.sh kafka.tools.JmxTool --object-name kafka.network:typeRequestChannel,nameResponseQueueSize --jmx-url service:jmx:rmi:///jndi/rmi://only:9999/jmxrmi --attributes Valuejmx例子四special 并不是所有的MBEAN都有对应的jmx指标值例如:kafka.log:typeLog,nameLogEndOffset 下面的命令会报错 /opt/kafka_2.12-3.1.0/bin/kafka-run-class.sh kafka.tools.JmxTool --object-name kafka.log:typeLog,nameLogEndOffset --jmx-url service:jmx:rmi:///jndi/rmi://only:9999/jmxrmi但是从jconsole localhost:9999的弹窗MBEAN可以找到;上面更深层的信息 /opt/kafka_2.12-3.1.0/bin/kafka-run-class.sh kafka.tools.JmxTool --object-name kafka.log:typeLog,nameLogEndOffset,topicflink2kafka,partition0 --jmx-url service:jmx:rmi:///jndi/rmi://only:9999/jmxrmi
