Kafka CLI Quick Reference

Q: How do I check consumer lag in Kafka?

Run kafka-consumer-groups.sh with --bootstrap-server, --describe and --group followed by the group name. For each partition it prints CURRENT-OFFSET, the position the group last committed, LOG-END-OFFSET, the offset the next message will get, and LAG, the difference between the two. Use --all-groups to see every group at once. Lag that grows steadily means consumers cannot keep up, while lag stuck on one partition usually points to a hot key or a stalled consumer. The figures come from committed offsets, so a group that commits rarely looks further behind than it is.

Q: Why is my Kafka consumer not receiving old messages?

A consumer group with no committed offsets starts where auto.offset.reset says, and the default is latest, so it only sees messages produced after it joined. Use --from-beginning with the console consumer, set auto.offset.reset to earliest in your client, or reset an existing group’s offsets with kafka-consumer-groups.sh. If the group already has committed offsets, those win over both settings. Also check retention: messages older than retention.ms have already been deleted and cannot be read by anyone.

Q: Can I reduce the number of partitions in a Kafka topic?

No. Kafka can add partitions to a topic but never remove them, because each partition is an independent log and deleting one would delete its data. Even adding partitions has a cost: the default partitioner maps a key to a partition by hashing it against the partition count, so keys move to new partitions and per-key ordering breaks across the change. If you really need fewer partitions, create a new topic with the right count, copy or re-produce the data into it, and switch producers and consumers over.

Q: What is the difference between Kafka and a message queue?

A traditional queue such as RabbitMQ or SQS removes a message once a consumer acknowledges it and hands messages to consumers one at a time. Kafka is a replicated, append-only log: messages stay for the retention period whether or not anyone read them, each consumer group tracks its own offset, and any number of groups can read the same data independently. That makes replay, auditing and adding new consumers easy, while ordering holds only within a partition and parallelism is capped by the partition count.

Create topics, produce and consume messages, and inspect consumer-group lag from the command line.

The rows use the Apache Kafka scripts in bin/; Confluent packages drop the .sh. Every tool needs --bootstrap-server host:9092, left out below to save space, and a secured cluster also needs a client properties file (--command-config).

The model: a topic is split into partitions, each an append-only log in which every message gets an offset. Producers append; consumers in a group divide the partitions among themselves and commit how far they have read. Messages stay until retention removes them, read or not, which is what makes replay possible.

kafka-topics.sh

--create --topic orders: with --partitions 6 and --replication-factor 3; RF cannot exceed the broker count
--list · --describe: all topics · partitions, leaders, replicas and ISR for --topic orders
--alter --partitions 12: add partitions — never fewer, and existing keys then map to different partitions
--delete --topic orders: delete a topic (allowed by default via delete.topic.enable)
--config k=v: set topic config at create time, e.g. retention.ms=86400000 (one day)
--unavailable-partitions: with --describe: partitions without a leader; --under-replicated-partitions shows lagging replicas
--if-not-exists: make --create safe to rerun in scripts; --if-exists does the same for alter and delete
--command-config c.props: client settings for a secured cluster: SASL, TLS, credentials
--topics-with-overrides: with --describe: only topics whose config differs from the broker defaults

kafka-console-producer.sh

--topic orders: one message per line from stdin; Ctrl-D to finish
parse.key=true: with --property, plus key.separator=:, reads key:value lines
acks=all: via --producer-property: wait for every in-sync replica — the default since Kafka 3.0
--topic t < file.jsonl: bulk-load a file, one message per line

kafka-console-consumer.sh

--from-beginning: read from the oldest retained offset; without it you only see new messages
--group my-group: join a consumer group, so offsets are committed and partitions shared
print.key=true: --property print.key=true; also print.timestamp, print.partition, print.offset
--partition 0 --offset 42: read one partition from an exact offset; --offset requires --partition
--max-messages 10: stop after 10 messages — peek without scrolling forever
--timeout-ms 5000: exit when nothing arrives for 5 seconds — handy in scripts
read_committed: with --isolation-level: hide writes from aborted transactions

kafka-consumer-groups.sh

--list: every consumer group the cluster knows
--describe --group g: per partition: CURRENT-OFFSET, LOG-END-OFFSET, LAG and the consumer that owns it
--describe --all-groups: lag for every group in one call
--members --verbose: with --describe: each member and its assigned partitions
--state: with --describe: Stable, Empty or rebalancing, plus the coordinator broker
--reset-offsets: --to-earliest, --to-latest, --shift-by -100; only a dry run until --execute
--to-datetime: reset to a timestamp such as 2026-01-01T00:00:00.000 — replay from a point in time
--by-duration PT1H: reset to one hour ago (an ISO-8601 duration)
Reset gotcha: offsets reset only while the group has no active members — stop the consumers first
--delete --group g: delete an inactive group and its committed offsets
--delete-offsets: with --group g --topic t: forget the group’s offsets for one topic only

kafka-configs.sh & key settings

--entity-type topics: --entity-name orders --describe shows overrides; add --all for every value
--alter --add-config: change a topic live, e.g. retention.ms=3600000; --delete-config restores the default
retention.ms · .bytes: how long or how large a partition grows before old segments are deleted
cleanup.policy=compact: keep only the latest value per key — changelogs and table-like topics
min.insync.replicas=2: with acks=all and RF 3, writes fail rather than land on fewer than 2 replicas
enable.auto.commit: commits on a timer inside poll() — safe only if each batch is finished before the next poll
linger.ms · batch.size: producer batching: wait a few ms to fill bigger batches; add compression.type=lz4
auto.offset.reset: where a group with no committed offset starts: latest (default) or earliest

Other tools

kafka-get-offsets.sh: --topic orders --time -1 latest offsets, -2 earliest; older Kafka: GetOffsetShell
kafka-delete-records.sh: truncate partitions up to the offsets in --offset-json-file — removes poison data
kafka-leader-election.sh: --election-type preferred with --all-topic-partitions rebalances leaders after a restart
kafka-acls.sh --list: show ACLs; --add --allow-principal User:app grants access per topic or group
kafka-metadata-quorum.sh: describe --status shows the KRaft controller quorum (no ZooKeeper)

Mental model

Partition = ordering unit: Order is guaranteed within a partition, never across them. Same key → same partition → ordered.
Consumer group: Each partition is read by exactly one consumer in the group — parallelism is capped at the partition count.
Offset: A consumer’s bookmark per partition. Committing it is what makes progress durable across restarts.
Lag: LOG-END-OFFSET − CURRENT-OFFSET: how far behind a consumer is. Growing lag = consumers can’t keep up.
Retention ≠ queue: Kafka keeps messages for a retention window even after consumption — replay is a feature, not a bug.
Rebalance: a consumer joining, leaving or exceeding max.poll.interval.ms reassigns partitions; processing pauses
Delivery guarantee: at-least-once by default: commit after processing and make handlers idempotent

Recipes

Create a topic and check it end to end

kafka-topics.sh --bootstrap-server localhost:9092 --create --topic orders --partitions 6 --replication-factor 3
echo 'o-1:{"total":42}' | kafka-console-producer.sh --bootstrap-server localhost:9092 --topic orders --property parse.key=true --property key.separator=:
kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic orders --from-beginning --property print.key=true --max-messages 1

Replication factor 3 needs at least three brokers; use 1 on a single-broker dev cluster. The consumer prints the key and value, then exits after one message.

Diagnose a lagging consumer group

kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group billing
kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group billing --members --verbose

Lag rising on every partition means the consumers are too slow overall (scale out, up to the partition count); lag on one partition points to a hot key or a stuck consumer.

Replay the last hour of events

kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group billing --topic orders --reset-offsets --by-duration PT1H --dry-run
kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group billing --topic orders --reset-offsets --by-duration PT1H --execute

Stop the group first, because offsets only reset when it has no active members. The dry run prints the new offset for each partition before anything changes.

Shorten retention on a topic that is filling the disk

kafka-configs.sh --bootstrap-server localhost:9092 --entity-type topics --entity-name logs --alter --add-config retention.ms=21600000
kafka-configs.sh --bootstrap-server localhost:9092 --entity-type topics --entity-name logs --describe

Six hours of retention applies live, but space returns only as closed segments age out, so it is not instant. --delete-config retention.ms restores the broker default.

Measure producer and consumer throughput

kafka-producer-perf-test.sh --topic perf --num-records 1000000 --record-size 1000 --throughput -1 --producer-props bootstrap.servers=localhost:9092 acks=all
kafka-consumer-perf-test.sh --bootstrap-server localhost:9092 --topic perf --messages 1000000

--throughput -1 removes the rate cap, and the producer summary reports records per second, MB per second and latency percentiles. Rerun with acks=1 to see what durability costs.

Questions about Kafka CLI Cheat Sheet

How do I check consumer lag in Kafka?

Run kafka-consumer-groups.sh with --bootstrap-server, --describe and --group followed by the group name. For each partition it prints CURRENT-OFFSET, the position the group last committed, LOG-END-OFFSET, the offset the next message will get, and LAG, the difference between the two. Use --all-groups to see every group at once. Lag that grows steadily means consumers cannot keep up, while lag stuck on one partition usually points to a hot key or a stalled consumer. The figures come from committed offsets, so a group that commits rarely looks further behind than it is.

Why is my Kafka consumer not receiving old messages?

A consumer group with no committed offsets starts where auto.offset.reset says, and the default is latest, so it only sees messages produced after it joined. Use --from-beginning with the console consumer, set auto.offset.reset to earliest in your client, or reset an existing group’s offsets with kafka-consumer-groups.sh. If the group already has committed offsets, those win over both settings. Also check retention: messages older than retention.ms have already been deleted and cannot be read by anyone.

Can I reduce the number of partitions in a Kafka topic?

No. Kafka can add partitions to a topic but never remove them, because each partition is an independent log and deleting one would delete its data. Even adding partitions has a cost: the default partitioner maps a key to a partition by hashing it against the partition count, so keys move to new partitions and per-key ordering breaks across the change. If you really need fewer partitions, create a new topic with the right count, copy or re-produce the data into it, and switch producers and consumers over.

What is the difference between Kafka and a message queue?

A traditional queue such as RabbitMQ or SQS removes a message once a consumer acknowledges it and hands messages to consumers one at a time. Kafka is a replicated, append-only log: messages stay for the retention period whether or not anyone read them, each consumer group tracks its own offset, and any number of groups can read the same data independently. That makes replay, auditing and adding new consumers easy, while ordering holds only within a partition and parallelism is capped by the partition count.

Go deeper

MCP cheatsheet →Lab: Queue Backpressure →