Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Kafkacat would read less records than expected #423

Open
YordanPavlov opened this issue Mar 9, 2023 · 0 comments
Open

Kafkacat would read less records than expected #423

YordanPavlov opened this issue Mar 9, 2023 · 0 comments

Comments

@YordanPavlov
Copy link

So this is most probably, some lack of knowledge on my part, but when reading a topic from start to finish with Kafkacat I would expect the number of records read to match exactly the offset of the topic reported. The only reason for reading less that I could think of is non commited messages, that's why I specify the 'read uncommitted' flag. Here is the command:

kafkacat -b blockchain-kafka-kafka-0.blockchain-kafka-kafka-headless.default.svc.cluster.local:9092 -C -t my_topic -p 26 -K: -e -X isolation.level=read_uncommitted | wc -l
% Reached end of topic my_topic [26] at offset 498376: exiting
441825

As seen, the last offset seen is 498376, that's also the value I would see if I use the Kafka console tools:

kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list KAFKA_URL --topic TOPIC --time -1

However, only 441825 records are shown. What is the reason behind this, the version I use is this:

Version 1.6.0 (JSON, Transactions, librdkafka 1.6.0 builtin.features=gzip,snappy,ssl,sasl,regex,lz4,sasl_gssapi,sasl_plain,sasl_scram,plugins,zstd,sasl_oauthbearer)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant