Number of commits and offset in each partition of a kafka topic -

how find number of commits , current offset in each partition of known kafka topic. using kafka v0.8.1.1

it not clear question, kind of offset you're interested in. there 3 types of offsets:

1. the offset of first available message in topic's partition. use -2 (earliest) --time parameter getoffsetshell tool
2. the offset of last available message in topic's partition. use -1(latest) --time parameter.
3. the last read/processed message offset maintained kafka consumer. high level consumer stores information in zookeeper (separately every consumer group) , takes care keeping date when call commit() or when auto-commit setting set true. simple consumer, code have take care managing offsets.

in addition command line utility, offset information #1 , #2 available via simpleconsumer.earliestorlatestoffset().

if number of messages not large, can specify large --offsets parameter getoffsetshell , count number of lines returned tool. otherwise, can write simple loop in scala/java iterate available offsets starting earliest.

from kafka documentation:

get offset shell offsets topic bin/kafka-run-class.sh kafka.tools.getoffsetshell  required argument [broker-list], [topic] option description  ------ -----------  --broker-list <hostname:port,..., required: list of hostname , hostname:port> port of server connect to.  --max-wait-ms <integer: ms> max amount of time each fetch request waits. (default: 1000)  --offsets <integer: count> number of offsets returned (default: 1) --partitions <partition ids> comma separated list of partition ids. if not specified, find offsets partitions (default)  --time <long: timestamp in milliseconds / -1(latest) / -2 (earliest) timestamp; offsets come before timestamp, in getoffsetsbefore  >  --topic <topic> required: topic offsets from.