3

While using Apache Kafka, is it possible to get the offset of the message that has been produced?

From the Kafka documentation page it says:

each message in the partition are assigned a unique sequential id/number called Offset

Now suppose when I am producing any message I want to retrieve the offset that has been assigned to it.

Spoike
  • 14,765
  • 4
  • 43
  • 58
user2720864
  • 255
  • 1
  • 3
  • 9

3 Answers3

4

The producer response protocol provides the offset of the first message sent in the messageset. Provided the API/Implementation you're using exposes this information, you should be able to retrieve it from the response to producing your message. More here: https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-ProduceResponse

David Corley
  • 141
  • 2
3

You shouldn't do that from a class that acts as a producer, like in most queue systems, as its responsibility is to just fire and forget the messages. The broker will do the appropriate meta data handling with id's, offsets, whatever it needs to handle the messages in a correct manner.

You can get the offsets as a consumer of messages from a Kafka broker. If you look in the SimpleConsumer class, you'll notice it fetches MultiFetchResponse objects that contains offsets as a list. Furthermore when you iterate the Kafka messages you'll end up with a MessageAndOffset objects that contains both the message sent and it's offset.

Spoike
  • 14,765
  • 4
  • 43
  • 58
  • 1
    Agreed, but why I am asking this because I want to keep a track of the message offset that has been produced. I then might able to select that offset to read from in the consumer end. Does this make sense? – user2720864 Sep 06 '13 at 12:06
  • 1
    @user2720864 why do you want to select just a particular message at the producer using offsets? you can select messages with using topics instead. – Spoike Sep 06 '13 at 12:19
  • Not a particular message but I want to consume starting from that offset – user2720864 Sep 06 '13 at 12:31
  • 2
    You don't need to use an offset for this. You can do this by just starting the consumer before sending messages. – Spoike Sep 06 '13 at 13:20
0

The javadoc states you can setup a callback for the send method that will be invoked once the send has been acknowledged. The send method returns a RecordMetadata that contains the offset the record was assigned to as well as other metadata. I know this is an old question but I came across it in some Kafka research I was doing. Hope this helps!

Terri Ellis
  • 1