Delivery Semantics

There are different ways Consumers will handle delivery.

At Least Once Delivery

This is a Delivery Semantics in which:

Offsets are committed after the Message is processed
If the processing goes wrong, the Message will be read again
This can result in duplicate processing of messages and the Consumer needs to be able to reprocess those messages in an Idempotent manner

At Most Once Delivery

This is a Delivery Semantics in which:

Offsets are committed as soon as messages are received.
If the processing goes wrong, some messages will be lost and not read again

Exactly One Delivery

This is a Delivery Semantics in which:

For Kafka to Kafka workflows: Use the Transactional API
Kafka to External System us an Idempotent Consumer

References

Flashcards

What are the 3 different ways Consumers handle message delivery?:: Atleast once, at most once, exactly once

while (true) {
        ConsumerRecords<String, String> records = consumer.poll(100);
         try {
          consumer.commitSync();
        } catch (CommitFailedException e) {
            log.error("commit failed", e)
        }
        for (ConsumerRecord<String, String> record : records)
        {
            System.out.printf("topic = %s, partition = %s, offset =
              %d, customer = %s, country = %s
",
                 record.topic(), record.partition(),
                 record.offset(), record.key(), record.value());
        }
}

What kind of delivery guarantee this consumer offers?

At-most-once. Here offset is committed before processing the message. If consumer crashes before processing the message, message will be lost when it comes back up.