Consumer Incremental Cooperative Rebalance

References

Flashcards

Which events will initiate the consumer rebalancing? (Pick three)

• Consumers joins the consumer group
• A new producer starts writing messages to the topic in question
• A consumer leaves the group
• A new broker is added to the cluster
• Partitions are added to the topic
  ?
• Consumers joins the consumer group
• A consumer leaves the group
• Partitions are added to the topic

Consumer Group Auto Offset Commit Behavior

References

Producer Retries

References

Idempotent Producer

References

Safe Producers

References

Producer Batch Settings

linger.ms and batch.size

References

Producer Partitioners

Default Partition and Sticky Partitioner

References

Kafka Extended APIs for Developers

Consumers and Producers are considered low level, and since their introduction other higher level APIs have been created, specifically:

• Kafka Connect

• Kafka Streams

• Schema Registry

• Kafka Brokers do NOT verify the messages they receive

• Kafka takes bytes as input with out loading them into memory (zero copy)

• Schema Registry then is a separate component

  • Producers and Consumers need to be able to talk to it
  • The Schema Registry must be able to reject bad data before sent to Kafka
  • Supports Avro, Protobuf and JSON

Gotchas:

• Needs to be highly available
• Schema formats have a learning curve (Avro, etc.)

You can also evolve schemas over time. The same problems you experience in changing any contract apply. Additions are backwards compatible, modifications or deletions are not.

Schema changes are versioned (v1, v2, etc.)

You can also assign default values for new properties as the schemas evolve.

• Known data source use kafka connect source or sink depending on data flow
• Use Producer to collect directly from the source
• Consumer for producing ephemeral messages
• Kafka Streams for ETL back into Kafka
• KSQL for queries on Kafka data

References

Flashcards