Message Broker Vocabulary

Topics and partitions are a two-level hierarchy in Kafka's architecture.

Topic:
• A logical category / channel that groups related events
• You produce to a topic and consume from a topic
• Like a "table" in a database — defines the schema/purpose of the events
• Example: order-placed, payment-processed, user-registered

Partition:
• A physical ordered sequence (log file on disk) that is a subdivision of a topic
• Enables parallelism — multiple consumers can read from the same topic simultaneously (each reads its own partition subset)
• Ordering guarantee is only within a partition, not across partitions
• Events with the same partition key (e.g., userId) always go to the same partition — ensuring order for a given entity

Example: order-events topic with 3 partitions:
• Partition 0: orders from users A-G (determined by key hash)
• Partition 1: orders from users H-N
• Partition 2: orders from users O-Z
• 3 consumer instances can read in parallel

Key vocabulary:
• Topic — logical named channel for a category of events
• Partition — physical, ordered, immutable log segment within a topic
• Partition key — determines which partition an event goes to (ensures ordering per entity)
• Offset — the position of an event within a partition (sequential integer)

Consumer groups are Kafka's mechanism for both load balancing and fan-out — understanding this distinction is essential.

Consumer group (load balancing / competing consumers):
• Multiple consumers share work: partitions are assigned one-to-one to consumers in the group
• Each message is processed by exactly one consumer in the group
• Scaling: adding more consumers to a group increases throughput (up to the number of partitions)
• Use case: scaling the processing of order events across multiple order-processor instances

Multiple consumer groups (fan-out / broadcast):
• Different consumer groups each receive ALL events independently
• Each group maintains its own offset — reads don't interfere with each other
• Use case: the same order-placed event goes to the billing group, inventory group, and analytics group — each processes 100% of events

Topic: order-placed (3 partitions)
├── Consumer Group: billing-service        → all 3 partitions (for load balance)
├── Consumer Group: inventory-service      → all 3 partitions
└── Consumer Group: analytics-pipeline    → all 3 partitions

Key vocabulary:
• Consumer group — a set of consumers sharing partition-level load balancing
• Group ID — the identifier for a consumer group (billing-service)
• Consumer rebalance — redistribution of partitions when consumers join or leave a group
• Fan-out — one topic → multiple consumer groups → all receive 100% of events

DLQs are essential for fault isolation — without them, one bad message can halt all processing.

The problem a DLQ solves — "poison messages":
① A message arrives that cannot be processed (malformed JSON, schema mismatch, unexpected null)
② The consumer throws an exception and requeues the message for retry
③ The same message is retried indefinitely, blocking all subsequent messages in the queue
④ Without a DLQ: entire processing pipeline stalls on one bad message

With a DLQ:
① Process fails N times (configurable: typically 3–5 attempts)
② Message is automatically moved to the Dead Letter Topic/Queue
③ Normal processing continues for all subsequent valid messages
④ Engineering team is alerted about the DLQ message
⑤ Team inspects the message, fixes the root cause, and optionally replays from DLQ

What a DLQ message should include:
• The original message payload
• Original topic, partition, and offset
• Timestamp of first failure and last failure
• Exception message and stack trace
• Retry count

Key vocabulary:
• Dead-letter queue (DLQ) — destination for messages that failed after N retries
• Poison message — a message that consistently fails processing and can block a queue
• DLQ replay — manually reprocessing messages from the DLQ after fixing the root cause
• Max delivery attempts — the configurable threshold before a message goes to the DLQ

Choosing the right delivery semantic requires understanding both the system's capabilities and the consumer's tolerance for duplicates.

Semantic	Messages lost?	Duplicates?	Complexity	Use when
At-most-once	Possible	No	Low	Logs, metrics — loss is acceptable
At-least-once	No	Possible	Medium	Most business events — use idempotent consumers
Exactly-once	No	No	High	Financial transactions where every operation must be unique

Why at-least-once + idempotent consumers is the most common pattern:
• Truly exactly-once is hard and expensive (requires distributed transactions)
• Most business operations are naturally idempotent or can be made so:
- Setting a value to X is idempotent (duplicate = same result)
- Adding to a total is NOT idempotent (duplicate = double-counted)
- Use a deduplication ID in the message and check before processing

Key vocabulary:
• Idempotent consumer — a consumer that handles duplicate messages safely (same result regardless of how many times applied)
• Deduplication ID — a unique message identifier used to detect and skip duplicates
• Transactional producer — Kafka producer configured for exactly-once semantics
• Offset commit — recording which messages have been processed (controls at-least-once vs at-most-once)

The log retention model is the most fundamental architectural difference between a queue and a broker like Kafka.

Queue model (SQS, RabbitMQ):
• Message is stored until one consumer reads and acknowledges it
• After acknowledgement: message is deleted
• Second consumer group wants the same message? Too late — it's gone
• Re-processing historical events? Impossible without a separate archive

Log model (Kafka):
• Messages are written to an immutable, append-only log on disk
• Consumers track their own offset (position in the log)
• Messages are retained for a configurable time or size (e.g., 7 days, 100 GB)
• Multiple consumer groups can read the same log independently at their own pace
• Event replay is possible: reset offset to 0 and reprocess from the beginning

Implications for system design:
• Kafka enables new consumer groups to be added later and catch up on all historical events
• Enables event sourcing-style patterns (the log IS the source of truth)
• Enables temporal decoupling — a downstream service can fall behind during an incident and catch up
• Log compaction: Kafka can keep only the last event per key (like a materialised view of the latest state)

Key vocabulary:
• Retention period — how long Kafka keeps messages (time-based or size-based)
• Log compaction — Kafka feature that retains only the latest event per message key
• Offset reset — moving a consumer group's read position to replay events
• Consumer lag — how many messages a consumer group is behind the latest offset