Kafka fundamentals: topics, partitions, and consumer groups

A traditional queue deletes a message once it's been consumed. Kafka does the opposite: every message published to a topic is appended to a durable, ordered log and retained for a configured period (hours, days, or indefinitely) regardless of whether anything has consumed it yet. Consuming a message in Kafka doesn't remove it — it just advances a pointer. This single difference is why Kafka supports replaying history (a new consumer can start from the beginning of the log) in a way a traditional queue fundamentally can't.

A topic isn't one log — it's split into partitions, each an independent, strictly ordered append-only log:

Each message is written to exactly one partition, chosen by a partition key (often the order ID, or user ID) — every message with the same key always lands in the same partition, in the order it was produced. This is the mechanism behind Kafka's one real ordering guarantee: ordering is preserved within a partition, never across partitions. Spreading a topic across partitions is what lets Kafka scale writes and reads horizontally — different partitions can be written to and read from in parallel, on different machines.

A consumer group is a named set of consumers cooperating to read one topic. Kafka assigns each partition to exactly one consumer within a group — so a topic with 3 partitions can have up to 3 consumers in a group actively reading in parallel, each owning a disjoint subset of partitions. A 4th consumer in that same group would sit idle, since there's no fourth partition left to assign.

Critically, a second, independently-named consumer group reading the same topic gets its own full copy of every partition assignment — this is the queue-vs-topic blur from the previous lesson made concrete: across groups, Kafka behaves like pub/sub (every group sees everything); within a group, it behaves like a queue (each partition's messages go to exactly one member).

Kafka doesn't push messages to consumers — consumers pull, tracking an offset (a position in a partition's log) for their own group. Kafka stores each group's committed offset, so a consumer that crashes and restarts resumes from its last committed position instead of re-reading the entire log or losing its place. Committing an offset too early (before processing actually finishes) risks losing a message if the consumer crashes mid-processing; committing too late (or not using Kafka's auto-commit carelessly) risks reprocessing a message that already succeeded — the same at-least-once-by-default reality System Design's own queue lesson already established, just with a different mechanism producing it.

Increasing a topic's partition count later is possible but disruptive — it changes which partition a given key hashes to, breaking the "same key always lands in the same partition" guarantee for existing keys, which can silently reorder a key's message history from a consumer's point of view. Real systems generally pick a partition count based on the maximum expected consumer parallelism upfront, erring generously, rather than planning to resize it casually later.