Give each event a business meaning

An event is a claim that something happened. Before choosing a broker, define the entity, operation, identity and version. A payment-created event and a payment-status snapshot are not interchangeable. One describes a transition; the other describes current state. Replaying transitions can rebuild history, while replacing snapshots can recover a latest view only if ordering is reliable.

Distinguish event identity from entity identity. A customer can produce many valid events, so deduplicating by customer ID alone destroys history. A retry of the same event should keep its event ID. A new correction should receive its own identity or version according to the producer contract.

Arrival order is not necessarily business order. Different producers, retries and network delays can reorder messages. A broker partition can preserve its own log order, but that does not automatically represent event time across partitions. If the business requires per-entity ordering, make the partition key and version rule explicit.

Use a deterministic conflict policy. A timestamp alone may tie or reflect a skewed clock. A source sequence or monotonic entity version can be stronger, provided the producer actually guarantees it. Preserve rejected or conflicting records with a reason so the pipeline can be audited.

This is where data engineering differs from writing a query over a static table. The meaning of the final rows depends on how updates, duplicates and corrections arrive. In an interview, demonstrate the rule on a short trace. If the result changes when the same input is replayed, explain whether that is intentional. Most current-state views should converge under repeated delivery.

A useful event envelope makes the producer's promises explicit. It separates business identity from transport metadata and preserves enough information to explain a rejection later. Do not use a broker offset as a universal event ID if the same business event can be republished into a different topic or partition. The offset identifies a position in that log, not necessarily the original business operation.

1{2  "event_id": "payment-p8-v3",3  "entity_id": "p8",4  "entity_version": 3,5  "operation": "status_changed",6  "status": "settled",7  "occurred_at": "2026-09-20T12:00:00Z",8  "schema_version": 29}

The values are invented. This envelope still requires a contract: who assigns the version, whether it is strictly increasing per entity, whether gaps are allowed, and whether the payload is a complete state or a delta. A latest-state consumer can sometimes apply version 3 without seeing version 2 if version 3 contains a complete snapshot. A transition processor may need the missing event because the intermediate operation has a business effect. One rule cannot safely cover both without knowing the event meaning.

Suppose a payment stream contains “add ten” as a delta. Taking only the highest version loses previous additions. Suppose instead each event contains the complete current balance with an authoritative entity version. Applying only the highest valid version can converge to the latest balance. This is why “upsert everything” is not a complete design. The operation semantics determine whether earlier records can be ignored, replayed or reconciled.

The last row is not automatically a harmless duplicate. Two producers may both believe they own the same version. If the contract promises one authoritative version, the conflict is evidence of a producer defect. If multiple sources are legitimate, define precedence or a merge rule using source identity and business semantics. A timestamp tie-breaker can make output deterministic while still choosing the wrong business state.

Deletes also need ordering. A deletion at version 8 should not be undone when an old version 6 snapshot arrives later. Keep enough version or tombstone information to reject stale updates. Physically deleting the current row and forgetting the last version can permit resurrection. Retention and privacy requirements can constrain what metadata remains, so specify an allowed minimal ordering record rather than retaining full deleted payloads indefinitely.

Clock timestamps have narrower guarantees than source versions. A device clock can be wrong; two events can share a timestamp; a backfill can preserve an old occurred-at value while arriving today. If no authoritative sequence exists, state the uncertainty. A deterministic order using timestamp plus event ID can make reruns stable, but it does not prove the chosen order is the true business order. Some conflicts require reconciliation with the source.

Test convergence by permuting the trace and repeating records. For a complete-snapshot latest-state view with valid monotonic versions, each permutation should end at the same highest version, including the same deletion state. For a transition ledger, compare the set of distinct accepted events and the resulting business effect under its own ordering requirements. Add conflicting duplicates deliberately and verify that the pipeline records them rather than silently hiding them.

In an interview, show this trace before discussing throughput. Once correctness is explicit, partitioning can preserve the required per-entity locality and the consumer can use a version-aware conditional update. Without that contract, a fast pipeline can produce a different answer every time the network changes arrival order.

Teaching events:

A latest-state view uses the highest valid version, so order o7 ends as shipped at version three. The repeated e13 does not add a second shipment. Keep the distinct event history e11, e12, e13 if audit is required. Sorting only by arrival would incorrectly leave the view at paid before the duplicate arrives.

Process versions 4, 2, 4 and 5 for account a 9. The two version-four events have the same event ID and payload. State the latest version and the number of distinct events retained for history.

Latest version is five. Retain three distinct events, for versions four, two and five. The repeated version four is a duplicate. If two different payloads share the same event ID, do not silently choose one; treat that as a producer-contract violation and quarantine or alert under policy. Test the same input in a different arrival order and confirm the latest-state result remains version five.

Score out of four: one point for the correct result, one for showing the intermediate reasoning, one for identifying the stated failure case, and one for a verification that could disprove the answer. Do not award the reasoning point for a tool name alone.

“The highest version always replaces all earlier work.” That is suitable for a complete-snapshot latest-state contract, not necessarily for deltas or required transitions.

“A deterministic timestamp tie-breaker proves business order.” It proves repeatability of the chosen rule. Clock accuracy and source ownership still determine whether the rule is correct.

Evidence class: recommended. Original practice.

Why is 'take the last row we received' unsafe for a change stream?

Strong answer: Retries and independent producers can reorder arrivals. I would use the source's documented entity version or ordering position, retain event identity for deduplication, and define conflict handling for ties or inconsistent payloads.

Follow-up: What if the source provides timestamps but no sequence number?

Weak answer indicators: Deduplicating by entity ID; assuming total broker ordering; silently resolving conflicting duplicate IDs.

Technical references: Kafka 4.1 design; Debezium PostgreSQL connector. Sources support the documented mechanisms. The numbers, decisions, rubrics and interview prompts in this lesson are original teaching examples, not measurements or employer question claims.