Streaming Data Pipeline — LIVE Scenario

Failure Handling

Production failure cases, handled by design

A streaming pipeline is defined by what happens when things go wrong — a crashed consumer, a duplicated message, a stalled relay, a hostile payload, a contract that moves underneath you. Every case below is handled in the running system, and most were proven by firing the failure at it rather than assumed.

• Dual-write problem — state changes commit business rows and the event into the outbox in one SQLite transaction; the Redpanda Connect relay is the only thing that publishes them. The database and the stream cannot disagree.
• Relay crash mid-batch — the cursor advances only after Redpanda acknowledges the whole batch, so a crash replays it rather than skipping it. Proven live: killed at cursor 1,331, twelve orders placed while it was dead, exactly twelve published on restart.
• Service crash mid-order — consumers commit offsets only after the work finishes, via a per-partition ack watermark. A crashed kitchen or delivery service replays its unfinished orders on restart instead of skipping them.
• Duplicate events (at-least-once) — every layer is idempotent: services skip seen order ids, the dashboard dedupes by event id, dbt staging keeps exactly one row per event id.
• Out-of-order arrival — cross-topic ordering isn't guaranteed, so the dashboard folds events monotonically: a late event can never move an order's status backwards.
• Malformed / hostile payloads — broker messages are treated as untrusted input and validated on parse. Failures go to the dead-letter topic with the reason in headers, are audit-logged, and never crash a consumer — fail closed, per message.
• Schema drift — deliberately asymmetric, because the three directions carry different risk. A new field added by a newer producer is ignored; a missing optional field reads as unset; a changed schema version is rejected to the dead-letter queue rather than half-parsed. Firing all three at the live pipeline is what caught a real bug: the Python consumers enforced the version but the dashboard did not, so an incompatible event was quarantined by one and silently accepted by the other. All three boundaries now agree.
• Poison message in the data lake — bronze is newline-delimited JSON, so one unparseable line makes an entire file unreadable and can take down a whole warehouse build. The sink groups each batch by destination and routes unparseable payloads to a quarantine prefix; the warehouse then re-validates the envelope itself and routes rejects to a dedicated table, so "the facts are clean" never quietly means "some data vanished".
• Illegal state transitions — guarded SQL verified by rowcount: a closed order cannot be re-closed, an abandoned one cannot confirm. Validation on every state change, not just at intake.
• Money correctness — all money is integer paise, no floats. dbt tests assert profit = price − ingredient cost − 5% GST − driver fee in-warehouse, on every run.
• Lost customers — ~10% of placed orders abandon, and that's a first-class event tracked in the funnel and the owner's abandonment rate — never silently missing.
• Failures & retries visibility — consumer crashes, retries, and timeouts are recorded to the audit log and persisted append-only as daily JSONL. Silence is never mistaken for success.
• Bad configuration — settings fail closed: malformed or out-of-range values raise on startup instead of falling back silently. Secrets live in gitignored env files and are injected as environment variables, never embedded in the pipeline definitions.

Beyond the failure cases, the live homepage walks through the full vocabulary of streaming — topics and partitions, consumer groups, offsets and commits, delivery semantics, DLQs, change data capture, replay, retention, backpressure, event time vs processing time, compression on the wire, fan-out, stream–table duality, the medallion architecture — each one pointing at where it is happening in the running pipeline.