Streaming Data Pipeline — LIVE Scenario
A complete streaming data platform demonstrated through the simplest story everyone understands — a Mumbai cloud kitchen. Customer orders → kitchen preparation → food delivery, with every step an event flowing in real time through a production-shaped stack: MySQL with a transactional outbox, Debezium CDC, Apache Kafka, a MinIO bronze data lake, dbt + DuckDB marts, and live auto-refreshing dashboards. Python powers the pipeline, TypeScript the dashboard, SQL (dbt) the warehouse.
This is a live, running pipeline — watch orders stream through it in real time. The demo lives exactly one business day: data is wiped every midnight and the restaurant reopens fresh.
stream.steleios.com ↗Objective
What this project sets out to do
Make streaming architecture legible — and prove it behaves like production. Every core streaming concept — topics, partitions, consumer groups, offsets, delivery semantics, idempotency, dead-letter queues, CDC, replay, backpressure, event time vs processing time, schema evolution — is not explained on a slide but demonstrated live in a running system. And the system earns the word production-shaped: no dual writes, no lost sale, effectively exactly-once processing, chaos-tested against broker destruction and mid-flight kills, with 186 automated tests behind it.
What we're delivering
Feature Highlights
No dual writes — outbox + CDC
Every state change commits the business rows and the event into a transactional outbox in one MySQL transaction; Debezium tails the binlog and publishes to Kafka. If the transaction rolls back, neither exists — the database and the stream can never diverge.
Kafka as the system of record
Four topics, three partitions keyed by order id — per-order ordering with parallel scale-out — on infinite retention and a persistent volume. Idempotent producers (acks=all) plus ack-after-work offset commits give at-least-once delivery; idempotent processing at every layer makes it effectively exactly-once. No sale is ever lost to a crash.
Bronze lake to tested marts
Kafka Connect sinks every event as date-partitioned raw JSON into MinIO — the bronze layer nothing can delete — and dbt on DuckDB builds order facts, item profitability, and per-minute demand, with in-warehouse tests asserting the profit equation, the delivery physics, and the fulfillment rules on every run.
Live operations dashboards
A fully typed Bun + KafkaJS backend reads the same stream with its own consumer group and rebuilds complete state from offset zero on every boot; the Vue 3 frontend serves owner analytics, kitchen and delivery ops boards, per-order live customer tracking, and an append-only audit log — auto-refreshing every two seconds.
Chaos-tested resilience
The Kafka broker was deliberately destroyed mid-stream: all 53 outbox events rebuilt into the new broker via Debezium snapshot, revenue identical to the paisa. The nightly reset force-killed the pipeline mid-write with no corruption. Hostile payloads are dead-lettered with reasons — never dropped, never crashing a consumer.
186 automated tests + live drills
158 Python unit tests with passing and failing cases for every business rule, 28 TypeScript dashboard tests, dbt schema and singular tests in-warehouse, a full business day streamed end to end, API hardening that fails closed, and a security pass: no PII in events, parameterized SQL, least-privilege DB users.
The pipeline, live · every dot is an order event; the callouts note the complexity handled at each hop
Failure Handling
Production failure cases, handled by design
A streaming pipeline is defined by what happens when things go wrong — a crashed consumer, a duplicated message, a destroyed broker, a hostile payload. Every case below is handled in the running system, and most were proven live rather than assumed.
- Dual-write problem — state changes commit business rows and the event into the outbox in one MySQL transaction; Debezium publishes it from the binlog. The database and Kafka cannot disagree.
- Service crash mid-order — consumers commit offsets only after the work finishes, via a per-partition ack watermark. A crashed kitchen or delivery service replays its unfinished orders on restart instead of skipping them.
- Duplicate events (at-least-once) — every layer is idempotent: services skip seen order ids, the dashboard dedupes by event id, dbt staging keeps exactly one row per event id.
- Out-of-order arrival — cross-topic ordering isn't guaranteed, so the dashboard folds events monotonically: a late event can never move an order's status backwards.
- Malformed / hostile payloads — broker messages are treated as untrusted input and validated on parse. Failures go to the dead-letter topic with the reason in headers, are audit-logged, and never crash a consumer — fail closed, per message.
- Broker destroyed entirely — Kafka data sits on a persistent volume with infinite retention, and even a full wipe rebuilds the order stream from the MySQL outbox via Debezium snapshot. Proven live: 53/53 events recovered.
- Illegal state transitions — guarded SQL verified by rowcount: a closed order cannot be re-closed, an abandoned one cannot confirm. Validation on every state change, not just at intake.
- Money correctness — all money is integer paise, no floats. dbt tests assert profit = price − ingredient cost − 5% GST − driver fee in-warehouse, on every run.
- Schema evolution — the driver-fee field was added as an optional field mid-stream; every layer treats missing as "no fee", so pre-change events still parse everywhere.
- Lost customers — ~10% of placed orders abandon, and that's a first-class event tracked in the funnel and the owner's abandonment rate — never silently missing.
- Failures & retries visibility — consumer crashes, retries, and timeouts are recorded to the audit log and persisted append-only as daily JSONL. Silence is never mistaken for success.
- Bad configuration — settings fail closed: malformed or out-of-range values raise on startup instead of falling back silently. Secrets live in gitignored env files, injected via config providers.
Beyond the failure cases, the live homepage walks through the full vocabulary of streaming — topics and partitions, consumer groups, offsets and commits, delivery semantics, DLQs, CDC, replay, retention, backpressure, event time vs processing time, fan-out, stream–table duality, the medallion architecture — each one pointing at where it is happening in the running pipeline.