[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"post:real-time-streaming-pipelines-batch-to-event-driven":3},{"id":4,"slug":5,"title":6,"excerpt":7,"body":8,"tags":9,"read_minutes":14,"status":15,"published_at":16,"updated_at":17},"23742789-f030-4a96-843b-23006c78bf25","real-time-streaming-pipelines-batch-to-event-driven","Building and Scaling Real-Time Streaming Pipelines","Moving from nightly batches to event-driven architecture is a mindset shift, not just a tooling swap. Kafka vs. Redpanda, late-arriving data, and where Flink earns its complexity.","\u003Cp>The nightly batch job served us well for decades, but the business has stopped waiting for it. Fraud teams want to block a transaction while it is happening, not flag it tomorrow; personalization is worthless if it reacts to yesterday's session; operations dashboards that lag by hours are just history lessons.\u003C\u002Fp>\u003Cp>Moving from batch to streaming is less a tooling swap than a mindset shift: data becomes an unbounded log of events, and 'the pipeline ran' becomes 'the pipeline is running, forever'.\u003C\u002Fp>\n\u003Ch2>Choosing the log: Kafka vs. Redpanda\u003C\u002Fh2>\u003Cp>Apache Kafka remains the default for good reasons — a vast ecosystem, battle-tested connectors, and an operational playbook every SRE has read. Its cost is operational surface: brokers, partitions, and (historically) ZooKeeper, though KRaft has simplified that story.\u003C\u002Fp>\u003Cp>Redpanda reimplements the Kafka protocol in C++ as a single binary — no JVM, no separate coordination layer — and delivers noticeably lower tail latencies on the same hardware. For lean teams the operational simplicity is the real selling point; for large enterprises, Kafka's ecosystem gravity usually still wins. Since Redpanda speaks the Kafka API, you can defer the decision: write your producers and consumers once and benchmark both.\u003C\u002Fp>\n\u003Ch2>Late data is the rule, not the exception\u003C\u002Fh2>\u003Cp>The hardest streaming lesson: events arrive out of order, and some arrive very late. A mobile purchase made in a tunnel may reach you minutes after the aggregation window it belongs to has closed. Streaming frameworks answer this with event-time processing and watermarks — a moving assertion that 'we have probably seen everything up to time T'.\u003C\u002Fp>\u003Cp>The trade-off is yours to make explicit: a tight watermark gives fast results that occasionally miss stragglers; a generous one gives complete results, later. For revenue metrics, add allowed lateness and emit corrections; for an ops dashboard, drop the stragglers and move on.\u003C\u002Fp>\n\u003Ch2>Stateful transformations with Flink\u003C\u002Fh2>\u003Cp>Simple filters and enrichments can live in lightweight consumers. The moment you need joins across streams, sessionization, or aggregates over millions of keys, you need managed state — and this is where Apache Flink earns its complexity. Flink keeps per-key state in embedded RocksDB, checkpoints it consistently, and restores it on failure, giving you exactly-once results without hand-rolled recovery logic.\u003C\u002Fp>\u003Cp>Start with one high-value, low-complexity pipeline — a fraud signal, a live order counter — and get its operational story solid: checkpoint monitoring, consumer lag alerts, replay procedure. Scale out from a working system, not from an architecture diagram.\u003C\u002Fp>",[10,11,12,13],"Streaming","Kafka","Flink","Architecture",8,"published","2026-07-05T00:00:00Z","2026-08-27T16:17:41.412599Z"]