The nightly batch job served us well for decades, but the business has stopped waiting for it. Fraud teams want to block a transaction while it is happening, not flag it tomorrow; personalization is worthless if it reacts to yesterday's session; operations dashboards that lag by hours are just history lessons.
Moving from batch to streaming is less a tooling swap than a mindset shift: data becomes an unbounded log of events, and 'the pipeline ran' becomes 'the pipeline is running, forever'.
Choosing the log: Kafka vs. Redpanda
Apache Kafka remains the default for good reasons — a vast ecosystem, battle-tested connectors, and an operational playbook every SRE has read. Its cost is operational surface: brokers, partitions, and (historically) ZooKeeper, though KRaft has simplified that story.
Redpanda reimplements the Kafka protocol in C++ as a single binary — no JVM, no separate coordination layer — and delivers noticeably lower tail latencies on the same hardware. For lean teams the operational simplicity is the real selling point; for large enterprises, Kafka's ecosystem gravity usually still wins. Since Redpanda speaks the Kafka API, you can defer the decision: write your producers and consumers once and benchmark both.
Late data is the rule, not the exception
The hardest streaming lesson: events arrive out of order, and some arrive very late. A mobile purchase made in a tunnel may reach you minutes after the aggregation window it belongs to has closed. Streaming frameworks answer this with event-time processing and watermarks — a moving assertion that 'we have probably seen everything up to time T'.
The trade-off is yours to make explicit: a tight watermark gives fast results that occasionally miss stragglers; a generous one gives complete results, later. For revenue metrics, add allowed lateness and emit corrections; for an ops dashboard, drop the stragglers and move on.
Stateful transformations with Flink
Simple filters and enrichments can live in lightweight consumers. The moment you need joins across streams, sessionization, or aggregates over millions of keys, you need managed state — and this is where Apache Flink earns its complexity. Flink keeps per-key state in embedded RocksDB, checkpoints it consistently, and restores it on failure, giving you exactly-once results without hand-rolled recovery logic.
Start with one high-value, low-complexity pipeline — a fraud signal, a live order counter — and get its operational story solid: checkpoint monitoring, consumer lag alerts, replay procedure. Scale out from a working system, not from an architecture diagram.