Apache Kafka & Real-Time Streaming

Why Kafka for Real-Time Architecture

Apache Kafka has become the default event backbone for real-time architectures. Not because it's the only option — Pulsar, Redpanda, and managed Kinesis all exist — but because Kafka's combination of throughput, durability, ordering guarantees, and ecosystem (Kafka Streams, Connect, Schema Registry) is hard to beat at production scale.

We help engineering teams design and run Kafka platforms that hold up under real production load — not Kafka demos, not "we have Kafka" PowerPoints, but Kafka clusters that are part of the team's daily operations.

Cluster Topology and Sizing

The first set of decisions in a Kafka deployment is topology. Wrong choices here haunt the cluster for years.

• Broker count: start with 3 for production fault tolerance. Scale up to 5 or 6 once partition counts grow past ~3000 per broker. Going below 3 sacrifices availability for cost and is rarely worth it.
• Disk choice: Kafka is disk-bound for many workloads. SSDs (gp3 or io2 on AWS, similar on GCP/Azure) outperform spinning disks dramatically for tail latency. Plan for 3–5x your steady-state retention to absorb traffic spikes and re-replication.
• Replication factor: 3 for production topics. RF=2 is acceptable only for non-critical data and should be a deliberate tradeoff.
• Partition strategy: more partitions = more parallelism but also more cluster overhead. We typically size partitions for expected consumer parallelism with a 2x headroom, not for "future scale we might never need."

We have inherited Kafka clusters where the original sizing was off and the cost of re-partitioning a high-volume topic was measured in weeks. Get this right at the start.

Exactly-Once Semantics in Practice

Kafka supports exactly-once semantics within a Kafka cluster: a producer can write to multiple topics atomically, and a consumer can process and produce in a single transaction. This is genuinely valuable for streaming pipelines that move data within Kafka.

The caveat: exactly-once within Kafka does not extend to external systems by default. A consumer that writes to PostgreSQL is responsible for its own idempotency. Patterns we use:

• Transactional outbox for service-to-Kafka writes — write the event to a database table in the same transaction as the business update; a Kafka Connect Debezium source replicates the table to a topic. Eliminates the dual-write problem.
• Idempotent consumers for Kafka-to-database writes — every message includes a deterministic key; the consumer upserts based on it.
• Exactly-once within Kafka Streams for stream-to-stream transformations — turn on processing.guarantee=exactly_once_v2 and the runtime handles transactional commits across input topics, state stores, and output topics.

End-to-end exactly-once across all systems requires architectural choices, not just configuration.

Schema Registry and Data Contracts

A Kafka cluster without a schema registry becomes a graveyard of "what does this field mean" investigations within a year. The registry is non-negotiable for any team beyond the smallest.

We standardize on:

• Avro for most production topics — strong typing, compact wire format, mature tooling.
• Protobuf where binary efficiency or polyglot tooling matters more.
• JSON Schema only for low-volume topics where human readability wins over efficiency.

Beyond format choice, the registry should enforce forward and backward compatibility by default. Breaking changes are gated by explicit overrides. This is the difference between a Kafka cluster you can evolve and one that becomes legacy within two years.

Stream Processing: Kafka Streams and Flink

Kafka topics by themselves carry events; stream processors transform them.

• Kafka Streams is the JVM-native option. It runs in your application process — no separate cluster — and integrates cleanly with Kafka's exactly-once semantics. Best for teams already on the JVM and processing topology that fits the application.
• Apache Flink is a separate cluster but offers richer windowing, more sophisticated state management, and language flexibility (Java, SQL, Python via PyFlink). Best for complex event time processing, larger stateful workloads, and unified batch + stream pipelines.
• ksqlDB runs SQL over Kafka streams — fastest path to "transform topic A into topic B with a SQL query." Good for analyst-driven workloads; not the right tool for complex stateful processing.

The choice depends on team and workload, not Kafka itself.

CQRS and Event Sourcing Patterns

Kafka is the natural backbone for CQRS (Command Query Responsibility Segregation) and event-sourced architectures. The patterns are powerful but worth using deliberately.

• Event sourcing stores state as the sequence of events that produced it. Replay derives current state. Worth it when audit, time-travel queries, or replayable derived views are valuable; overkill for most CRUD applications.
• CQRS separates the write path (commands → events) from the read path (events → projections). Worth it when read and write workloads have very different scaling needs or shape.
• Event-driven microservices without full event sourcing — services publish domain events to Kafka, others consume. Lighter pattern, broadly applicable.

We help teams pick the right level of these patterns for their actual problem, not the level the architecture diagram looks coolest at.

Operations: Monitoring, Security, Cost

Production Kafka requires real operations:

• Monitoring: under-replicated partitions, consumer lag per group, request latency p99, broker disk usage, controller election rate. Prometheus + Grafana is the open-source default; managed options (Confluent Cloud, AWS MSK with CloudWatch) bundle this.
• Security: TLS for all client and inter-broker communication. SASL or mTLS for authentication. ACLs to enforce who can produce or consume which topics — not optional in multi-team environments.
• Cost optimization: tiered storage moves cold partitions to object storage at a fraction of broker SSD cost. Compression (zstd, lz4) at the producer cuts both network and storage. Right-sized retention beats over-provisioned brokers.

These are the daily operations of Kafka teams. We hand off runbooks and dashboards alongside the cluster.

How We Build Kafka Architectures

For most engagements, we typically engage in three modes:

• Greenfield Kafka platform — design and stand up the cluster, schema registry, monitoring, and security from zero. Usually 6–10 weeks, ending with the client team owning operations.
• Kafka platform rescue — inherit an underperforming or fragile cluster, diagnose the actual issues (typically partition strategy, schema discipline, or consumer design), and remediate. Usually 4–8 weeks.
• Streaming pipeline build-out — Kafka cluster exists; specific high-value pipelines (real-time analytics, change-data-capture, ML feature streaming) need design and implementation. Usually 4–6 weeks per pipeline.

We do not parachute in to write code and leave. Every engagement ends with the client team's engineers operating what we built.

Summary: Production Kafka Implementation Stack

1. Get cluster topology right at the start. RF=3, 3+ brokers, SSDs sized for headroom. Repartitioning later is painful.
2. Mandate a schema registry. Avro by default, Protobuf for binary efficiency, compatibility checks enforced.
3. Use exactly-once where it earns its complexity. Within Kafka it's real; end-to-end exactly-once needs architectural patterns.
4. Pick Kafka Streams or Flink based on team and workload, not novelty.
5. Use CQRS and event sourcing only where their cost is justified. Don't over-architect a CRUD system.
6. Build monitoring, security, and cost controls before launch. Adding them after first incident is harder than doing it upfront.

Kafka rewards teams that take it seriously and punishes teams that treat it as a write-only message broker. The investment compounds across every downstream system that consumes from it.