Kafka vs Kinesis: portable log, or zero-ops AWS stream?

Kafka is open-source software you run yourself (or via a managed offering like Confluent or Amazon MSK) — a distributed, partitioned, replicated log, portable across any cloud or on-prem. Kinesis is AWS's fully-managed streaming service, built on the same append-only log idea but organized into shards instead of partitions, and it exists only inside AWS.

Unlike RAG-and-fine-tuning or MCP-and-A2A, you don't typically run both in the same pipeline — pick one streaming backbone. The existence of Amazon MSK (managed Kafka, running on AWS) is itself telling: it's AWS acknowledging that some teams want Kafka's specific model and ecosystem even while staying on AWS, rather than Kinesis's shard model, which shows this is a genuine architectural trade-off and not just "AWS vs not-AWS."

Both systems implement the same core idea: an append-only, partitioned log where consumers track their own read position (offset in Kafka, sequence number in Kinesis) and can replay by resetting it. That means the semantic guarantees — ordered delivery within a partition/shard, at-least-once consumption, replay from any retained point — are essentially the same on both. The difference that actually bites in practice is the throughput ceiling per unit: a Kafka partition's throughput is tunable and can be pushed very high by adjusting broker resources, while a Kinesis shard has a hard, fixed limit of 1MB/s in and 2MB/s out, full stop, regardless of what you're willing to pay for bigger hardware.

This matters specifically for "hot key" streams — if one partition key (say, one very active user or one very busy IoT device) needs to sustain more than 1MB/s, Kinesis forces you to solve it with application-level key-splitting tricks, because there's no "just make this shard bigger" option the way there effectively is with a Kafka partition on beefier hardware. It's a small-sounding limit that becomes a real architecture constraint at the tail of a workload's traffic distribution.

An analytics team already runs Lambda consumers, streams to S3 via Firehose, and visualizes in QuickSight — all AWS-native. Kinesis fits this pipeline directly: no brokers to run, and the integration points (Lambda triggers, Firehose delivery) are first-class, managed features rather than something to wire up manually. The zero-ops story is a genuine win when the rest of the stack is already this AWS-native.

The same pipeline, if it also needs to feed an on-prem fraud-detection cluster today, or needs to stay portable for a possible multi-cloud future, points to Kafka instead — the portability isn't a nice-to-have here, it's the actual requirement Kinesis structurally can't satisfy, since it doesn't run outside AWS at all. The right call depends entirely on whether "stay inside AWS" is a real constraint or an assumption worth questioning before committing.

Kinesis bills per shard (fully managed, no ops); self-managed Kafka is a fixed cluster cost plus your engineers’ time. If a shard costs P per hour and a Kafka cluster costs M per hour, Kinesis is cheaper below M / P shards, and Kafka’s fixed cost amortizes above it.