Real-time & Messaging

Real-time systems push changes to users the moment they happen instead of waiting for the next request: a chat message, a driver’s position on a map, a collaborator’s cursor in a shared document. Messaging is the backbone underneath, the queues and event logs that let one part of a system hand work to another reliably without both being available at the same instant.

Core ideas and trade-offs

• Pull, push or callback. Polling is simple but wastes requests and adds, on average, half the polling interval of delay. Server-Sent Events hold one connection open for server-to-client updates; WebSockets add full two-way messaging; webhooks reverse the roles, so the provider calls your endpoint when something happens.
• Persistent connections are state. Each WebSocket lives on one server, so you need a registry of who is connected where, a pub/sub layer to route messages between servers, and a reconnection strategy for mobile clients that drop constantly.
• Fan-out has two shapes. Fan-out on write copies each new post into every follower’s precomputed feed, which makes reads fast and gets expensive for accounts with millions of followers. Fan-out on read assembles the feed when it is opened. Many large feeds use a hybrid.
• A queue forgets; a log remembers. RabbitMQ-style brokers route each message and delete it once it is acknowledged. Kafka-style logs retain messages, so each consumer tracks its own position and can replay history.
• Delivery is at least once. Retries create duplicates, so consumers must be idempotent, and ordering usually holds only within a partition or key, not across a whole topic.

How the formats fit together

The handbooks cover the transport choices, WebSockets and the poll-versus-push math, Kafka as a distributed log, and the comparisons that trip teams up: Kafka versus RabbitMQ, Kafka versus Kinesis and streaming versus batch. The system designs are the heart of this hub, because chat apps, notification systems, news feeds, collaborative editors, live streaming and ride matching each stress a different part of the problem. The algorithm games and challenges cover the streaming structures that summarize endless data in small memory, such as reservoir sampling, Count-Min sketches and HyperLogLog, and the partition calculator sizes a Kafka topic from your throughput.

Common mistakes

• Choosing WebSockets when one-way updates over Server-Sent Events, or plain polling, would do.
• Assuming messages arrive exactly once and in a single global order.
• Using Kafka as a task queue that needs per-message routing and delays, or RabbitMQ as a replayable event store.
• Forgetting the offline user: messages need durable storage and delivery on reconnect, not just a live socket.