#456 · timed-out
Arrived 5,200.0 ms · read · database path
Database wait 67.0 ms.
Deadline expired at stage: database.
INVESTIGATION 02 · FOLLOW THE WAITING WORK
Traffic triples. Requests pile up. You double the application workers, and the timeouts barely move. Find the limit, test a repair, and defend what it costs.
A synthetic service model that runs in your browser. Every result is calculated from the workload, settings, and versioned assumptions below.
01 · THE CUSTOMER BRIEF
Harbor’s fictional support API normally receives 60 requests per second. A launch pushes that to 180 for six seconds. The application accepts work, processes it briefly, then either serves a cached read or waits for the database.
The baseline has twelve application worker slots and four database service slots. One database operation takes about 40 ms; each worker stays occupied while waiting for its operation. An additional application worker can join that wait without increasing database throughput.
Serve at least 95% of all offered requests within 300 ms, using at most 28 capacity points. Count rejected and timed-out requests against that target. The client gives up at 1000 ms, even when the server is still busy.
02 · DIAGNOSE, THEN CHANGE ONE THING
Inspect the baseline first. Predict which queue should shrink, change one control, and save a run. A fast response for the few users admitted can hide a poor experience for everyone else.
At baseline, 12 application slots wait on 4 database service slots. Compare 24 application slots with 8 database slots while keeping the arrival stream fixed. The first adds waiting capacity; the second changes modeled service capacity. Compare timely responses and capacity points.
Your independent task: Try a cache-based repair on the read-heavy workload, then test the write-heavy and cold-cache conditions. Explain which assumption fails.
Predict first. Compare one change, inspect an actual passage/request/event, then explain a condition where the repair fails. The changed-case check removes these hints.
Within the illustrative capacity budget. One point per worker, two per DB service slot, and three if caching is enabled.
Selected workload: 670/1800 → 670/1800 users timely; 20 → 20 capacity points. Rejected and timed-out users remain in the denominator.
Compared settings: No setting differences. Both columns use the same case set.
The baseline is already simulated. Save an experiment to add your comparison.
| Measure | Reference | Selected run |
|---|---|---|
| Useful responses ≤300 msThe denominator includes rejections and timeouts | 670/1800 37.2% | — |
| Completed responsesMay include responses slower than the objective | 932 | — |
| Client timeoutsOne-second deadline from arrival | 868 | — |
| Rejected immediatelyAdmission or full ingress queue | 0 | — |
| Successful-response p95Excludes timed-out and rejected requests | 878 ms | — |
| Peak ingress / DB waitWaiting requests, recorded at every event | 173 / 8 | — |
| Database busy timeAverage slot utilization over the arrival window | 79% | — |
| Queries finishing after deadlineWork consumed without a successful client response | 486 | — |
| Capacity pointsIllustrative budget ≤28 | 20 | — |
Times are simulated milliseconds, independent of your browser’s speed. Capacity points are a teaching budget. They are not cloud prices.
Choose a measured passage, workload, or event. The attachment keeps its run and version when newer experiments are saved. Each record holds up to six observations; existing attachments are never evicted automatically.
An average hides the burst. Compare the queues over time, then inspect which resource was occupied while callers waited.
ORIGINAL BASELINE · The launch burst
A support API rises from 60 to 180 requests/second for six seconds, then returns to 60. Eighty percent of requests are reads eligible for caching.
Find where requests wait. More application concurrency cannot create more database slots, and a queue stores unfinished work rather than creating throughput.
Event markers: load change at 4 s · Inspect an affected request ↓
These saved configurations use the same selected workload. A dominated row has no better timely count and no lower capacity cost than another row. This comparison covers only these two objectives.
| Run | Capacity points | Timely / offered | Dominated by |
|---|
The seed changes the specified read mix and bounded service jitter, not the arrival-rate phases. This is deterministic model sensitivity, not a production confidence interval.
Samples are spaced 250 ms apart. Peak counters use every event, so a short peak can occur between samples. The final samples include draining after arrivals stop.
| Time (s) | Ingress wait | DB wait | Workers occupied | DB slots busy | Completed | Timeouts | Rejected |
|---|---|---|---|---|---|---|---|
| 0.00 | 0 | 0 | 1 | 0 | 0 | 0 | 0 |
| 0.25 | 0 | 0 | 2 | 2 | 13 | 0 | 0 |
| 0.50 | 0 | 0 | 3 | 3 | 27 | 0 | 0 |
| 0.75 | 0 | 0 | 4 | 3 | 42 | 0 | 0 |
| 1.00 | 0 | 0 | 2 | 2 | 58 | 0 | 0 |
| 1.25 | 0 | 0 | 4 | 3 | 72 | 0 | 0 |
| 1.50 | 0 | 0 | 4 | 3 | 87 | 0 | 0 |
| 1.75 | 0 | 0 | 2 | 2 | 103 | 0 | 0 |
| 2.00 | 0 | 0 | 2 | 2 | 118 | 0 | 0 |
| 2.25 | 0 | 0 | 3 | 2 | 133 | 0 | 0 |
| 2.50 | 0 | 0 | 3 | 2 | 148 | 0 | 0 |
| 2.75 | 0 | 0 | 4 | 3 | 162 | 0 | 0 |
| 3.00 | 0 | 0 | 3 | 2 | 178 | 0 | 0 |
| 3.25 | 0 | 0 | 2 | 2 | 193 | 0 | 0 |
| 3.50 | 0 | 0 | 2 | 2 | 208 | 0 | 0 |
| 3.75 | 0 | 0 | 2 | 2 | 223 | 0 | 0 |
| 4.00 | 0 | 0 | 3 | 2 | 238 | 0 | 0 |
| 4.25 | 15 | 8 | 12 | 4 | 259 | 0 | 0 |
| 4.50 | 34 | 7 | 12 | 4 | 285 | 0 | 0 |
| 4.75 | 55 | 7 | 12 | 4 | 309 | 0 | 0 |
| 5.00 | 75 | 7 | 12 | 4 | 334 | 0 | 0 |
| 5.25 | 94 | 7 | 12 | 4 | 360 | 0 | 0 |
| 5.50 | 115 | 8 | 12 | 4 | 384 | 0 | 0 |
| 5.75 | 133 | 6 | 12 | 4 | 411 | 0 | 0 |
| 6.00 | 153 | 6 | 12 | 4 | 436 | 0 | 0 |
| 6.25 | 172 | 6 | 12 | 4 | 456 | 10 | 0 |
| 6.50 | 172 | 6 | 12 | 4 | 456 | 55 | 0 |
| 6.75 | 171 | 5 | 12 | 4 | 456 | 100 | 0 |
| 7.00 | 172 | 6 | 12 | 4 | 456 | 145 | 0 |
| 7.25 | 172 | 7 | 12 | 4 | 456 | 190 | 0 |
| 7.50 | 172 | 7 | 12 | 4 | 456 | 235 | 0 |
| 7.75 | 170 | 4 | 12 | 4 | 456 | 280 | 0 |
| 8.00 | 171 | 5 | 12 | 4 | 456 | 325 | 0 |
| 8.25 | 172 | 6 | 12 | 4 | 456 | 370 | 0 |
| 8.50 | 172 | 6 | 12 | 4 | 456 | 415 | 0 |
| 8.75 | 172 | 6 | 12 | 4 | 456 | 460 | 0 |
| 9.00 | 171 | 5 | 12 | 4 | 456 | 505 | 0 |
| 9.25 | 172 | 6 | 12 | 4 | 456 | 550 | 0 |
| 9.50 | 172 | 6 | 12 | 4 | 456 | 595 | 0 |
| 9.75 | 172 | 7 | 12 | 4 | 456 | 640 | 0 |
| 10.00 | 172 | 6 | 12 | 4 | 456 | 685 | 0 |
| 10.25 | 142 | 6 | 12 | 4 | 456 | 730 | 0 |
| 10.50 | 112 | 7 | 12 | 4 | 456 | 775 | 0 |
| 10.75 | 82 | 6 | 12 | 4 | 456 | 820 | 0 |
| 11.00 | 52 | 6 | 12 | 4 | 456 | 865 | 0 |
| 11.25 | 40 | 6 | 12 | 4 | 476 | 868 | 0 |
| 11.50 | 30 | 8 | 12 | 4 | 501 | 868 | 0 |
| 11.75 | 20 | 7 | 12 | 4 | 526 | 868 | 0 |
| 12.00 | 9 | 6 | 12 | 4 | 552 | 868 | 0 |
| 12.25 | 0 | 5 | 10 | 4 | 578 | 868 | 0 |
| 12.50 | 0 | 0 | 3 | 2 | 600 | 868 | 0 |
| 12.75 | 0 | 0 | 4 | 3 | 614 | 868 | 0 |
| 13.00 | 0 | 0 | 4 | 3 | 629 | 868 | 0 |
| 13.25 | 0 | 0 | 3 | 2 | 645 | 868 | 0 |
| 13.50 | 0 | 0 | 4 | 3 | 659 | 868 | 0 |
| 13.75 | 0 | 0 | 3 | 2 | 675 | 868 | 0 |
| 14.00 | 0 | 0 | 4 | 3 | 689 | 868 | 0 |
| 14.25 | 0 | 0 | 3 | 2 | 705 | 868 | 0 |
| 14.50 | 0 | 0 | 4 | 3 | 719 | 868 | 0 |
| 14.75 | 0 | 0 | 4 | 3 | 734 | 868 | 0 |
| 15.00 | 0 | 0 | 3 | 2 | 750 | 868 | 0 |
| 15.25 | 0 | 0 | 4 | 3 | 764 | 868 | 0 |
| 15.50 | 0 | 0 | 3 | 2 | 780 | 868 | 0 |
| 15.75 | 0 | 0 | 3 | 2 | 795 | 868 | 0 |
| 16.00 | 0 | 0 | 4 | 3 | 809 | 868 | 0 |
| 16.25 | 0 | 0 | 4 | 3 | 824 | 868 | 0 |
| 16.50 | 0 | 0 | 4 | 3 | 839 | 868 | 0 |
| 16.75 | 0 | 0 | 3 | 2 | 855 | 868 | 0 |
| 17.00 | 0 | 0 | 4 | 3 | 869 | 868 | 0 |
| 17.25 | 0 | 0 | 3 | 2 | 885 | 868 | 0 |
| 17.50 | 0 | 0 | 3 | 2 | 900 | 868 | 0 |
| 17.75 | 0 | 0 | 4 | 3 | 914 | 868 | 0 |
| 18.00 | 0 | 0 | 2 | 2 | 930 | 868 | 0 |
| 18.25 | 0 | 0 | 0 | 0 | 932 | 868 | 0 |
| 18.50 | 0 | 0 | 0 | 0 | 932 | 868 | 0 |
| 18.75 | 0 | 0 | 0 | 0 | 932 | 868 | 0 |
| 19.00 | 0 | 0 | 0 | 0 | 932 | 868 | 0 |
| 19.25 | 0 | 0 | 0 | 0 | 932 | 868 | 0 |
| 19.50 | 0 | 0 | 0 | 0 | 932 | 868 | 0 |
Up to eight failed and eight completed requests, in arrival order within each group. A timeout during a database operation leaves the operation running until completion.
Arrived 5,200.0 ms · read · database path
Database wait 67.0 ms.
Deadline expired at stage: database.
Arrived 5,205.6 ms · write · database path
Database wait 65.7 ms.
Deadline expired at stage: database.
Arrived 5,211.1 ms · read · database path
Database wait 57.0 ms.
Deadline expired at stage: database.
Arrived 5,216.7 ms · read · database path
Database wait 60.5 ms.
Deadline expired at stage: database.
Arrived 5,222.2 ms · read · database path
Database wait 73.0 ms.
Deadline expired at stage: database.
Arrived 5,227.8 ms · read · database path
Database wait 59.4 ms.
Deadline expired at stage: database.
Arrived 5,233.3 ms · read · database path
Database wait 61.8 ms.
Deadline expired at stage: database.
Arrived 5,238.9 ms · write · database path
Database wait 72.9 ms.
Deadline expired at stage: database.
Arrived 0.0 ms · read · database path
Response after 57.1 ms.
Database wait 0.0 ms.
Arrived 16.7 ms · read · database path
Response after 43.1 ms.
Database wait 0.0 ms.
Arrived 33.3 ms · read · database path
Response after 51.5 ms.
Database wait 0.0 ms.
Arrived 50.0 ms · write · database path
Response after 47.3 ms.
Database wait 0.0 ms.
Arrived 66.7 ms · write · database path
Response after 43.3 ms.
Database wait 0.0 ms.
Arrived 83.3 ms · write · database path
Response after 45.8 ms.
Database wait 0.0 ms.
Arrived 100.0 ms · read · database path
Response after 48.4 ms.
Database wait 0.0 ms.
Arrived 116.7 ms · read · database path
Response after 55.7 ms.
Database wait 0.0 ms.
03 · CHANGE THE CONDITIONS
Keep your configuration fixed for the first attempt at three new conditions: a cold cache, a slower database, and a graceful reduction in application concurrency. Make a prediction before revealing their results.
If you retune against these public scenarios, they become practice. Record what broke and which capacity assumption you would validate with a production load test.
Save a practice run first. These public fixtures are a learning exercise; the scenario outcome is a property of this model.
04 · DEFEND THE DESIGN
0/4 checkpoints recorded. The practice objective is calculated from the simulation; written reasoning remains self-reported.
A saved prediction can be compared with an actual event-simulation run.
The latest practice configuration serves at least 95% of all requests within 300 ms in both practice workloads, at no more than 28 capacity points.
The three fresh scenarios have been attempted. Passing the practice objective does not imply that these conditions are safe.
Diagnosis, repair, and remaining risk are written down. A reviewer still needs to inspect the reasoning.
Progress and notes save in this browser.
Exports preserve the latest 20 runs with their predictions and settings. Share a configuration to make the decision reproducible; ask another person to challenge its assumptions.
THE MECHANISM
The simulator schedules each arrival, application completion, database completion, and client deadline. It processes events in time order. Events that share a timestamp use a fixed order: a completion wins a tie with its deadline. A seeded hash assigns read eligibility, cache hits, and service-time variation to each request. Changing capacity preserves the incoming requests and their service requirements.
The potential cache path is assigned at arrival; a cold-cache interval applies to those arriving requests. Only a completed cached response counts as a cache hit. A database slowdown applies when a query starts, and a worker reduction restricts new dispatches while existing requests drain. These boundaries make the experiment reproducible.
The default 18-second arrival window has four seconds at 60 requests/second, six seconds at 180, then eight seconds at 60. It offers 1,800 requests. Application work has a nominal duration of 10 ms; database work has a nominal duration of 40 ms. Both vary by up to twenty percent per request. These are authored assumptions, not measurements from a cloud service.
The ingress queue waits for a worker. Once dispatched, a noncached request keeps that worker while waiting for a database slot. Adding workers can move waiting work from ingress to the database queue without making the database faster. The timeline’s database-slot counter helps distinguish those states.
The nominal four-slot database can execute about 100 operations/second at 40 ms each. That estimate motivates a hypothesis; the event simulation determines actual outcomes with jitter, deadlines, workload mix, and queueing. The average utilization over the whole window can hide a fully busy burst interval.
A deadline expires one second after arrival. A request still queued or doing application work is canceled in this model, releasing its worker if needed. A database operation already running continues to completion and holds both resources. Its client receives a timeout, even when the database operation later finishes successfully.
This is why completed-response latency is reported separately from the fraction of all requests served within 300 ms. A configuration that admits very few requests can look fast while disappointing most users. Read the rejection and timeout counts alongside p95.
The illustrative budget charges one point per application slot, two per database service slot, and three when caching is enabled. It represents a reserved-capacity trade-off, rather than a currency. A higher real database connection limit may increase contention instead of creating the ideal parallel capacity assumed here.
The cache applies only to eligible reads. Writes always need the database. The read-heavy practice scenario and the write-heavy practice scenario deliberately share the arrival curve, so a cache-based repair has to survive a different mix. A fixed-window admission ceiling rejects excess work before queueing; the bounded ingress queue can reject it too.
If the database slots remain saturated, extra workers mostly wait for that same resource. Test it with unchanged database capacity and compare DB wait, useful responses, and capacity points.
Here p95 includes successful responses only. It leaves rejected and timed-out users out. The all-arrivals objective makes that missing experience visible.
Buffering and additional waiting time, not service capacity. A larger queue may absorb a short burst, but may also keep work until its client has given up.
Autoscaling a stateless application while its shared database is saturated, sizing for a warm cache, increasing connection pools without measuring database capacity, and judging overload only by successful-response latency all create misleading design reviews. The same reasoning appears in API gateways, webhook consumers, worker farms, and interview follow-up questions about the first bottleneck.
This model deliberately leaves out retries, cache invalidation correctness, transaction conflicts, network failures, variable payload costs, and real database contention. It holds service-slot capacity independent and does not simulate a CPU scheduler. Use it to form a testable operational hypothesis, then measure those omitted mechanisms in the actual environment.
Version harbor-service-2026-11-13.1. Each workload uses its own fixed seed. Requests arrive evenly within each phase; read classification and service-time jitter are deterministic. Worker reduction drains existing work rather than killing it.
practice · read-burst · seed 127
A support API rises from 60 to 180 requests/second for six seconds, then returns to 60. Eighty percent of requests are reads eligible for caching.
Find where requests wait. More application concurrency cannot create more database slots, and a queue stores unfinished work rather than creating throughput.
practice · write-burst · seed 211
The traffic curve stays the same, but a bulk integration changes the mix: only ten percent of requests are cacheable reads.
An apparent fix based on a high cache hit rate may stop working when the workload changes. Compare database starts and the fraction of all users served promptly.
transfer · cold-cache · seed 307
The read-heavy launch burst returns. During seconds 4–8 the cache has no hits, even if caching is configured.
A hit-rate assumption is a capacity dependency. Check the worst interval and explain what happens while the cache warms.
transfer · slow-database · seed 401
The same read-heavy arrival stream meets database operations that take twice as long during seconds 4–10. Existing queries keep the duration assigned when they started.
A slower dependency lowers service capacity even with the same number of connections. Timeouts after query start leave capacity occupied until the query finishes.
transfer · worker-drain · seed 503
During seconds 4–8, only half the configured application worker slots can accept new work. In-flight requests finish; this models a graceful capacity reduction.
Database headroom alone cannot rescue an application-concurrency bottleneck. A worker waiting for the database is still occupied.
Yes. Rejection can protect a system and preserve useful responses when capacity is unavailable. This customer’s objective still counts those users honestly. Explain the incident response separately from the long-term capacity needed to meet the objective.
It demonstrates this specified workload and event model. Validate request mix, cache behavior, dependency service times, deadlines, and resource scaling under real load, then test changed conditions. A private production workload can differ materially from these public fixtures.
The principles below motivate the investigation. The fictional workload, capacity points, and simulated timings are authored separately.
Handling Overload · Google SRE →Using load shedding to avoid overload · AWS Builders’ Library →NEXT LAYER · RETRIES ARE OFFERED LOAD
Compare no retry, immediate retry, and bounded backoff with jitter against a finite read service. A timeout does not cancel service already started. Useful completions count original customer operations once.
If 100 user operations each produce three attempts, the dependency sees 300 attempts, or 3× amplification. More attempts can recover transient failures and can also consume capacity while obsolete attempts are still running. Predict the effect before comparing policies.
This is a separate read-only capacity model, with FIFO waiting, 24 queue entries, an 1,800 ms original-operation budget, and seeded 0.4–1.6× service-time variation. It does not implement writes or payment deduplication. A write retry requires a valid idempotency protocol; a client timeout alone cannot establish whether a write committed.
Predict which policy will help, then run all three against the same initial operation stream.
Version read-retry-2026-11-17.1. Compare the measured result with your prediction and explain a condition that reverses your choice.
PUT THE DECISION BACK INTO THE ARCHITECTURE