The big idea

What does a code review bot actually need to get right?

The obvious version is "send the diff to an LLM, post whatever it says as PR comments." Try that at real scale and you get a bot developers mute within a week: it repeats itself on every push, comments confidently on things it doesn't have context for, buries the one real bug under nitpicks, and occasionally quotes a leaked API key straight back into a public comment thread.

Every one of those failures has a specific fix: run cheap, deterministic checks before any LLM call, scope review to the diff plus just enough context, retrieve repo-wide context the diff alone can't show, gate on confidence before posting, track what's already been said, and redact secrets. We'll build all of it.

How to read this: Each step opens with a real design decision — make the call before I show you what ships. Watch the diagram grow, hover the boxes, and at the end kill the LLM layer and disable the comment ledger to see what degrades and what spams. Hit Begin.

Step 1 · Cheap checks first

Don't send everything to an LLM

A PR touches 30 files. Some issues — an unused import, a hardcoded credential pattern, a missing semicolon — are trivially, deterministically detectable. Should everything, including these, go through an LLM call?

Design decision: An unused variable and a hardcoded secret pattern are both deterministically detectable. Route them through an LLM anyway?

The call: Run fast, deterministic static analyzers and linters FIRST — they catch a large, well-understood class of issues instantly and for free — and reserve the LLM for issues that genuinely require understanding intent. — Static analysis tools are precise and instantaneous for their class of problems (unused code, known-bad patterns, style). Running them first means the LLM never has to (and never should try to) rediscover what a linter already knows with certainty — its budget goes entirely toward the semantic reasoning only it can do.

Run static analyzers and linters first — deterministic, instant, and precise for their class of issues. Only what genuinely requires understanding intent (does this logic actually do what the PR claims, is this the right approach) needs to reach the LLM at all.

Cheap deterministic tool before expensive probabilistic one: The same principle that says "use a symbolic checker before an LLM grader" in an adaptive tutor applies here: reach for a tool that's exactly correct and nearly free whenever one exists, and reserve the expensive, probabilistic model for what actually requires judgment.

Step 2 · Diff-scoped, not file-scoped

Review the change, with just enough context

A file is 800 lines; a PR changes 12 of them. Should the LLM see the whole file every time, or just the diff?

Scope review to the diff plus surrounding context — the changed lines, a reasonable window of unchanged code around them for coherence, but not the entire file (let alone the entire repo). This keeps every review focused on what actually changed (comments on 770 untouched lines waste tokens and, worse, invite the bot to comment on pre-existing code that wasn't part of this PR at all), and keeps the token/cost budget proportional to the size of the change, not the size of the file.

Relevance and cost are the same lever: Scoping to the diff isn't only a cost optimization — it's also what keeps every comment relevant to what the PR author is actually asking to be reviewed. A comment on unrelated pre-existing code is both wasted tokens and a trust-eroding non-sequitur to the developer.

Step 3 · Context the diff alone can't show

What about the function this diff calls?

The diff modifies a function's call site, passing an argument in a new order. The LLM can't judge whether that's correct without knowing the CALLED function's actual signature — which isn't in the diff at all, because that function wasn't touched.

Design decision: The diff calls a function whose definition isn't in the diff. How does the LLM get that context?

The call: Retrieve relevant code from a Repo Embedding Index — an indexed, searchable representation of the whole codebase — and pull in just the specific definitions, related files, or conventions the diff actually references. — This is RAG applied to a codebase: rather than shipping the whole repo to the LLM (far too large, mostly irrelevant), retrieve exactly the symbols and context the diff touches — the called function's signature, a relevant type definition, the project's style guide — and include only that.

Maintain a Repo Embedding Index — an indexed, retrievable representation of the codebase's symbols, definitions, and conventions. When the diff references something outside itself (a called function, an imported type, a style guideline), the Context Assembler retrieves just that, giving the LLM the specific cross-file context it needs without shipping the entire repository.

RAG isn't just for chat: The same retrieve-then-generate pattern used for grounding chatbot answers in documents applies directly to code review: retrieve the specific, relevant symbols a diff touches from an indexed codebase, rather than either guessing without them or dumping everything.

Step 4 · The semantic layer

What only an LLM can actually judge

With static checks done and context assembled, the LLM reviews for logic correctness, missed edge cases, naming, and whether the change actually does what the PR description claims. Should every single thing it flags get posted as a comment?

Design decision: The LLM flags a dozen potential issues, from "this could theoretically be null" to a clear logic bug. Post all of them?

The call: Score each finding by confidence/severity, and only surface the ones that clear a bar — suppress or batch low-confidence nitpicks rather than posting everything the LLM generates. — Not every LLM-generated observation is worth interrupting a developer for. Attaching a confidence/severity score to each finding and gating what actually gets posted keeps the bot's signal-to-noise ratio high, which is the difference between a tool developers trust and one they mute.

Have the LLM attach a confidence and severity score to each finding, and gate posting on that score — clear logic bugs and security issues surface immediately; speculative or stylistic nitpicks are suppressed or batched into a single low-priority summary rather than posted as individual interruptions.

Precision over recall, for anything interruptive: For a tool that interrupts a human's workflow with every output, precision (is this comment actually worth their time) matters more than recall (did we catch every conceivable issue). A bot that's right 95% of the time it speaks earns trust; one that's right 60% of the time, even with higher raw catch rate, gets muted.

Step 5 · Never repeat yourself

A re-push shouldn't repost old comments

A developer pushes a fix, then pushes again to address review feedback, then again for a typo. Each push re-triggers review. Should each run repost everything it finds, including issues already flagged (and possibly already fixed) on an earlier push?

Design decision: A PR gets pushed 5 times. Should each push's review repost all previously-flagged issues again?

The call: Track every posted comment against its specific diff-hunk in a Comment Ledger, and on each new push, only surface findings that are genuinely new — resolved issues are recognized as resolved, unchanged flagged issues aren't reposted. — A ledger of "what has already been said, about what specific piece of code" lets each new review run diff against its own prior findings: already-flagged-and-unresolved issues stay silent (they were already said once), resolved issues can even be acknowledged, and only genuinely new findings from the latest push get posted.

Maintain a Comment Ledger keyed by (finding, diff-hunk) that records exactly what's already been posted. Every new review run checks against it: a previously-flagged, still-unresolved issue isn't reposted; a resolved one can be quietly closed; only genuinely new findings from the latest push generate a new comment.

Idempotency is a product requirement here, not just an engineering nicety: For a bot that re-runs on every push to a long-lived PR, "did I already say this" is load-bearing for user trust, not an edge case — without it, comment volume scales with push COUNT rather than with actual issue count, which is a bug users experience directly as spam.

Step 6 · Handle secrets carefully

Don't leak what you're flagging

A diff accidentally includes a real API key. The bot should flag it — but sending that raw secret up to an external LLM provider as part of "look what I found," and then quoting it back verbatim in a public PR comment, would make the leak WORSE, not better.

Detect likely secrets with a pattern-based scanner (part of the deterministic Step 1 layer) before anything reaches the LLM, and redact the actual value in both the LLM request and the posted comment — flag "a credential-shaped string was found on line 42" without ever repeating the credential itself anywhere in the pipeline, including in logs.

A finding about sensitive data shouldn't itself become a new exposure: The bot's output is, itself, a place data can leak from — a comment thread, a log line, a request to a third-party LLM provider. Redaction has to happen at detection time, upstream of every downstream consumer of that finding, not as an afterthought on the final comment text alone.

Step 7 · Learn from developer reactions

Fewer repeat false positives over time

Developers thumbs-down a category of comment repeatedly — say, a particular style suggestion this team has explicitly decided against. Should the bot keep making the same suggestion forever?

Feed 👍/👎 reactions and explicit dismissals back into per-repo confidence thresholds — a category of finding that's consistently downvoted on a given repo gets its posting bar raised (or is suppressed outright) for that repo specifically, without affecting other repos where the same finding might be genuinely valued. This is a lightweight feedback loop, not a full retraining pipeline: threshold and prompt adjustments, scoped per repository.

Feedback loops make trust compound: A static bot is equally annoying on day 1 and day 100. A bot that visibly adapts to a team's specific preferences over its first few weeks earns increasing trust — the same reason spam filters and recommendation systems both lean on explicit user feedback signals, applied here to code review specifically.

Step 8 · The sharp edges

The 5,000-line PR

Most PRs are small. Occasionally one isn't — a large refactor or a vendored dependency update touches thousands of lines. Reviewing all of it with full LLM depth would be slow and expensive, and most of a huge mechanical diff is genuinely low-risk.

Apply a budget and prioritization strategy for large PRs: chunk the diff, prioritize files by risk signal (source code over generated/vendored files, files touching auth/payments/security-sensitive paths over config), and apply full LLM-depth review to the highest-priority chunks first within a fixed budget — falling back to static-analysis-only coverage for the long tail rather than either reviewing everything at full depth (too slow/expensive) or skipping huge PRs entirely (too risky).

Design for the unhappy path: LLM down → static checks still run. Ledger down → comments spam. Huge PR → prioritized, budgeted review instead of an unbounded cost or a silent skip. A review bot that only works cleanly on small, cooperative PRs is a demo; one that degrades gracefully and prioritizes sensibly under real-world PR sizes is a product.

You did it

You just designed an AI code review bot.

• C — h
• R — e
• A —
• E — v
• A —
• L — i
• D — e
• L — a