Your system works. The pilot went well. Then a team you have never met sends a 300-row spreadsheet, and eight weeks disappear. The vendor security review stops more enterprise AI deployments than any technical problem, and almost nobody practises it. Three rounds here, on one realistic deployment: a SIG/CAIQ-style questionnaire where the honest answer scores higher than the impressive one, a live CISO call with a trust meter that punishes bluffing exactly as the real one does, and a threat model of an LLM application where the plausible mitigation is usually not the one that holds.
Standardised vendor questionnaires. Hundreds of control questions, answered once, reused across customers. Yes/no with evidence.
compensating control
A different control that reduces the same risk when the expected one is missing. Named, owned, dated — and usually accepted.
the trust meter
Reviewers are calibrating whether your answers can be believed. A caught overstatement costs trust faster than strong answers earn it back, and it stays on the report as a high finding.
finding
What gets written down. Severity decides whether you ship now, ship with conditions, or wait for a remediation.
northwind-mutual — vendor security review
Round 1 · Questionnaire
You are deploying a document-triage assistant into a mid-size insurer. It reads claim documents from their warehouse, drafts a routing decision, and writes back to their claims system. Their security team has never approved an LLM application before.
1/3
round
0
score
0
findings
reviewer trust
How the review actually works
It is not one event. It is three, run by different people with different incentives, and losing any one of them costs you the same eight weeks.
01
The questionnaire (async, 1–3 weeks)
A SIG, a CAIQ, or a bespoke spreadsheet. Hundreds of rows covering access control, encryption, logging, incident response, sub-processors and business continuity. Answered by whoever is available — which is the first mistake, because an answer written by someone who has not seen the deployment describes company policy rather than this system.
02
The live call (60–90 minutes)
The security architect or CISO probes the answers that looked thin. This is where trust is won or lost, and the mechanism is simple: they are checking whether the written answers were written by someone who understood the question.
03
The technical review (1–4 weeks)
Architecture diagram, data-flow diagram, sometimes a penetration test or a live prompt-injection demonstration. Increasingly they ask for the AI-specific parts explicitly, because their existing framework was written before any of this existed.
04
The findings, and the three outcomes
Approved — ship. Approved with conditions — ship with remediations on dated commitments; this is the normal good outcome and people mistake it for failure. Blocked — a finding severe enough that no compensating control was offered, which is nearly always avoidable.
# the sequencing mistake that costs the launch date
week 1 discovery, architecture, first code
week 6 demo goes well, everyone is happy
week 9 "we'll need to run this past security" <-- the review starts HERE
week 17 approved with conditions
# the review took as long as the build.# what it looks like when someone has done this before
week 1 discovery + "who runs your vendor security review, and how long does it take?"
week 2 questionnaire started, architecture diagram drafted alongside the build
week 8 review runs in parallel with hardening
week 10 approved with conditions <-- seven weeks saved
The answer patterns that work
Every question in a security questionnaire has three possible answers, and the ranking is consistent enough to be a rule.
Answer shape
Example
What happens
Specific yes, with evidence
“Yes. Per-user OIDC, tokens validated against your tenant, session revoked within 60s of a SCIM deactivate. Sample audit lines attached.”
Approved and rarely re-opened. Evidence ends the thread.
Specific no, with a compensating control
“No SCIM today. A nightly reconciliation deactivates any user absent from your directory, log inspectable by your team. SCIM committed for Q4, owner named.”
Usually accepted. Becomes a condition, not a block.
Vague yes
“We follow industry best practices for access management.”
Escalates. It reads as concealment, and it invites a deeper look at everything else you wrote.
worst answer
vague yes
best recovery
gap + date
costs the deal
one caught bluff
start it in
week 1
The counter-intuitive part is that volunteering a gap raises your score. A reviewer’s entire job is finding what you did not mention. When you name it first, with a plan, you have done their job for them and given them a reason to believe the rest of the document.
The threats their framework does not cover
Enterprise security frameworks predate LLM applications. Round 3 covers the five that come up in practice — and in each case the obvious mitigation is weaker than it sounds.
A claim document contains text addressed to the model. The model reads documents all day.
Content is delimited and labelled as data, tool actions require confirmation, and the model’s permissions are the user’s. Not “we filter suspicious phrases”.
Over-broad retrieval
One index, one service account, so any user’s question can surface any indexed document.
Permissions enforced at retrieval time against the asking user — not filtering after the model has already seen the text.
Data leaving the boundary
Prompts and documents sent to a model provider; traces sent to an observability vendor.
A named list of every egress with its data classification, plus zero-retention or in-boundary inference. Their real question is “who else sees this?”
Unbounded tool actions
The agent can write to the claims system, and a bad decision is a real business event.
Write tools separated, confirmation required, every action reversible or ticketed. Reversibility beats accuracy claims.
No audit evidence
Nobody can reconstruct what the agent did on the 14th, or why.
Per-action logs with user, input, retrieved sources, output and model version, retained to their policy. “We log everything” is not evidence; ten lines are.
Expect to be asked for a live demonstration of the first one. Run it against yourself first — having a recorded answer converts a hard meeting into a short one.
The picture: how one answer moves the review
The rules the simulation runs on, read from its own code. Note the asymmetry. A strong answer earns back one bar of trust, and a bluff of −2 or worse costs two. A vague answer and an outright false one land in the same column: both are high findings. Four of those block the review even if trust never runs out.
Worked example: the plausible run that gets blocked
One run through all 12 questions, picking the answer that sounds reasonable in a meeting. No answer here is an outright lie. Points out of 36, trust out of 8.
The result is 5 of 36 points, or 14%. The review is Blocked, and not by trust, which ends at 4. Row 12 is the fourth high finding, and four is the limit. Now change only the four high-finding answers to the evidenced ones. The score becomes 22 of 36 (61%), with no high findings, and the verdict is still only Approved with conditions, because the five partial answers keep it under 65%. Change only row 12 and the result is 25% with three high findings: conditions again. Remove the high findings first. The partial answers are what decide between conditions and a clean approval.
Cost and trade-offs of each answer shape
Here Time is what the answer costs you now, and Space is what it leaves open for later.
Case
Time
Space
Why
Specific yes, with evidence
An hour pulling real log lines or config
Nothing. The row is closed
Evidence ends the thread. Nobody re-asks a question you answered with proof.
Specific no, with a compensating control
Writing the control, its exposure bound, an owner and a date
One dated condition
A small, scoped obligation that you proposed yourself, not one imposed on you.
Vague yes
None
A high finding, and closer reading of every other row
It cost nothing to write and gets probed on the call.
False yes
None
The deal, if they test it
Claims like SCIM revocation can be tested in minutes. A caught bluff reprices everything else you wrote.
“We will add whatever you need”
None
An open-ended commitment nobody scoped
You are now held to whatever they write into the conditions.
Review started in week 1, not week 9
One discovery question
Hardening runs in parallel with the review
In the page’s timeline, approval moves from week 17 to week 10.
The cheap answers are cheap only on the day you write them. Every row that costs nothing now leaves something open for later.
Common mistakes in a security review
Answering the question next to the one asked
TLS for a residency question, or encryption at rest for “who can read this?”. Each is a true statement about a different risk, and the reviewer hears that you did not understand the requirement. Restate the question in one line before answering it.
A gap with no bound
“A nightly job deactivates leavers” leaves the reviewer to work out the worst case. Say it yourself: at most 24 hours, the interval plus the job’s run time. An unbounded gap gets read as an unmanaged one.
Forgetting an egress
The model API is the egress everyone lists. The trace exporter, the error tracker and the eval tool get forgotten, and any of them may be capturing prompt bodies. Check what each one actually sends before you write “no other processors”.
The questionnaire and the diagram disagree
The questionnaire is answered in week 2 and the architecture diagram is drawn in week 6. If they describe different systems, the reviewer believes neither. Answer both from one source and re-check the answers when the design changes.
Treating a condition’s date as soft
An approval with conditions turns each date into a tracked commitment. Miss one and the conversation is about the slipped date, not your controls, and every other date you gave is now in doubt. Only commit to dates you control.
Where this shows up in real-world deals
Regulated buyers
Insurers, banks and healthcare providers run this review on every vendor that touches their data. The standard forms are SIG, from Shared Assessments, and CAIQ, from the Cloud Security Alliance. Teams keep an answer library so each new review starts from reviewed answers, not a blank sheet.
Audit reports as evidence
A SOC 2 report covers many generic rows at once: access, change management, incident response. It says nothing about retrieval permissions or prompt injection, so the AI-specific rows still need answers about this deployment.
The AI rows being added now
Reviewers increasingly add their own rows: does the provider train on our data, how long does it keep prompts, what can the model do on its own. Round 3’s threats are the ones these rows are trying to reach.
Interviews for customer-facing engineers
Forward-deployed and solutions interviews often role-play the CISO question. The structure that works is the one this lab scores: a specific statement, the mechanism, the bound on the exposure, an owner and a date, and evidence they can check.
Check yourself
You give the evidenced answer on 11 questions and a vague yes on the first. What verdict does the gauntlet give, and why?
Approved with conditions. The score is 32 of 36 (89%) and trust ends at 8, but the vague yes is a high finding, and a clean approval needs zero. One unsupported row is enough to attach a condition.
Their policy is revocation within 4 hours. Your reconciliation job takes 30 minutes to run. Does running it every 4 hours meet the policy?
No. The worst case is the interval plus the run time: a leaver removed just after a run starts waits about 4.5 hours. Every 3 hours gives a worst case of 3.5 hours, which does meet it. State the bound, not just the schedule.
You are asked to list sub-processors and are not sure whether your error tracker captures prompt text. What do you write?
Find out before answering. If it captures prompts, list it with its data classification, or redact before export and say so. Leaving it out is the answer that fails when they read your diagram. Listing it “just in case” without knowing what it receives is a vague answer.
Trust is 4 of 8 and you never told an outright lie, yet the review is blocked. What happened?
Four high findings. The block rule is trust ≤ 2 or four or more high findings, and a vague answer counts as high just as a false one does. The worked example above gets there through answers that each sounded reasonable.
Common questions
What is a vendor security review?
The process an enterprise runs before letting a third-party system touch its data: a written questionnaire such as SIG or CAIQ, a live call with the security team, and a technical review or threat model. For AI deployments it is the most common cause of a delayed launch, and it is normally discovered late — when the timeline is already fixed.
How do you answer honestly without losing the deal?
State what you do, name what you do not, and pair every gap with a compensating control, an owner and a date. Reviewers are trained to find what you left out, and they check. A specific admission with a plan is routinely accepted; a vague assurance that turns out to be false ends the review and often the relationship.
What is a compensating control?
A different control that reduces the same risk when the expected one is absent. No SCIM yet? A daily reconciliation that deactivates any user no longer in the customer directory, with an inspectable log, is a compensating control. Named, owned and dated, it is usually acceptable. An unacknowledged gap is not.
Who should fill in the questionnaire?
The engineer who built the system, reviewed by whoever owns compliance. Questionnaires answered purely by a sales or compliance function describe company policy rather than this deployment, and the discrepancy surfaces on the technical call — which damages you far more than the original gap would have.
When should the security review start?
Week one, during discovery. Ask who runs it and how long it usually takes, then run it in parallel with the build. Teams that discover the review after the demo routinely add seven or eight weeks to the launch, because the review takes about as long as the build did.