The whole design, in writing
Every step of the build above, written out: the problem each piece solves, the option that was taken and the ones that were not, the numbers, and how it fails in production.
The big idea
Meetings are where decisions go to be forgotten
Companies run on meetings, and meetings run on the worst storage medium ever invented: human memory. Decisions evaporate, action items orphan, and the one person who remembers "we already decided this in March" left the company. An AI notetaker turns an hour of messy multi-speaker audio into structured, searchable, actionable record — if you can solve capture, transcription, attribution, extraction and trust, live.
A pipeline: a bot that joins the call under consent rules, streaming ASR + diarization producing a speaker-attributed transcript, an LLM pass that extracts decisions and action items with timestamped receipts, a memory that makes months of meetings searchable, and delivery into the tools where work actually happens.
What the new pieces do
- Meetingclient
- The live call: multiple speakers, screen shares, jargon, interruptions — an hour of unstructured audio that contains real decisions.
Step 1 · Get the audio
The capture bot
Before any AI: how do you get clean audio out of a Zoom call? The options shape everything downstream — a phone on the table hears the room; a per-participant stream hears each person.
How should the system capture meeting audio?
One mixed-down stream, whatever that laptop hears — speakers blended together, no per-person channels, quality hostage to one machine. Diarization starts life crippled.
The bot is a first-class attendee: visible to everyone (consent!), platform-blessed, and — where the API offers it — receiving separated per-participant audio, which makes "who said what" nearly free.
N installs, N OS permission fights, N reasons the meeting starts without capture. The bot needs zero attendee setup.
A Capture Bot joins through the platform’s API (Zoom/Meet/Teams), appears in the participant list by name, and streams audio out — per-speaker channels where the platform provides them, a mixed stream plus voice separation where it doesn’t. Visible presence is a feature: everyone knows the meeting is being recorded.
Why this piece earns its place
The bot is not a service in the usual sense — it is a client, one long-lived media session held open for the length of a call, and that changes how you size the fleet. Capacity is counted in concurrent meetings, not requests, and meetings do not arrive evenly: they start on the hour and the half hour, so a fleet planned against the average is short exactly when every call begins at once, and the ones it turns away started on time. The failure worth designing for is the join, not the stream — waiting rooms, a host who admits participants by hand, a passcode, a meeting that runs late, someone removing the bot mid-call. Each yields a transcript that exists, parses and reads as complete while the opening is simply missing, a gap no error rate will show you because every word present is correct. So record coverage explicitly: which spans the bot was in the room for, carried with the recording, so nothing downstream summarises a stretch nobody captured. You can skip the fleet entirely if you are single-platform and every call is already cloud-recorded — you trade the live surface for somebody else’s infrastructure.
What the new pieces do
- Capture Botbackend
- A headless participant that joins via the platform’s API, announces itself, and streams per-speaker audio out of the call.
Step 2 · May we even record?
Consent is architecture, not a checkbox
Recording laws differ by state and country (one-party vs all-party consent), companies have no-record norms, and some calls — HR, legal, health — must never touch the pipeline. Where does consent live?
How does the system handle recording consent?
Legally flimsy in all-party-consent jurisdictions and socially corrosive everywhere. Nobody reads the ToS; everybody notices a secret recorder.
The recording WAS the violation. All-party-consent law doesn’t offer an undo, and neither does trust.
Consent as code: the bot announces itself, honors "don’t record this call/series," respects per-region rules, and retention timers delete on schedule.
A Consent & Policy layer gates the bot: audible/visible announcement on join, host controls and opt-outs, calendar- and keyword-based no-record rules, jurisdiction-aware defaults, and per-workspace retention policies that hard-delete on schedule. The audit log records who accessed what, forever.
Why this piece earns its place
Two things about consent only surface once this is running. First, jurisdiction is a property of the participants, not of the meeting. One attendee dialling in from an all-party state makes the whole call all-party, and the participant list keeps changing after the bot joins — so the check belongs on every join event, not one decision at the top of the call. The half nobody specifies is what that re-check does to audio already captured: when a late arrival tightens the rules, the policy has to say whether the earlier minutes survive. Second, retention is a delete problem, and by the last step of this design the same sentence lives in the transcript store, the memory index, a recap email, a task in the tracker, a CRM field and a provider’s logs. A timer that sweeps one store is theatre. Several of those copies must be content, not a link — a task with an owner and a due date is useless as a URL — so deletion is a fan-out of writes, one per destination, and every artifact records where it was written and which meeting it came from. What you cannot reach that way belongs in the policy you publish, not the one you enforce.
What the new pieces do
- Consent & Policyservice
- Enforces recording law and company policy: announcement, opt-outs, no-record calendars, and per-workspace retention rules.
Back of the envelope
- join announcement
- the bot is visibly, audibly present
- no-record rules
- calendars, keywords, series, HR/legal blocklists
- retention timers
- hard delete per workspace policy
- access audit
- every view of every meeting, logged
Step 3 · Words and voices
Streaming ASR + diarization
Now the pipeline earns its keep: turn audio into text (ASR) and attribute every word to a person (diarization). Meetings are the adversarial case — jargon, accents, interruptions, laughter, "can you see my screen?"
Transcribe live during the call, or batch-process the recording after?
Batch is simpler and slightly more accurate, but it forfeits the live product surface: in-call captions, real-time highlights, "catch me up, I just joined." The market moved live.
Stream for the live experience (captions, running notes), then re-transcribe hard segments offline with a bigger model and full context. Two passes, each doing what it’s best at.
The search-across-months product (step 6) needs everything. Selective capture guts the memory.
Streaming ASR transcribes in near-real-time — primed with a custom vocabulary (attendee names, product terms) and emitting per-word timestamps and confidence. Diarization attributes speech using per-participant channels when available, voice fingerprinting when not. A post-call second pass cleans up the hard segments.
Why this piece earns its place
Two passes means two transcripts of the same hour, and reconciling them is what bites. If a segment’s identity comes from the streaming pass’s utterance boundaries, the cleanup pass renumbers the record, and every citation made during the call — a live highlight, a note already shared — now points at different words. Anchor identity to time, so the second pass revises text inside a stable span instead of re-cutting it. The other thing worth saying is that diarization does not produce people. It produces consistent anonymous clusters — speaker one, speaker two — and turning those into names is a separate join against the participant list: a conference room is one participant and several voices, and somebody who never speaks sits on the list all hour. Cluster count and attendee count need not agree, and where they don’t the system should say it is unsure rather than put a name on a decision. Build the affordance that lets someone fix that mapping once, because one correction re-owns every action item in the meeting. And expect the live text to rewrite itself: streaming ASR revises words as context arrives, so anything reading the stream must tolerate text changing underneath it.
- ~real-timestreaming first pass
- 2nd passoffline cleanup of hard audio
- per wordtimestamp + confidence
What the new pieces do
- Streaming ASRservice
- Transcribes audio in near-real-time, primed with attendee names and product vocabulary. Emits words with timestamps and confidence.
- Diarizationservice
- Attributes every word to a speaker — from per-participant audio channels when the platform provides them, voice fingerprints when it doesn’t.
Step 4 · The durable record
One transcript store, everything points at it
Words, speakers, timestamps, confidence — where does it all live, and in what shape? The schema decision here quietly determines whether summaries can carry receipts later.
What is the canonical stored form of a meeting?
Summaries are derived artifacts. Store only the derivative and you can never re-derive, re-verify, or answer "what EXACTLY was said?"
The transcript is the source of truth; summaries, action items and search results all cite segment IDs. Receipts become structural.
Audio is the backup, not the record — unsearchable, unquotable, and expensive to reprocess for every question.
The Transcript Store holds the canonical record: segments with speaker, start/end timestamps, text and confidence — plus a pointer to the audio span. Everything downstream (summaries, tasks, search hits) references segment IDs, so every claim about the meeting has a receipt you can click and hear.
Why this piece earns its place
The segment boundary rule is an API, and you choose it once. Artifacts hold segment ids forever, so deciding later that utterances were the wrong unit — too short to hold an argument, too long to quote — re-ids the whole corpus and breaks every citation already sitting in a task or a CRM field. It is the cheapest decision on this page today and among the most expensive to revisit. Second: a correction is an append, not an update. Audio stays the ground truth, and an edited segment has to carry who changed it and when, because the question that eventually arrives is not what was said but who altered the record and whether they were in the room. One step downstream, that means an artifact cites a segment revision, not merely an id — otherwise re-deriving quietly rewrites a recap somebody already read and acted on, with nothing marking that it moved. Storage shape follows the same split: segments are small and stay hot, audio is the expensive half and tiers down, so click-to-hear carries a retrieval delay on old meetings. Decide whether you tell the user that, or pin the audio spans an artifact actually cites.
What the new pieces do
- Transcript Storestore
- The durable record: timestamped, speaker-attributed, confidence-scored segments. Every downstream artifact points back here.
Step 5 · From words to work
Grounded extraction: decisions, actions, owners
An hour of transcript is still an hour of reading. The product is the distillation: what was decided, who owes what by when. And here lives the notetaker’s scariest failure mode — a fluent summary of things nobody said.
How does the LLM turn a transcript into notes users can trust?
Fluent, plausible, and occasionally fabricated. An invented action item with a real person’s name attached is a trust-ending event.
Every extracted item carries segment IDs — clickable receipts. Items that can’t cite a segment don’t ship. Extraction becomes checkable, not vibes.
Meetings don’t talk like that — decisions emerge across ten fuzzy minutes. This is why LLM understanding is the unlock; regexes never worked.
The Understanding LLM produces structured outputs — summary sections, decisions, action items with owner and due date — and every item is grounded: it must cite the segment IDs (and thus timestamps) it derives from, with low-confidence audio flagged rather than paraphrased confidently. Click any note, hear the moment it came from.
Why this piece earns its place
A model asked to emit segment ids will emit plausible segment ids. Grounding is only real if the citation is verified after generation, mechanically: every id resolves, belongs to this meeting, and — with a second cheap pass — actually supports the claim it is attached to. Skip that check and grounding is a prompt instruction, which is to say a suggestion, and a fabricated item now ships wearing a receipt. That is worse than an uncited one, because the receipt is the signal that tells a reader somebody checked. The operational half is identity. Extraction runs again after the cleanup pass and again after every human correction, so an action item needs stable identity derived from what it cites and who owns it, not a fresh id per generation. Get that wrong and re-deriving a meeting after one fixed name opens a second ticket for work that already exists — which earns a notetaker a reputation for noise faster than any hallucination will.
What the new pieces do
- Understanding LLMservice
- Reads the transcript and extracts the meeting’s structure: summary, decisions, action items with owners — each grounded to segment timestamps.
Step 6 · The compound asset
Meeting memory: search across months
One meeting’s notes are convenient. The moat is longitudinal: "what did we tell this customer about pricing?", "when did we last discuss the migration?", "what did I promise legal?" — questions that span dozens of meetings nobody will re-read.
How do months of meetings become answerable?
"Pricing" appears in 200 meetings. Keyword search finds the word; the question needs the discussion — meaning, speakers, dates, decisions.
Months of meetings is millions of tokens per query — slow, expensive, and context-rotted. Retrieval exists for exactly this.
Hybrid retrieval scoped by metadata, answers that quote and link the exact moments. The notetaker becomes the company’s conversational memory.
Meeting Memory indexes segments and extracted items — embeddings for meaning, metadata (speakers, attendees, date, customer) for filtering. Questions run retrieval-augmented answers that quote segments with links, scoped by the asker’s access rights. This is where a notetaker becomes organizational memory.
Why this piece earns its place
Semantic similarity has no sense of time, and that is what hurts here. Ask what was decided about pricing and the index returns March and September, both genuinely about pricing — and the answer confidently reports a decision overturned months ago. Neither is stale, which is what makes it hard: both are accurate records of what was said. Recency weighting is the floor; the real fix is linking decisions on a subject into a chain, so an answer can say it was decided once and changed later, which is usually what the asker needs. The second production surprise is that rights move and an index does not. Narrowing on workspace, attendees and the meeting ACL before ranking is the right mechanism — where those values come from is the bug. Copy a permission snapshot into the index at write time and it is a cache nobody invalidates: someone leaves the team, a meeting is locked down, an attendee is removed, and the filter keeps matching the row it stored. It fails open, silently, because over-broad retrieval looks exactly like retrieval that worked. Resolve rights against the live source on every query, build the filter from what comes back, and pay the lookup.
What the new pieces do
- Meeting Memorystore
- Embeddings + metadata over every past meeting: "what did we decide about pricing in March?" answered with quotes and links.
Step 7 · Meet users where work happens
Delivery and the feedback loop
Notes nobody sees change nothing. The recap has to reach email/Slack, action items have to become tracked tasks with owners, sales calls have to update the CRM — and user corrections have to flow back in.
What happens with the finished notes?
Another inbox nobody checks. The value of an action item is zero until it exists where tasks are actually tracked.
Deliver into the workflow, not beside it. And every human edit (fixed transcript, corrected owner) is training signal + audit trail.
Nobody reads an hour of transcript; sending it all also maximizes the privacy surface. Deliver the distillation; link the receipts.
Delivery pushes each artifact to its home: recap to email/Slack, action items into the task tracker with owners and due dates, structured fields into the CRM for customer calls. Corrections flow backward — an edited segment re-derives its artifacts — and everything lands in the audit/telemetry stream: WER by meeting type, extraction-accuracy evals, access logs.
Why this piece earns its place
Delivery is the only place this system writes into tools it does not own, and those writes are not reversible the way a bad summary is. A wrong task gets assigned, notified and half-discussed before anyone checks it; deleting it unsends none of that. Delivery is also a fan-out of independent writes, so it needs per-destination state: Slack takes the recap, the tracker refuses it because the project requires a field nobody mentioned, and one delivered flag on the meeting is a lie. And every owner has to survive a lookup in another system — the tracker wants an account, the CRM a contact, and the meeting hands you a display name a human chose. Where that fails, ship the item unassigned and say so rather than guess the nearest match. Then the thing that quietly misleads teams: corrections are not a sample. You only get an edit for a meeting somebody read, and those are the important ones and the obviously broken ones. Word error rate over corrected segments is not the fleet’s, and it will look like it is improving while the untouched long tail does not move. Use corrections to decide where to look; keep a separate randomly sampled labelled set for the number you report.
- tasksborn with owners + due dates
- editsflow back and re-derive
- WER + evalstracked per meeting type
What the new pieces do
- Deliveryservice
- Pushes artifacts where work happens: recap email, tasks into the tracker with owners, CRM fields updated from sales calls.
- Telemetry / Auditbus
- Quality and access logs: word-error-rate tracking, extraction accuracy evals, and who viewed which meeting — the audit trail.
The payoff
You built an AI meeting notetaker
From an hour of chaotic multi-speaker audio to a trustworthy record: a consented capture bot, streaming ASR with a cleanup pass, diarization that names every voice, a canonical timestamped transcript, grounded extraction with clickable receipts, memory that answers questions across months, and delivery into the tools where work happens.
Now unleash the crosstalk and watch error containment work: confidence flags, grounded-only summaries, editable transcripts, and corrections that repair every downstream artifact.
Everything you assembled, in order
- Capture bot — platform API, per-speaker streams, visibly present
- Consent as code — announcements, no-record rules, retention timers
- Stream + cleanup — live ASR pass, offline second pass on hard audio
- Diarization — attribution turns transcripts into evidence
- Segment IDs — one canonical store; every artifact carries receipts
- Grounded extraction — no citation, no claim — hallucination contained
- Meeting memory — RAG across months, scoped by access rights
- Deliver + learn — artifacts into real tools; corrections flow back