The big idea

Meetings are where decisions go to be forgotten

Companies run on meetings, and meetings run on the worst storage medium ever invented: human memory. Decisions evaporate, action items orphan, and the one person who remembers "we already decided this in March" left the company. An AI notetaker turns an hour of messy multi-speaker audio into structured, searchable, actionable record — if you can solve capture, transcription, attribution, extraction and trust, live.

A pipeline: a bot that joins the call under consent rules, streaming ASR + diarization producing a speaker-attributed transcript, an LLM pass that extracts decisions and action items with timestamped receipts, a memory that makes months of meetings searchable, and delivery into the tools where work actually happens.

How to read this: Each step opens with a real design decision — make the call before seeing what ships. Watch the pipeline grow, hover any box, replay the flow. At the end, unleash crosstalk and watch error containment earn its keep. Hit Begin.

Step 1 · Get the audio

The capture bot

Before any AI: how do you get clean audio out of a Zoom call? The options shape everything downstream — a phone on the table hears the room; a per-participant stream hears each person.

Design decision: How should the system capture meeting audio?

The call: A bot participant that joins via the platform API and receives per-speaker streams. — The bot is a first-class attendee: visible to everyone (consent!), platform-blessed, and — where the API offers it — receiving separated per-participant audio, which makes "who said what" nearly free.

A Capture Bot joins through the platform’s API (Zoom/Meet/Teams), appears in the participant list by name, and streams audio out — per-speaker channels where the platform provides them, a mixed stream plus voice separation where it doesn’t. Visible presence is a feature: everyone knows the meeting is being recorded.

Capture quality is destiny: Every downstream stage — transcription, attribution, extraction — inherits the capture. Per-speaker streams from the source beat any amount of ML cleanup on a muddy room mix.

Step 2 · May we even record?

Consent is architecture, not a checkbox

Recording laws differ by state and country (one-party vs all-party consent), companies have no-record norms, and some calls — HR, legal, health — must never touch the pipeline. Where does consent live?

Design decision: How does the system handle recording consent?

The call: A policy engine gating the bot: announcements, opt-outs, blocklists, and per-workspace retention — enforced before capture starts. — Consent as code: the bot announces itself, honors "don’t record this call/series," respects per-region rules, and retention timers delete on schedule.

A Consent & Policy layer gates the bot: audible/visible announcement on join, host controls and opt-outs, calendar- and keyword-based no-record rules, jurisdiction-aware defaults, and per-workspace retention policies that hard-delete on schedule. The audit log records who accessed what, forever.

Trust is the product: A notetaker hears a company’s most sensitive raw material. One consent failure ends the deployment company-wide. Policy enforced in architecture — not in settings nobody finds — is what lets the product exist.

Step 3 · Words and voices

Streaming ASR + diarization

Now the pipeline earns its keep: turn audio into text (ASR) and attribute every word to a person (diarization). Meetings are the adversarial case — jargon, accents, interruptions, laughter, "can you see my screen?"

Design decision: Transcribe live during the call, or batch-process the recording after?

The call: Streaming, with a second higher-accuracy pass after the call. — Stream for the live experience (captions, running notes), then re-transcribe hard segments offline with a bigger model and full context. Two passes, each doing what it’s best at.

Streaming ASR transcribes in near-real-time — primed with a custom vocabulary (attendee names, product terms) and emitting per-word timestamps and confidence. Diarization attributes speech using per-participant channels when available, voice fingerprinting when not. A post-call second pass cleans up the hard segments.

Diarization is half the value: "Someone said we should delay launch" is trivia. "The VP of Sales said delay" is a decision record. Attribution is what turns a transcript into evidence — and overlapping speech is why it’s genuinely hard.

Step 4 · The durable record

One transcript store, everything points at it

Words, speakers, timestamps, confidence — where does it all live, and in what shape? The schema decision here quietly determines whether summaries can carry receipts later.

Design decision: What is the canonical stored form of a meeting?

The call: Timestamped, speaker-attributed, confidence-scored segments — with every downstream artifact linking back to segment IDs. — The transcript is the source of truth; summaries, action items and search results all cite segment IDs. Receipts become structural.

The Transcript Store holds the canonical record: segments with speaker, start/end timestamps, text and confidence — plus a pointer to the audio span. Everything downstream (summaries, tasks, search hits) references segment IDs, so every claim about the meeting has a receipt you can click and hear.

Store the evidence, derive the artifacts: Same law as every data system: keep the raw truth immutable, regenerate derivatives freely. When a user corrects a transcript segment, every artifact citing it can be rebuilt — that’s the payoff of pointing at IDs, not prose.

Step 5 · From words to work

Grounded extraction: decisions, actions, owners

An hour of transcript is still an hour of reading. The product is the distillation: what was decided, who owes what by when. And here lives the notetaker’s scariest failure mode — a fluent summary of things nobody said.

Design decision: How does the LLM turn a transcript into notes users can trust?

The call: Structured, GROUNDED extraction: decisions/actions/owners as typed fields, each required to cite the transcript segments it came from. — Every extracted item carries segment IDs — clickable receipts. Items that can’t cite a segment don’t ship. Extraction becomes checkable, not vibes.

The Understanding LLM produces structured outputs — summary sections, decisions, action items with owner and due date — and every item is grounded: it must cite the segment IDs (and thus timestamps) it derives from, with low-confidence audio flagged rather than paraphrased confidently. Click any note, hear the moment it came from.

Receipts beat fluency: The grounding requirement does double duty: it suppresses hallucination (can’t cite what wasn’t said) and it builds trust UI for free (every claim links to audio). Structured + grounded is the whole extraction doctrine.

Step 6 · The compound asset

Meeting memory: search across months

One meeting’s notes are convenient. The moat is longitudinal: "what did we tell this customer about pricing?", "when did we last discuss the migration?", "what did I promise legal?" — questions that span dozens of meetings nobody will re-read.

Design decision: How do months of meetings become answerable?

The call: RAG over segments and extracted items: embeddings + metadata filters (speaker, date, attendees), answers grounded in quoted segments. — Hybrid retrieval scoped by metadata, answers that quote and link the exact moments. The notetaker becomes the company’s conversational memory.

Meeting Memory indexes segments and extracted items — embeddings for meaning, metadata (speakers, attendees, date, customer) for filtering. Questions run retrieval-augmented answers that quote segments with links, scoped by the asker’s access rights. This is where a notetaker becomes organizational memory.

The asset compounds: Each meeting adds rows; the value grows super-linearly, because answers increasingly span meetings ("we decided this twice already"). Access control matters here most — memory that leaks across teams is a lawsuit, not a feature.

Step 7 · Meet users where work happens

Delivery and the feedback loop

Notes nobody sees change nothing. The recap has to reach email/Slack, action items have to become tracked tasks with owners, sales calls have to update the CRM — and user corrections have to flow back in.

Design decision: What happens with the finished notes?

The call: Push artifacts into existing tools — recap to email/Slack, tasks to the tracker with owners, CRM fields from sales calls — with edits flowing back. — Deliver into the workflow, not beside it. And every human edit (fixed transcript, corrected owner) is training signal + audit trail.

Delivery pushes each artifact to its home: recap to email/Slack, action items into the task tracker with owners and due dates, structured fields into the CRM for customer calls. Corrections flow backward — an edited segment re-derives its artifacts — and everything lands in the audit/telemetry stream: WER by meeting type, extraction-accuracy evals, access logs.

Close the loop: Human corrections are the notetaker’s eval set: every fixed name, re-owned task and edited summary is labeled data telling you where the pipeline fails. Teams that harvest corrections improve weekly; teams that don’t, plateau.

The payoff

You built an AI meeting notetaker

From an hour of chaotic multi-speaker audio to a trustworthy record: a consented capture bot, streaming ASR with a cleanup pass, diarization that names every voice, a canonical timestamped transcript, grounded extraction with clickable receipts, memory that answers questions across months, and delivery into the tools where work happens.

Now unleash the crosstalk and watch error containment work: confidence flags, grounded-only summaries, editable transcripts, and corrections that repair every downstream artifact.

• Capture bot — platform API, per-speaker streams, visibly present
• Consent as code — announcements, no-record rules, retention timers
• Stream + cleanup — live ASR pass, offline second pass on hard audio
• Diarization — attribution turns transcripts into evidence
• Segment IDs — one canonical store; every artifact carries receipts
• Grounded extraction — no citation, no claim — hallucination contained
• Meeting memory — RAG across months, scoped by access rights
• Deliver + learn — artifacts into real tools; corrections flow back