The whole design, in writing
Every step of the build above, written out: the problem each piece solves, the option that was taken and the ones that were not, the numbers, and how it fails in production.
The big idea
Grounded in a life, not the web
This isn't search over the public internet — it's answering questions using years of someone's own journals, notes, and documents. That makes two things true at once: the corpus is deeply personal (privacy has to be central, not bolted on), and it's messy, heterogeneous, and constantly growing in a way a curated knowledge base never is.
We'll build retrieval and ingestion designed for that reality: incremental updates, disambiguation using personal context, citations back to the real source document, and privacy as an architectural constraint that shapes the whole design, not a feature added at the end.
What the new pieces do
- Userclient
- Wants answers grounded in years of their own notes, journals, and documents — and needs to trust the assistant with genuinely private information.
Step 1 · The baseline
Dump everything into the context window
Simplest version: for every question, stuff all the user's personal content directly into the LLM's context window and ask it to answer. What happens as years of notes and documents accumulate?
Every query dumps the user's entire personal corpus into the context window. What happens as it grows over years?
Years of journal entries, notes, emails, and documents from an actively-used personal assistant can genuinely exceed even large context windows, and even before hitting a hard limit, stuffing everything in gets expensive and slow for every single query.
Personal corpora genuinely grow without bound over years of use — dumping everything in doesn't scale, both because of hard context limits eventually and because it's a wasteful, expensive way to answer a question that usually only needs a small, specific slice of the whole history.
Even moderate, ordinary personal note-taking and document accumulation over multiple years adds up to a substantial corpus — this isn't only an edge case for unusually prolific users.
A personal corpus grows without bound over years of real use, eventually exceeding any context window — and even before that, sending everything for every query is wasteful when only a small, specific slice is actually relevant. This needs real retrieval, not brute-force context stuffing.
Why this piece earns its place
Build the naive version anyway and keep it running over a small slice of the corpus, as a control. When an answer comes out wrong later, re-asking it with the right documents pasted in by hand separates two failures that look identical from outside: retrieval never fetched the right note, or it did and the model still answered badly. Without that split, a retrieval regression arrives as a model problem. It is tempting to call that a quality ceiling, and it is not one: a window packed with years of unrelated content is mostly distractors, and models get worse at finding the relevant line as the pile grows, so stuffing usually sits below what a good retriever manages. The limit itself is loud: providers reject an over-window request outright with an explicit error rather than trimming it for you, so nothing degrades gently — the assistant works, then one day it errors for one user and not another. It breaks as a function of tenure, so your longest-standing users hit it first while the aggregate error rate reads fine. The blast radius lands on Step 6: stuffing ships the user's whole life to the provider on every question, so one mishandled request exposes everything, not the few documents that query needed. For a new user with a dozen notes, stuffing is genuinely correct — the design question is where the cutover sits and who is watching.
What the new pieces do
- Continuous Ingestiongateway
- New personal content arrives constantly — notes, emails, documents — and has to be absorbed incrementally, not in periodic batch reprocessing.
- Answer + Citationservice
- Generates answers grounded in retrieved personal content — and cites exactly which document it drew from, every time.
Step 2 · Retrieval over messy personal content
Not a curated knowledge base
Build real retrieval (RAG) over the personal corpus. Personal documents — journal entries, scanned receipts, meeting notes, half-finished drafts — are far more heterogeneous and noisier than a curated knowledge base. What does that mean for retrieval?
Personal documents are heterogeneous and noisy compared to a curated knowledge base. What does retrieval need to account for?
Standard retrieval techniques designed for relatively consistent, well-formed document collections can struggle with the genuine mess of real personal content — wildly varying formats, quality, and structure that a curated corpus doesn't have.
Real personal content spans everything from a two-word reminder to a scanned multi-page document to a rambling journal entry — retrieval built assuming consistent, well-structured documents will underperform on this genuine heterogeneity, so the retrieval approach has to be designed for messiness from the start, not adapted later.
Ignoring poorly-formatted or non-text content (scanned documents, quick fragments) would discard a large and often genuinely important share of real personal content — the goal is handling the mess, not filtering it out.
Build the Personal Corpus Index and Retriever to handle genuine format heterogeneity and noise from the start — short fragments, scanned content, unstructured notes — rather than assuming the relatively clean, consistent structure a curated knowledge base would have.
Why this piece earns its place
Two properties of a personal corpus change the engineering, and neither is about mess for its own sake. First, the unit is wrong: plenty of personal items are shorter than a chunk. A two-word reminder embeds to a vector that sits near nothing in particular, and it stays unretrievable forever unless you attach its context — the note's title, the thread it arrived in, the date — before embedding it. Chunking a personal corpus is as much about growing small items as splitting large ones. Second, the axis that costs money is not the usual one. Any single person's corpus is small beside a web-scale index, so query latency is rarely what hurts; what you carry is one index per user across the whole install base. That argues for a shared store partitioned by owner rather than an index process each — and that owner key is also the boundary privacy and deletion will later be enforced against, so it is worth choosing deliberately instead of inheriting whatever the store defaults to. One more thing worth saying in an interview: personal content is dense with proper nouns, and embeddings blur names, addresses and account numbers together. Keyword matching alongside vectors earns its place here far more than it does over general prose.
What the new pieces do
- Personal Corpus Indexdata
- A searchable index over the user's own heterogeneous personal documents — messier and sparser than any curated public knowledge base.
- Retrieverservice
- Finds the actually-relevant personal documents for a query — a genuinely different retrieval challenge than searching curated web content.
Step 3 · Continuous, incremental ingestion
New content arrives constantly
New personal content — a note, an email, a document — arrives continuously, not in scheduled batches. Should the index require a full reprocessing job to absorb each new item?
New personal content arrives constantly, in an ongoing stream. Should the index require a full batch reprocessing for each addition?
Full reprocessing on every new item (or even on a periodic batch schedule) means new content isn't actually searchable until the next reprocessing cycle runs — for an assistant meant to reflect the user's CURRENT knowledge and notes, that lag is a real usability problem.
Incremental indexing means a note written five minutes ago is already retrievable — matching the reality that personal knowledge accumulates continuously, not in scheduled batches, and that a user reasonably expects the assistant to know about something they just wrote down.
A daily batch schedule still introduces a meaningful lag between writing something down and being able to ask the assistant about it — for a genuinely personal, actively-used tool, that gap is a real limitation.
Make the Personal Corpus Index update incrementally — each new item becomes searchable as it arrives, without waiting for a scheduled batch reprocessing job. This matches the reality that personal knowledge genuinely accumulates continuously.
Why this piece earns its place
Incremental sounds lighter than batch and is actually stricter, because it turns ingestion into a stream with all a stream's problems. Personal sources redeliver constantly: a mail sync replays a folder, a notes app fires a change event on every pause in typing, a file watcher fires twice on one save. So the write path needs an id derived from source plus item identity rather than a fresh one per event, an upsert instead of an insert, and a content hash so an unchanged redelivery costs nothing. Skip that and the same note lands three times; retrieval dutifully returns all three, and to the user the assistant simply seems to repeat itself. An edit then becomes a delete plus an insert, which means the removal path you need here is the one Step 7 needs — build it once and genuine forgetting is most of the way done. What incremental ingestion takes away is the moment when everything gets reprocessed, and you still need one: a chunking change or a new extraction rule has to reach content already indexed. That backfill runs beside the live stream, against a versioned index, so nothing ever queries a half-migrated corpus.
Step 4 · Disambiguate using personal context — or ask
"The contract" could mean several things
A user asks "what did the lawyer say about the contract" — but they've discussed several contracts over the years. How should the assistant resolve which one is actually meant?
A query could plausibly refer to any of several things in the user's history. How should the assistant handle this?
Silently picking the top-ranked match when the query is genuinely ambiguous risks confidently answering about the WRONG thing — for personal data specifically, being confidently wrong about someone's own life and documents is uniquely unsettling and damages trust fast.
Recent context often resolves ambiguity naturally (if the user was just discussing a specific contract, "the contract" likely means that one) — but when it genuinely doesn't, asking rather than guessing avoids confidently answering about the wrong referent, which is a far worse outcome than a brief clarifying question.
Requiring exhaustive specification for every query defeats the point of a natural, conversational personal assistant — the goal is resolving reasonable ambiguity intelligently, not pushing all disambiguation burden onto the user by default.
Add a Disambiguator that uses recent conversation context to resolve which specific personal referent a query means, and — when genuinely ambiguous even with that context — asks a clarifying question rather than guessing and risking a confidently wrong answer about the user's own life.
Why this piece earns its place
What this component produces is a rewritten query, and that is why it stands in front of the retriever rather than behind it. The sentence the user typed carries almost no retrievable signal — the words that identify which contract are in the last few turns of the conversation, not in the question. Resolving first turns that question into an explicit referent the retriever can filter on; resolving afterwards means paying for a search built on the ambiguous wording and then trying to repair the ranking it gave you. The price of standing in front is that it has to decide on less evidence than a candidate set would have offered, so it needs a cheap way to be wrong — which is exactly what asking is. Calibrate that, because asking is not free. A clarifying question spends a turn, and an assistant that asks every third time reads as less capable rather than more careful. Log two numbers together: how often you ask, and how often the referent the user picks is the one you would have chosen anyway. If those track each other, the threshold is too eager and you are taxing the user for certainty you already had. Nothing in your error rates will show you that ratio, because both outcomes look like a turn that went fine.
What the new pieces do
- Disambiguatorservice
- Uses recent conversation context to resolve which of several plausible personal referents a query means — and asks when it genuinely can't tell.
Step 5 · Cite the source, always
The user needs to verify, not just trust
An answer is generated from retrieved personal content. Should it just present the answer, or something more?
Every answer includes source attribution — exactly which personal document or note it was grounded in, so the user can verify the answer, find the original context, or catch a retrieval mistake. For an assistant reasoning about someone's own life and documents, being able to trace an answer back to its actual source is what makes the system trustworthy rather than a confident black box.
Why this piece earns its place
Attribution is a link, not a label, and the engineering is in making the link survive. It needs a durable item id that outlives re-ingestion, a rename and a move between folders — the same deterministic id the incremental write path already depends on. Mint fresh ids per ingest run instead and every citation older than the last sync quietly points at nothing, with no error anywhere to tell you. Point at the retrieved span too, not just the file: a forty-page scan offered as a citation is technically accurate and practically useless, and it is the span, not the document, the answer actually rests on. Citations also hand you an evaluation signal, as long as you name it honestly. Someone who opens a citation and then immediately rephrases the question has told you that answer was wrong — but only on queries where something was retrieved, shown and clicked. That is a precision signal, not a coverage one: the queries where nothing relevant surfaced and the user shrugged and gave up emit no event at all, and that silence is the population you most need to see. A single-user assistant has no labeled eval set to fall back on, so the only way to catch the silent misses is to keep a handful of questions you know the corpus can answer and re-run them after every chunking or extraction change.
Step 6 · Privacy as the central architectural constraint
This is uniquely sensitive data
Personal notes, journals, and documents are about as sensitive as data gets. Should this system handle privacy the same way a typical product handles any other sensitive-but-routine data?
How central should privacy be in this system's architecture, compared to a typical product handling sensitive data?
Standard measures are necessary but arguably not sufficient here — personal journals, financial documents, and private notes are about as sensitive as data gets, and this system's entire value proposition depends on users trusting it with exactly that kind of content.
Given how uniquely sensitive this specific corpus is, privacy has to shape core architectural decisions (not just be a compliance checkbox) — including seriously considering HOW MUCH of a user's private content should ever leave the system boundary for third-party model inference, and minimizing that exposure deliberately.
Relying solely on device security ignores that the SYSTEM itself (ingestion, indexing, any server-side processing) is also a place personal data lives and could be exposed — platform-level privacy engineering is still necessary regardless of device security.
Build a Privacy Layer that makes privacy a primary architectural driver: encryption at rest for the entire personal corpus, and deliberate minimization of what content leaves the system boundary for any third-party processing — redacting the most sensitive fields where feasible, or preferring more private inference options for the most sensitive parts of the corpus.
Why this piece earns its place
Encryption at rest is the half that is easy to say. The half that decides the design is that every derived artifact is a shadow of the plaintext and has to be covered by the same promise. An embedding is not anonymous, index metadata — titles, dates, participants — is usually stored in the clear precisely so it can be filtered on, and cached summaries, prompt logs and crash dumps all hold real content. Encrypt the documents and leave a searchable copy of them next door and you have bought very little. Then name the key-management fork out loud, because it sets the ceiling on everything else. If the user holds the key, no server-side component can read their content, so ingestion, ranking and any cross-document work moves client-side and the feature set shrinks to what a phone can run. If you hold it, you can build anything and your breach surface is every user at once. Both are defensible; switching later means re-encrypting every corpus you have, so it is not a decision to defer. And note what minimization does not buy — the slice you do send has still left the boundary, and retention terms are policy, not architecture.
What the new pieces do
- Privacy Layerservice
- Encrypts personal data at rest and minimizes what leaves the system for any third-party processing — the central architectural constraint of this whole design.
Step 7 · "Forget this" should mean forgotten
Genuine deletion, not hiding
A user asks the assistant to forget a specific note. Should that just hide it from a default view, or something more thorough?
Build a Deletion Handler that removes a requested item genuinely — from the retrieval index AND from any summary, cached derivation, or memory that referenced its content — so a future query can't surface it, directly or indirectly. Marking something merely "hidden" while its content still lives in the index or a derived summary doesn't honor what the user actually asked for.
Why this piece earns its place
Deletion is only tractable if you wrote down the lineage before anyone asked. You cannot find which summaries a note touched by searching them — its text is not in them, only its influence — so every derived artifact has to carry the source ids that fed it, recorded at the moment it was derived. Without those edges the only honest way to honor a request is to regenerate every derivation from scratch, which is an expensive way to learn you should have kept a graph. Two places the promise gets uncomfortable, and both are better said than skipped. Backups are immutable on purpose, so deletion in practice means removed from live systems and expired from backups on a stated retention clock — name the clock rather than imply instant erasure. And anything the content already fed into personalization or fine-tuning is not removable by deleting a row, which is a reason to keep personal content out of training paths in the first place. Finally, make it checkable: the test for a deletion is a query that used to surface the item, re-run afterwards and asserted empty. Automate it, because a deletion path nobody exercises rots quietly.
What the new pieces do
- Deletion Handlerservice
- Removes a deleted item from the index AND any derived summary or memory — "forget this" means actually forgotten, not just hidden.
Step 8 · The sharp edges
Facts change, and not everything is text
Two real complications: the user's own notes can genuinely contradict each other as facts change over time (an old address, a since-changed plan), and a meaningful share of personal content — scanned documents, photos with text — isn't plain text to begin with.
For facts that change over time, prefer recency when retrieved content genuinely conflicts — "my address is X" from three years ago shouldn't be presented as equally valid alongside "my address is Y" from last month; the retriever and answer generation should weight more recent, contradicting information appropriately rather than surfacing whichever matches the query best regardless of when it was written. For multi-modal content, run OCR/vision processing at ingestion time to extract text from scanned documents and images before they can be meaningfully indexed and retrieved alongside plain-text notes.
Why this piece earns its place
Both edges come down to something the system has to record at ingestion rather than decide at query time. Recency needs a clock, and a personal corpus carries at least three that disagree: when the content was written, when the file was last modified, and when it reached your index. A lease signed years ago and scanned this morning is the newest item by ingest time and one of the oldest by content time, so ranking on the wrong clock inverts the answer. Extract an event date from the content where you can, keep all three separately, and be explicit about which one recency means. OCR fails in the other direction — quietly. A misread digit in an account number produces a chunk that indexes cleanly, never matches the query that wanted it, and can still be quoted back as fact. So carry the extraction confidence into the index, keep the page image beside the text so a citation shows the original rather than the transcription, and treat a low-confidence span as retrievable but not quotable. OCR is also the one ingestion step too slow for the write path, which carves an exception into the immediacy Step 3 promises: a scan lands indexed on what you already know — file name, source, arrival time — and becomes retrievable by its contents only when extraction finishes. Show that state rather than letting a freshly scanned document look simply missing.
You did it
You just designed a personal knowledge assistant.
Everything you assembled, in order
- A personal corpus grows unbounded over years — dumping everything into a context window per query doesn't scale, in cost or eventually in hard limits.
- Retrieval has to handle genuine format heterogeneity and noise, since personal content is far messier than a curated knowledge base.
- Incremental ingestion makes new content searchable immediately, matching how personal knowledge actually accumulates.
- The assistant disambiguates using recent conversation context, and asks rather than guesses when genuinely uncertain — a wrong guess about someone's own life is uniquely trust-damaging.
- Every answer cites its source document, letting the user verify rather than just trust a confident-sounding claim about their own history.
- Privacy is a primary architectural driver here, not a parallel concern — encryption at rest and deliberate minimization of third-party exposure.
- A deletion request genuinely removes content from the index and every derived summary, not just from a default view.
- Recency-aware retrieval handles contradicting facts that changed over time, and OCR/vision processing brings non-text personal content into the retrievable corpus.