Vibe Engines
YouTube

INVESTIGATION 01 · RETRIEVAL, UNDER THE MICROSCOPE

The right match.
The wrong answer.

A support search finds a beautifully matching refund policy. It belongs to another customer. Fix the retrieval pipeline, then prove what your fix actually changed.

12 synthetic documents9 practice cases3 fresh casesRetrieval practice 45–75 min · answer support 20–30 min

A local retrieval experiment. Results come from the passages below. Your progress and notes stay in this browser.

01 · THE BRIEF

A citation can still be wrong.

You are building support search for the fictional Harbor workspace. Its collection contains current help articles, one retired refund policy, and two private Cedar contract documents. Harbor users should receive only current Harbor passages.

Your job is to return enough evidence to answer a question without borrowing another customer’s terms or an expired policy. The baseline searches everything. Its strong matches make the mistake easy to miss.

What you will leave with

  • A reproducible comparison of chunking and retrieval settings.
  • A trace showing exactly which passages caused a failure.
  • A design decision with a trade-off and an unresolved limitation.
  1. PredictName the failure you expect to change.
  2. Change one thingKeep a comparison you can explain.
  3. InspectRead the actual returned passages.
  4. DefendTry fresh cases and export the evidence.

02 · THE WORKBENCH

Make a call. Run the cases.

Start with the refund case. Inspect the baseline’s private and retired sources, then choose your first change. Every saved run retains its own settings.

Choose how to work
Start with a worked example

The refund baseline returns two private passages and one retired passage across the practice set. First enforce Harbor permissions. Run the same questions and inspect refund-current and refund-old; permissions do not make a retired Harbor policy current. Then test the freshness filter separately.

Your independent task: Keep the filters fixed and compare top-k 2 with top-k 3. Cite a question where evidence coverage or context size changes.

Predict first. Compare one change, inspect an actual passage/request/event, then explain a condition where the repair fails. The changed-case check removes these hints.

Your next experiment

Advanced chunking controls
Chunk boundaries
Source boundaries · before ranking

Compared settings: No setting differences. Both columns use the same case set.

Measured results

Practice cases

The baseline is already measured. Run your experiment to add a comparison.

Reference and selected saved run on the same cases
MeasureReferenceSelected run
Source recall ↓Answerable cases with the labeled document7/8 —
Complete evidence ↓Every required phrase is in valid returned passages6/8 —
Private passages ↓Returned across all cases · aim for zero2—
Archived passages ↓Returned across all cases · aim for zero1—
Unsupported cases with contextInspect these; a match does not establish support0/1 —
Average contextApproximate tokens per question~69—
Index sizeAll chunks, before access/freshness filtering 27 chunks
~965 tokens
—

Tokens use ceil(characters ÷ 4), an estimate. Source recall and evidence coverage use this fixture’s authored relevance labels. These counts do not measure generated-answer quality.

Keep this observation with your decision

Choose a measured passage, workload, or event. The attachment keeps its run and version when newer experiments are saved. Each record holds up to six observations; existing attachments are never evicted automatically.

Open decision record →

Read the evidence, passage by passage.

A high average can hide one damaging case. Select a question and compare the sources behind the numbers.

ORIGINAL BASELINE · HARBOR USER

Annual subscription refund window

The retired policy and the other tenant repeat the query. A strong text match does not establish permission or freshness.

Labeled evidenceSource: refund-current. Required: 30 calendar days.
#1 · Annual subscription refund window — Cedar contractBM25 9.62
CedarcurrentCharacters 0–180 · end exclusive
Annual subscription refund window: Cedar annual subscription customers have 120 calendar days to request a refund. This annual subscription refund window is a private Cedar contrac

Private to Cedar. This passage must not reach a Harbor answer.

#2 · Annual subscription refund window — retired policyBM25 9.00
HarborarchivedCharacters 180–250 · end exclusive
se its refund window for a current annual subscription refund request.

Archived. Its matching words do not make the policy current.

Compare reference passages for this question
cedar-refund · 0–180 · reference
Annual subscription refund window: Cedar annual subscription customers have 120 calendar days to request a refund. This annual subscription refund window is a private Cedar contrac
refund-old · 180–250 · reference
se its refund window for a current annual subscription refund request.

03 · CHANGE THE QUESTION

Does your decision travel?

Choose your settings before opening three fresh cases. They include a private commercial term, a new account-recovery question, and a plausible capability the help center never documents.

Keep the configuration fixed for the first attempt. Explain a miss before adjusting the threshold. If you tune against these cases afterward, treat them as practice and use another dataset for a fresh evaluation.

Run the practice set first. These are public practice fixtures and a self-check of transfer, rather than a credential or a private exam.

04 · LEAVE SOMETHING INSPECTABLE

Your decision, with receipts.

0/4 checkpoints recorded. The source-safety check is calculated; the written explanation remains yours to defend.

  1. Prediction and experiment Still to do

    A prediction is recorded alongside a reproducible run.

  2. Source safety measured Still to do

    The latest practice run enables both source filters and returns zero private or archived passages.

  3. Fresh cases attempted Still to do

    A saved configuration has been evaluated on the three transfer cases. Inspect failures before drawing a conclusion.

  4. Decision explained Still to do

    Your decision, trade-off, and remaining limitation are written down. This reflection is self-reported.

Progress saves in this browser.

Exports include the latest 20 runs, predictions, settings, and notes. A completed checklist records work; it does not establish professional competence or production readiness.

THE MECHANISM

What the experiment is measuring.

Ranking and source validity answer different questions

BM25 ranks passages by matching terms, their rarity in the eligible collection, their frequency in a passage, and passage length. Here the constants are k1 = 1.2 and b = 0.75. The tokenizer lowercases text, keeps hyphenated identifiers together, and removes a small English stop-word list. Titles are displayed as provenance; only passage text is scored.

The score is neither a probability nor a guarantee that a source can answer the question. The Cedar refund document can be an excellent lexical match and still be forbidden. Permission and current-status filters run before ranking, so filtered passages never occupy the shortlist. Recomputing collection statistics after filtering also changes scores.

Source recall is weaker than answer evidence

Every answerable question has one authored relevant document. Source recall counts questions where at least one passage from that current Harbor document appears in top-k. Complete evidence requires every labeled phrase to appear intact in at least one valid returned passage; the phrases can come from different passages of that document.

A cut through a required phrase can fail this exact-text check even when a human could reconstruct it. This deliberately inspectable metric is narrower than semantic answer correctness. An unsupported case has no labeled source; a nonempty result is a reason to inspect its relevance, rather than proof of a generated hallucination.

Complexity and trade-offs

The browser builds term counts for each eligible passage and scans them for every question. For this tiny collection that makes the mechanism easy to inspect. A production search engine uses an inverted index and different serving machinery; these runs do not estimate its throughput or latency.

Overlap stores text more than once. A larger shortlist supplies more potential evidence and increases context. Whole paragraphs preserve authored boundaries but produce variable-sized passages. Higher score thresholds reduce returned context and can also reject useful evidence. Use the measured cases to defend the balance you choose.

Common mistakes

  • Equating a matching source with permission to use it.
  • Looking only at average recall while one private passage slips through.
  • Changing chunk size, top-k, and filtering together, then attributing the outcome to one change.
  • Treating approximate token counts as a measured model bill.
  • Tuning until every public fixture passes, then calling the result a general guarantee.

Answer generation, citation faithfulness, retrieval with embeddings, adversarial document instructions, and production identity enforcement need their own implementations and evaluations. This investigation isolates the retrieval decisions that should happen before a prompt is assembled.

Inspect the complete synthetic dataset · 12 documents

The product, policies, companies, and commercial terms below are fictional teaching fixtures. Dataset version: harbor-helpdesk-2026-11-13.1.

Annual subscription refunds

refund-current · Harbor · current

Harbor annual subscriptions qualify for a refund within 30 calendar days of the first payment. The workspace owner submits the request from Billing settings. Renewals are excluded from this first-payment policy. The refund window starts at the first successful payment, rather than when an invitation is accepted. Support records the payment identifier and request date before deciding whether the request qualifies.

Annual subscription refund window — retired policy

refund-old · Harbor · archived

Annual subscription refund window: annual subscription customers can request a refund within 90 calendar days. This retired annual subscription refund policy was replaced. Do not use its refund window for a current annual subscription refund request.

Canceling an automatic renewal

renewal · Harbor · current

To cancel a subscription renewal, the workspace owner opens Billing settings and chooses Disable renewal. This action prevents the next payment. Canceling a subscription renewal leaves access available until the end of the paid term. Cancellation does not immediately delete workspace documents or create a refund request.

Resolve error E-404

error-e404 · Harbor · current

Error E-404 means the search collection is missing. A workspace administrator should create the collection, wait for indexing to finish, and retry the search. Retrying a missing collection without creating it cannot repair error E-404. A deleted collection must be recreated from its source documents. This product error is distinct from an HTTP page returning status 404.

Export audit logs

audit-export · Harbor · current

To export audit logs, open the Audit page, choose a date range, and select Export CSV. Only a workspace administrator can export audit logs. Audit logs are retained for 90 days. A date range outside that retention period returns no records. CSV exports include event time, actor identifier, action, and document identifier.

Invite a workspace member

invitations · Harbor · current

Only a workspace administrator can invite a member. Open Members, enter the email address, choose a role, and send the invitation. An invitation expires after seven days. A workspace viewer can read shared documents but cannot invite members or change billing. Resending an invitation starts a new seven-day expiry window.

EU workspace data location

region · Harbor · current

Harbor EU workspace data is stored in Frankfurt. The region is chosen when the workspace is created. Support cannot move an existing workspace between regions. A customer who needs another region creates a new workspace and imports its documents. This fictional example describes the application database, not any model provider processing location.

Deleted documents and the search index

deletion · Harbor · current

Deleted documents leave the search index within 15 minutes. A tombstone is written immediately, so the retrieval gateway must exclude tombstoned documents while the index catches up. Deleting the source file without writing its tombstone leaves stale passages searchable. The ingestion worker removes the vector and stored passage using the same document identifier.

Retry a rate limited API request

rate-limits · Harbor · current

A rate limited API request returns HTTP 429 and a Retry-After header. Wait for the interval in Retry-After before retrying. Add jitter when several workers retry together. An API key belongs to one workspace. Its rate limit is shared by that workspace, so starting more workers does not create additional quota.

Account recovery after losing an authenticator

recovery · Harbor · current

After losing an authenticator, use a recovery code to recover the account. If no recovery code remains, an administrator starts identity verification. The manual recovery review takes up to 24 hours. Support never asks for a password or a current one-time code in a public ticket. Recovery codes are single-use and must be stored outside the affected device.

Annual subscription refund window — Cedar contract

cedar-refund · Cedar · current

Annual subscription refund window: Cedar annual subscription customers have 120 calendar days to request a refund. This annual subscription refund window is a private Cedar contract term. It must never appear in a Harbor customer answer.

Cedar private renewal discount

cedar-discount · Cedar · current

The Cedar private renewal discount is 35 percent. The Cedar private renewal discount belongs only to Cedar contract negotiations. Harbor users have no permission to read the Cedar private renewal discount.

Why start with keyword search instead of embeddings?

It provides a transparent baseline whose ranking can be reproduced in the browser. Source permissions, freshness, chunk boundaries, and evaluation remain necessary when a vector retriever is introduced. Compare a future embedding implementation on the same labeled cases; its gains must come from real computed results.

Why can a higher threshold still return unsupported context?

Keyword overlap is not a support judgment. A question about encryption keys can match a document about API keys. Inspect the terms, introduce answerability evaluation, and preserve a way to decline. A threshold tuned to this collection will not transfer unchanged to every retriever or corpus.

Can I use the report as portfolio evidence?

Yes, as a clearly labeled learning investigation. Include your reasoning, the exported settings and cases, and what you would test next. A reviewer can rerun the configuration and challenge your conclusions. Keep the synthetic-data scope visible.

Primary sources and the next layer

The fixture data and relevance labels are authored for this investigation. The references below explain the underlying methods; they do not supply the fictional policies.

BM25 parameters and similarity scoring · Elasticsearch reference →Retrieval-Augmented Generation · Lewis et al., 2020 →

NEXT LAYER · RETRIEVAL IS ONLY ONE CHECK

Does the evidence support the answer?

Six authored drafts test multi-source answers, contradictions, ambiguity, missing evidence, and paraphrases. Decide which claims the sources support. These are fixed teaching cases, not generated answers or a live model benchmark.

Worked example: reject an unsupported extension

A source says “renewal cancellation prevents the next payment.” A draft adds “and deletes your account.” The first clause is supported; the deletion claim is absent. Label the draft partially supported, cite the actual passage, and reject or correct the draft before responding. Citation presence alone is insufficient.

annual subscription reseller refund 30 days first payment

A first-payment annual reseller purchase can fall within the 30-day refund window. Contact the reseller to process that purchase refund.
How well is this draft supported?
Response policy for this draft
Sources needed to justify your decision (none for absent evidence)

Inspect the actual sources

Uses the current main-workbench settings only when you press Run. Supplemental challenge documents are indexed in this exercise; the original investigation fixtures stay unchanged.

Read the full challenge collection

refund-current · Harbor · current

Harbor annual subscriptions qualify for a refund within 30 calendar days of the first payment. The workspace owner submits the request from Billing settings. Renewals are excluded from this first-payment policy. The refund window starts at the first successful payment, rather than when an invitation is accepted. Support records the payment identifier and request date before deciding whether the request qualifies.

refund-old · Harbor · archived

Annual subscription refund window: annual subscription customers can request a refund within 90 calendar days. This retired annual subscription refund policy was replaced. Do not use its refund window for a current annual subscription refund request.

renewal · Harbor · current

To cancel a subscription renewal, the workspace owner opens Billing settings and chooses Disable renewal. This action prevents the next payment. Canceling a subscription renewal leaves access available until the end of the paid term. Cancellation does not immediately delete workspace documents or create a refund request.

error-e404 · Harbor · current

Error E-404 means the search collection is missing. A workspace administrator should create the collection, wait for indexing to finish, and retry the search. Retrying a missing collection without creating it cannot repair error E-404. A deleted collection must be recreated from its source documents. This product error is distinct from an HTTP page returning status 404.

audit-export · Harbor · current

To export audit logs, open the Audit page, choose a date range, and select Export CSV. Only a workspace administrator can export audit logs. Audit logs are retained for 90 days. A date range outside that retention period returns no records. CSV exports include event time, actor identifier, action, and document identifier.

invitations · Harbor · current

Only a workspace administrator can invite a member. Open Members, enter the email address, choose a role, and send the invitation. An invitation expires after seven days. A workspace viewer can read shared documents but cannot invite members or change billing. Resending an invitation starts a new seven-day expiry window.

region · Harbor · current

Harbor EU workspace data is stored in Frankfurt. The region is chosen when the workspace is created. Support cannot move an existing workspace between regions. A customer who needs another region creates a new workspace and imports its documents. This fictional example describes the application database, not any model provider processing location.

deletion · Harbor · current

Deleted documents leave the search index within 15 minutes. A tombstone is written immediately, so the retrieval gateway must exclude tombstoned documents while the index catches up. Deleting the source file without writing its tombstone leaves stale passages searchable. The ingestion worker removes the vector and stored passage using the same document identifier.

rate-limits · Harbor · current

A rate limited API request returns HTTP 429 and a Retry-After header. Wait for the interval in Retry-After before retrying. Add jitter when several workers retry together. An API key belongs to one workspace. Its rate limit is shared by that workspace, so starting more workers does not create additional quota.

recovery · Harbor · current

After losing an authenticator, use a recovery code to recover the account. If no recovery code remains, an administrator starts identity verification. The manual recovery review takes up to 24 hours. Support never asks for a password or a current one-time code in a public ticket. Recovery codes are single-use and must be stored outside the affected device.

cedar-refund · Cedar · current

Annual subscription refund window: Cedar annual subscription customers have 120 calendar days to request a refund. This annual subscription refund window is a private Cedar contract term. It must never appear in a Harbor customer answer.

cedar-discount · Cedar · current

The Cedar private renewal discount is 35 percent. The Cedar private renewal discount belongs only to Cedar contract negotiations. Harbor users have no permission to read the Cedar private renewal discount.

reseller-exception · Harbor · current

Harbor annual subscriptions bought through a reseller use the first-payment 30-day refund window. The buyer must contact the reseller to process the refund; Harbor Billing cannot process a reseller purchase refund.

audit-retention-plan · Harbor · current

Harbor audit logs are retained for 365 days. This current policy page does not identify the applicable account plan or policy version.

Version answer-support-2026-11-17.1. Your source choices and response-policy decisions save locally; feedback is checked against authored keys.

CARRY THE SAME QUESTION FORWARD

Understand it. Implement it. Evaluate it.

Use these results in the retrieval decision checkpoint →