Ask my portfolio: a chatbot that answers questions about my work

The pitch

/ask is a retrieval-augmented chat agent that answers questions about the rest of this portfolio. You give it a prompt like "what's the hardest production system Owen has shipped?" and it streams back an answer grounded in the case studies and writing on this site, with numbered citations that link to the source pages. It's the live AI demo, and it doubles as a small production system I can point at.

This case study is the writeup for it. The same things I'd want to see in any other AI engineer's portfolio (evals, guardrails, cost, latency) are below.

If you read one section, read the evals. An LLM feature without an evaluation harness is a vibe, not a system. This one has a golden-set recall gate: CI fails the pull request if retrieval quality drops (when the embedding key is configured in CI). That harness is the difference between a demo and something I'd put my name on.

Context and constraints

The constraint that shaped the design was this is a portfolio. That meant:

• It must not lie about me. Hallucinations here are not amusing failures. They're resume fraud. I'd rather it refuse than invent.
• It must be cheap to run. I'd rather it stay up for a year than be flashy for a week. Budget: a hard cap of 1,500 questions a month, which at the $0.005-per-question target is at most about $7.50.
• Cold start has to be invisible. Target: first-token latency under two seconds, even on a freshly woken serverless function.
• No vector database. The corpus is about a dozen documents. Pulling in pgvector or Pinecone for that is theatre.

What I built

A small system, on purpose:

1. At build time, a Node script walks content/ (case studies, blog posts, a markdown mirror of the resume, an about.md), chunks each file by section heading with a one-paragraph overlap, embeds each chunk with Voyage AI's voyage-3.5 (page title and heading path prepended, so a chunk that never says "CoStar" still carries it), and writes lib/rag/chunks.json: about 100 chunks (95 today), roughly 2MB once embedded. The chunk text is committed; the embeddings are generated by the prebuild step at deploy time, so the repo carries the corpus without the vectors.
2. At runtime, /api/ask is a Node-runtime route handler that loads chunks.json once on cold start, embeds the user's query, and takes the 12 nearest chunks by cosine similarity (at most two per page, so the FAQ can't crowd out a case study). A reranker, Voyage's rerank-2.5-lite, then scores those 12 against the question; the top 6 go into a system prompt with persona + guardrails + retrieved context.
3. The completion streams from Claude Haiku 4.5 via the Anthropic SDK. The UI renders tokens as they arrive, with numbered citations and a list of source links under each answer.

That's it. No vector DB, no LangChain, no agent framework. The core of the retrieval is a few lines of TypeScript:

export async function retrieve(question: string, queryEmbedding: number[], chunks: Chunk[], k: number) {
  const candidates = topK(queryEmbedding, chunks, CANDIDATES); // cosine, in memory
  const order = await rerank(question, candidates.map((c) => contextualText(c.chunk)));
  return order.slice(0, k).map(({ index, score }) => ({ chunk: candidates[index]!.chunk, score }));
}

Key decisions and tradeoffs

Guardrails

The system prompt establishes three rules the agent has to follow:

const SYSTEM = `You are an assistant answering questions about Owen Richards' work,
drawing only from the provided excerpts.
 
1. Never invent employers, dates, or metrics.
2. If the excerpts don't cover the question, say so and suggest a topic you do know.
3. Speak about Owen in the third person. You are not him.
...`;

The third rule is the one that took the most iteration. Early versions of the prompt let the model speak as Owen ("I led a $1B initiative…"), and the output read as a roleplay rather than a tool. Forcing third-person broke the spell and made the citations feel like citations, not personal claims.

Evaluation

I keep a small golden set of twenty-seven handwritten questions in tests/rag/golden.jsonl: twenty-two with an expected source, plus five off-corpus probes ("who won the 2026 World Cup?") whose best reranker score must stay below the refusal threshold, the same condition that makes the live agent refuse. A few broad questions accept more than one page as correct: "what does Owen use day-to-day?" is answered by the FAQ and the résumé as much as by a case study. CI runs pnpm rag:eval on every PR when the embedding key is configured; the check fails if recall@5 falls below 0.9 or any probe retrieves. Cost per question is reviewed manually against a $0.005 budget. It isn't a CI gate.

Those are targets. Every run of the harness writes its results to lib/rag/eval-results.json, and this scorecard renders whatever the last recorded run says (or admits when there isn't one):

The targets are what pnpm rag:eval checks against, and the harness, the golden set and the gate all live alongside the code. The scorecard above is the latest recorded run, so it moves when the corpus or the retrieval does. Two changes helped retrieval most: switching from fixed-size to markdown-heading chunking, and capping how many chunks one page can contribute.

What I'd do differently

I'd build the eval harness before the agent, not after. I spent two evenings tuning the prompt by vibes and got it about 70% of the way; the golden set found three subtle regressions in an hour. Eval-first is the lesson that keeps re-teaching me.

I'd also write the refusal prompts before the answer prompts. The default failure mode of an LLM is to be helpful and confident. Making refusal feel native required exemplars in the system prompt. I'd put that work upfront next time.