Shipping RAG to production: notes from building /ask

Update, October 2026. /ask now embeds with Voyage AI's voyage-3.5 and reranks candidates with rerank-2.5-lite before the refusal check. The cost figures and the OpenAI client note below describe the original OpenAI version. Why the switch needed a reranker is in the case study.

The chatbot on /ask is small. About 100 chunks, one embedding model, one generation model, no vector database, no agent framework. It runs for under five dollars a month at portfolio traffic levels. Looking back at three weekends of building it, here's what I'd reuse and what I'd skip if I had to do it again — or if you were doing it for your own product.

Skip the vector DB until you can't.

The embedded chunks for /ask are about 2MB on disk. Cosine similarity across ~100 vectors is roughly 157K multiply-adds, which is well under a millisecond in Node. A round trip to a managed vector store costs tens of milliseconds on a good day and adds one more system to monitor on a bad day. The line in the sand for me is roughly "if your corpus fits comfortably in a Lambda's memory and changes less than once an hour, JSON-on-disk is correct." For most products that have an AI feature glued on the side, that's the starting point.

The exception: if your corpus is going to grow by an order of magnitude in six months — multi-tenant SaaS, user-uploaded documents, ingestion pipelines — design for the vector store up front. The migration is not the work; the rewrite of the retrieval contract usually is.

Chunk by structure, not by token count.

The biggest single win between v0 and v1 was switching from fixed-size token chunking (450 tokens with 80 token overlap) to chunking by markdown heading with a one-paragraph overlap. It was the change that helped retrieval most; I'll publish the recall numbers once a recorded eval run is committed.

The intuition: section headings are semantic anchors. "Key decisions and tradeoffs" is about one thing; a 450-token window that straddles the boundary between the "Approach" section and the "Tradeoffs" section is about two half-things, and the embedding for it ends up at the cosine center of nothing.

If your corpus is markdown, code, or anything with structure — chunk on the structure.

Build the eval harness first.

I built the eval harness for /ask after I had a working agent, which meant I tuned the prompt by vibes for two evenings and got it 70% of the way. The golden set I finally wrote (nineteen handwritten Q→expected-source pairs) found three subtle regressions in an hour, including one where the agent had silently started preferring the blog posts over the case studies on technical questions.

The cost of an eval set is roughly an hour. The cost of a hallucinated metric on a portfolio site is your credibility. Build it first.

Refusal is a feature, not a fallback.

The default failure mode of an LLM is to be helpful and confident. If retrieval doesn't give it good context, it'll cheerfully extrapolate from world knowledge. For an authoritative use case — a product spec, a portfolio, a docs site — that's worse than a refusal.

I learned to write the refusal prompts before the answer prompts. A specific refusal ("I don't have material on Owen's GPU optimization work, but I can speak to the production AI work he's shipped at CoStar or the RAG agent on this site") reads as trustworthy. A generic "sorry, I can't help with that" reads as broken.

The cost numbers are easy.

For a small RAG system, the math is:

• Embeddings (build time): negligible. About $0.01 to embed the entire /ask corpus.
• Embeddings (runtime, per query): ~$0.00002. Effectively free.
• Generation (per query): ~$0.002 on Claude Haiku 4.5 at 1k input + 300 output tokens.

The dominant cost on /ask is the rate limit not getting hit. At 10 questions per IP per hour, the worst plausible spike is in the cents-per-day range. Budget anxiety is not the bottleneck — engineering time is.

What I'd do differently next time.

• Write the system prompt's voice last. I spent an evening getting the assistant to "sound right" before I had retrieval working well. None of that work survived.
• Cache the embeddings client globally. The Vercel function cold start was eating 300ms of latency until I moved the new OpenAI() call out of the request handler.
• Make the citations clickable from day one. Inline [1] markers that don't link to the source case study look like noise. The signal is the navigation.

What I wouldn't do.

• LangChain or LlamaIndex for a corpus this small. The retrieval is twelve lines. The abstraction tax is not worth the reuse you get back.
• An agent loop. Tool-use and multi-step retrieval are great when you actually need them. For a Q→A flow over a documented corpus, a single shot of retrieval and a single generation is faster, cheaper, and easier to evaluate.
• A "knowledge graph" — at any scale where a knowledge graph helps, you've graduated past blog-post territory.

A retrieval-augmented agent over your own writing is the highest-leverage AI demo I know how to ship in a weekend. The hard part isn't the code; it's the discipline to keep it small.

— Owen