An eval harness for a one-person AI product

Nov 1, 2026

How I decide whether a prompt change ships in Everblue, an AI journaling app.

When you're the only engineer, it's tempting to change a prompt, try three examples and ship. I did that early on. It went fine until a companion reply "remembered" something the user never wrote. In a journaling app, that breaks trust immediately.

So every prompt change now goes through one harness.

1. Run the real pipeline

The harness calls the same production code, not a copy, on synthetic day payloads. No real user data is involved.

2. Grade with the production rubric

Voice (does it sound like the companion?), specificity (does it mention what actually happened?) and hallucination (does it say anything the day didn't contain?).

3. A separate fabrication suite

Recall questions where the honest answer is "I don't see that in your entries." The companion has to say so instead of inventing.

4. A hard bar

92%+ generation quality and 100% no-fabrication. It's a manual gate, not CI. I read every failure myself, because the failures teach me more than the score.

5. Free training data

Passing runs export a JSONL file I can use for fine-tuning later.

Guards at runtime, too

Evals catch problems before release; guards catch them live:

• A voice lint runs inside the generation retry loop.
• On a blank day, the companion says there's nothing recorded instead of making something up.
• A grounding pass can only remove unsupported claims, never add new ones.
• If a recalled memory contains words that can't be traced back to the user's own entries, the reply is re-rolled.

Porting it on-device

When I rebuilt the companion to run on Apple's on-device Foundation Models, I ported the Python grading bars to JavaScript guards and diffed the two for parity. The diff found 8 porting bugs that would have quietly weakened the guards.

What I deliberately didn't use

No vector database and no LLM framework. Each user's corpus is small and the call graph is linear, so SQLite with hybrid retrieval (BM25 + cosine + recency decay + MMR) was enough. I know roughly where that stops being true, and I'd switch then.

Takeaways

1. Run evals on the real pipeline, or you're testing a different system.
2. Make "no fabrication" its own suite with a 100% bar.
3. Pair offline evals with cheap runtime guards.
4. When you port, diff the guards. Bugs hide in translations.

I'm looking for an AI engineering role (remote, US). If evals are a first-class job on your team, let's talk.