Here is an uncomfortable fact about building with LLMs: your QA process, the one your company spent years refining, cannot test your AI features.

Traditional QA assumes determinism. A tester clicks the button, the same thing happens every time, the case passes or fails. LLM features break that contract. The same input produces different outputs on different days, across model updates, after a one-word prompt change you made for an unrelated reason. There is no "expected result" to assert against, only a distribution of possible results, some of them wrong in ways you have not imagined yet.

Most teams respond in one of two ways. They ship on vibes: a few people poke at the feature, it seems fine, it goes out. Or they freeze: every change requires a manual review so heavy that iteration slows to a crawl. The first approach ships regressions to users. The second one ships nothing.

At Mindlid we took a third path, and I have been calling it evals-as-QA: treat evaluations not as a research dashboard or a vanity benchmark, but as the QA department itself. Nothing ships unless the evals pass. That is the whole policy.

What an eval gate actually is

An eval, at its simplest, is a structured test for a non-deterministic system. You define a set of inputs, run them through your AI feature, and score the outputs against criteria, with another model as judge, with rules, or with humans for samples. So far, standard stuff.

The shift that matters is positional. Most teams that have evals run them descriptively: a dashboard someone checks weekly, a benchmark in a slide. We run them as ship gates: a release candidate either passes the eval suite or it does not deploy. No override by enthusiasm, no "it looked good in the demo." The eval suite holds the authority a QA sign-off used to hold.

This sounds rigid. It is the opposite. The gate is what makes speed safe.

What we gate on

Four categories, in priority order:

Safety. We build a therapy-inspired, non-clinical emotional wellness product, so this is non-negotiable: refusal behavior, crisis-adjacent inputs routing to crisis resources, staying inside non-clinical language. Safety evals are the reason we can iterate quickly on everything else; the floor is always verified, never assumed.

Reliability. Does the feature do what it claims, for real user-shaped inputs, not just the happy path? For us that includes whether retrieved context actually matches what the user needs. Memory is core to our product, and we measure it: combining text, audio, and visual embeddings for conversational memory lifted top-1 recall by 20 percent, a result cited in Google's Gemini Embedding 2 launch announcement.

Regression. The quiet killer. You improve one thing and silently degrade three others, because in prompt-land everything touches everything. Every change runs against the full suite, not just the cases it was meant to improve. Most of the value of evals-as-QA is here, in the failures you never see because they never ship.

Experience. Tone, format, length discipline. The difference between an AI feature people tolerate and one they return to usually lives in this layer, and it is the layer most teams never test at all.

Evals are the definition of done

In my previous piece, Stop prompting AI. Start defining done, I argued that the leverage in applied AI has moved from prompts to checks: a goal-based autonomous loop is only as good as the definition of "done" it runs against.

Evals-as-QA is what that definition looks like in production. The eval suite is "done," stated formally. Which means the loop and the gate compose into one system: generate a candidate, run it against the evals, fail, adjust, run again, and only exit the loop on a PASS. Call it eval-gated loop engineering. The prompt iteration that used to be a human staring at outputs and guessing becomes a loop that converges on a measurable target.

Flowchart of the eval-gated loop: a candidate change enters an eval suite of four gates (safety, reliability, regression, experience); failures feed back into an adjust step, and only a full pass deploys.
The gate and the loop are one system: fail feeds back, only PASS deploys.

This composition is not a theoretical nicety. It is how we collapsed our iteration cycle. We built an internal workbench that runs exactly this loop, and it cut our prompt-iteration-to-deploy time from months to under one hour. Not because anyone typed faster, but because "is this safe to ship?" stopped being a meeting and became a gate that runs in minutes. The speed did not come from skipping checks. It came from automating them.

That is the part I would tattoo on the roadmap: teams think evals slow you down; gated evals are the only thing that lets you go fast without lying to yourself.

What I got wrong along the way

Principles are cheap, so here is what the practice taught us.

  • Evals rot. A suite that was rigorous in January is a rubber stamp by June, because your product changed and your users found new edges. We treat eval coverage like test coverage: every incident or weird user interaction becomes a candidate case. An eval suite is a garden, not a monument.
  • Passing too easily is a warning, not a victory. When everything sails through, the suite has stopped discriminating. The reflex to celebrate should be a reflex to tighten.
  • You can overfit to your own evals. A loop that optimizes against a fixed target will find its blind spots. Sampled human review never goes away; it is how the gate itself gets audited.
  • The judge needs judging. If a model scores your outputs, its rubric drifts with model updates too. Pin versions where you can, spot-check the judge where you cannot.

None of these problems is a reason to skip evals. They are the maintenance schedule of owning them.

Steal this

If you build LLM features and your release process still depends on someone eyeballing outputs, start here:

  1. Write down, in plain language, what "unacceptable output" means for your product. That sentence is your first safety eval.
  2. Collect 30 real, user-shaped inputs. Not synthetic perfection, the weird stuff.
  3. Gate one release on them. Just one. Feel the difference between "we think it's fine" and "it passed."
  4. Add a case every time reality surprises you.

I keep a free, copyable version of our framework structure here: Evals-as-QA Framework. No email wall, take it.

Your QA team is not obsolete. It just works for a different department now: the one that writes the gates instead of clicking the buttons.