The model is not the product. The harness is.
Everyone argues about which model is smartest. The thing that decides whether an AI agent can be trusted with real work is almost never the model — it's the harness: the machinery around it that lets it act, see the result of its action, and be stopped from doing the wrong thing. A model on its own is text-in, text-out. It cannot open your mailbox, run its own code, remember last Tuesday, or notice it was wrong. Below: the same request, to the same model, twice.
Honest-AI note. This page makes no live model call. The left pane is scripted from an actual first-draft answer. The right pane is not scripted where it counts: the two patterns are compiled and really run against the 24 subject lines below, in your browser — "0 of 24" and "24 of 24" are computed, not typed. Break the fix in the source and the numbers on this page change with it. The incidents are the builder's own engineering failures, caught by the loop — not anonymised client stories.
The sample: 24 e-mail subject lines, underscore-delimited the way forwarded mail arrives — click to inspect
| Chat window | Harnessed agent | |
|---|---|---|
| Your data | You paste it in | It fetches, opens, parses — under permissions |
| Its code | "Looks right" ships | Run, failed, fixed, locked with tests |
| Its errors | Confidently wrong | Exposed by execution, same minute |
| Memory | Gone when the tab closes | Corrections become permanent rules |
| Boundaries | A polite system prompt | Enforced in code: review gates, allowlists, calendars |
01Plausible is not true
The broken pattern reads correct — any engineer would nod at it in review. It is
structurally incapable of matching, because \b needs a boundary that underscore-delimited
text never provides. The act–observe–fix loop is not a nicety; it is the difference between
plausible and true.
02Ask the system, not the file
A safety rule with a tidy explanatory comment was silently void — ignore files don't support end-of-line comments, so dozens of confidential documents sat one command away from an upload. What caught it was a dry run against git itself, and the fix ships with a test that was deliberately broken once to prove it turns red.
03The product needs its own harness
The finished system calls a model internally — inside a scaffold we built: structured facts in, parseable output or discard, wrong-language output replaced by a plain template, delivery that still happens if the model call fails, and a send button that belongs to deterministic code. Smart is rented. Safe is built.
Plain-language key (harness, word boundary, dry run, self-test, degrade honestly)
- Harness
- Everything around the model that lets it act and be checked: tools, permissions, execution, memory, tests, gates.
- Word boundary
- A regex marker for the edge between a "word" character and a non-word one. Underscore counts as a word character — which is why the first pattern could never fire.
- Dry run
- Performing an action in a mode that shows what would happen, without the consequence. Here: staging files and asking git what is about to be committed.
- Self-test
- A check that runs with the system and pins a behaviour. If reverting the fix doesn't turn it red, it isn't one.
- Degrade honestly
- When the model fails, the system does something plainer and says so — rather than failing silently, or not at all.