The first reaction is usually to blame the model. The agent missed a file, changed the wrong part of a document, opened the wrong page, or produced a confident summary that nobody could safely use. Someone then says the model is not ready, the prompt is weak, or the agent needs a larger context window.
Sometimes that is true. More often, I see a simpler problem. The model was dropped into a messy job with no proper working setup around it.
That setup is what people are starting to call the harness. I do not use the word because it sounds fashionable. I use it because it gives a name to the part of AI agent work that decides whether the result survives contact with the office: context packaging, tool access, permissions, checks, logs, handoff rules, and the point where a person still has to say yes or no.
If the harness is weak, a strong model behaves like a clever intern with a laptop, no onboarding, and vague authority. It may do something useful once. It will not be dependable.
The failure pattern I wanted to pin down
I wrote this from the kind of work that looks simple until someone has to approve it. AI agents usually fail inside real workflows because the surrounding context, tools, permissions, checks, logs, approvals, and recovery paths are weak. The useful question was not which AI sounds clever, but whether the output survives the next handoff without quiet rework.
The agent workflow I used as the test bed
I used this as the work packet: A model-selection worksheet built around this question: AI agents usually fail inside real workflows because the surrounding context, tools, permissions, checks, logs, approvals, and recovery paths are weak. Before judging OpenAI Codex, ChatGPT, Claude, LangChain, Databricks, and AI agents, I wrote down the source material, the review owner, where the result had to land, and what would break if the output was wrong. A neat answer was not enough; the workflow had to become easier to inspect.
The harness parts I checked
| Check | What I watched | Failure signal |
|---|---|---|
| Input quality | Whether the source material was clear enough for the AI to use | The tool guessed missing context instead of asking for it |
| Human review | Whether a person could approve, edit, or reject the result quickly | The reviewer had to read everything again from scratch |
| Handoff | Whether the result could move into a document, table, ticket, or workflow | The next person had to reformat or reinterpret it |
| Repeatability | Whether the same pattern worked twice with different material | The first run looked good, the second run drifted |
The notes that separated action from demo
| Evidence | What I checked | Why it mattered |
|---|---|---|
| Input packet | One task brief, one source document or table, and the expected handoff format | A model ranking without input shape says little about real work |
| Output check | Whether the answer could become a document, table, ticket, or instruction without rebuilding | Most cleanup hides here |
| Failure log | The output looked polished, but the receiving person still had to rebuild the table, verify sources, or rewrite the handoff note. | The failed case marks the real boundary |
Where a good answer still stalled
The first answer was not the part I trusted most. The real test came after it: The output looked polished, but the receiving person still had to rebuild the table, verify sources, or rewrite the handoff note. When that still happens, the output is only a draft with good manners, not a workflow I would put in front of another team.
What I would require before widening access
I would keep OpenAI Codex, ChatGPT, Claude, LangChain, Databricks, and AI agents outputs, one comparison table, and one handoff-ready memo as evidence before choosing the tool or workflow. If that evidence cannot be checked in a few minutes, I would narrow the automation scope before changing models or adding another integration.
Checks before expanding an agent
- Write down the exact input the AI receives.
- Name the person who approves or rejects the result.
- Decide where the output has to land next.
- Keep one example of a failed run, not only the clean example.
- Measure saved review time, not just generation speed.
- Stop the workflow if the next person keeps rebuilding the output.
Sources I checked
For claims that can change, I used official documentation, product pages, and source notes that support claims likely to change. I keep those sources separate from opinion because pricing, model access, and platform features move quickly.
The short version I would use in a meeting
An AI agent is not just a model with tools. It is a model working inside an operating setup.
| Layer | What it controls | Failure signal |
|---|---|---|
| Work packet | Goal, files, scope, current state | The agent asks the same setup questions every run |
| Tool boundary | Browser, terminal, documents, APIs, storage | It can read too little or change too much |
| Permission model | Read, draft, edit, delete, send, pay, publish | A useful action becomes a business risk |
| Check path | Tests, lint, diff, screenshots, citations, reviewer notes | The output looks fine but nobody can prove it |
| Recovery path | Retry, rollback, fallback owner, blocked state | One unexpected page or API error stops the whole flow |
| Memory and reuse | Saved rules, playbooks, learned steps | The agent starts from zero every morning |
The model matters. I am not arguing otherwise. But the model is only one part of the operating setup. If the surrounding harness is thin, every agent trial turns into a fresh supervision job for a human.
What “harness” means without the jargon
For a coding agent, the harness is not just the IDE. It includes the repository state, branch rules, test commands, package manager, local services, secrets policy, deployment script, code review habit, and the exact definition of done.
For a browser agent, the harness includes login state, allowed domains, source priority, screenshot evidence, download folder rules, and what to do when a page blocks automation.
For a document workflow, it includes source files, naming conventions, template structure, table formats, reviewer comments, version history, and the rule for what the agent may rewrite versus only mark up.
For a support or sales workflow, it includes customer data boundaries, CRM fields, escalation rules, tone policy, approval points, and whether the agent may actually send the message or only draft it.
This is why I do not like testing agents with one neat prompt and one clean file. It proves the model can perform a task. It does not prove the work can run next week when the file name changes, a login expires, a customer writes in bad grammar, or the reviewer asks what exactly changed.
Where agent work usually breaks first
The failures are rarely cinematic. They are boring, which is why teams underestimate them.
One agent can summarize a product requirements document well, but it does not know that the legal section is out of scope for rewriting. Another agent can inspect a spreadsheet, but it does not know that a blank cell means “not yet approved,” not “zero.” A coding agent can fix a test, but it misses the internal script that generates the file again during deployment. A browser agent can gather sources, but it cannot tell which source is the official policy page and which one is a scraped repost.
In each case, the model is not useless. The brief was incomplete.
I would not treat those as prompt failures first. I would mark them as harness failures:
| What went wrong | What I would fix before changing models |
|---|---|
| The agent touched the wrong file | Add an allow-list and a “show diff before edit” rule |
| The answer had no evidence | Require citations, screenshots, or command output before completion |
| The same exception repeats | Write the exception rule into the runbook, not another longer prompt |
| Review time did not fall | Split draft, check, and final action into separate stages |
| Nobody knows what changed | Force a final change log with files, sources, and remaining risk |
| The agent asked for access mid-run | Predefine allowed tools and blocked actions before starting |
That table sounds simple. In practice, it is the difference between “interesting agent demo” and “work I would let near production.”
A coding example that makes the point
Take a coding agent such as Codex. The impressive part is not that it can edit code. Many models can produce code. The real question is whether it can work inside the existing project without creating a second job for the developer.
For one repository, a useful harness might include this packet:
- The current branch and whether new branches are allowed.
- The files that define the runtime behavior.
- The exact test command and the command that is too slow for normal runs.
- The build command that works while the live app is running.
- The deployment path and the evidence required after deployment.
- The rule that customer data, secrets, and unrelated dirty files must not be touched.
Without that packet, the agent may still look productive. It can read code, make a patch, and explain itself. But the developer then has to check whether it used the right test command, changed the wrong boundary, broke an unstated convention, or left a deployment step undone.
That is not automation. That is assisted guessing with a review bill attached.
With the harness, the same agent becomes far more useful. It knows where to start, how to prove the work, when to stop, and what kind of handoff a human can actually trust.
A document example from ordinary office work
The same pattern shows up outside code. Suppose the agent is asked to prepare a vendor comparison memo from three PDFs, a spreadsheet, and notes from two calls.
The weak version is familiar: “Read these and make a comparison.”
The stronger harness looks more like this:
- Use the spreadsheet as the list of vendors, not the PDFs.
- Pull pricing only from the latest dated document.
- Mark missing information as “not confirmed” instead of guessing.
- Put security concerns in a separate section, not inside the score.
- Keep a source note for every claim that affects the recommendation.
- Draft the memo, but do not rewrite the executive recommendation without review.
Nothing there is glamorous. It is plain operating detail. But this is where trust is built. If the agent cannot separate confirmed facts from call notes and cannot show where the recommendation came from, I would not let it drive the decision.
Field judgment: what I check before expanding access
Before I give an agent more scope, I look at four things.
First, does the agent know the shape of the job before it starts? If a person has to explain the same folder structure, source order, and output format every time, the harness is not ready.
Second, can the agent show its work without being asked? I want diffs, citations, command output, screenshots, or a reviewer-friendly change list. A confident paragraph is not evidence.
Third, is the permission boundary boring? Read-only work can be loose. Drafting can be broader. Sending, deleting, publishing, paying, or updating customer records should be narrow until the checks are dull enough to be trusted.
Fourth, does the failed run make the next run better? If a blocked run only ends with “try again,” the system is not learning. If the exception becomes a rule, a checklist item, or a saved skill, then the harness is maturing.
The metric I would use instead of “agent quality”
Teams like to ask whether the agent is good. That question is too vague.
I would track the review burden. How many minutes did a person spend checking the output? How many facts needed repair? How many times did the agent stop because an input was missing? How often did a reviewer reject the final action? How many steps were rerun by hand?
If the agent produces a better-looking draft but review time stays the same, the harness has not paid for itself. If the agent produces a less flashy draft but cuts rework and makes the handoff clearer, I would take the second outcome.
The practical scorecard is blunt:
| Question | Good sign | Bad sign |
|---|---|---|
| Did the agent know the current state? | It references the right files, dates, and owner | It asks for basics already in the system |
| Did it use tools safely? | It reads broadly but writes narrowly | It edits before showing the plan |
| Did it produce evidence? | The reviewer can verify the path | The reviewer has to recreate the work |
| Did failure improve the system? | The exception becomes a rule | The next run repeats the same mistake |
| Did the human workload fall? | Review time drops | The output is polished but still risky |
Do not choose an agent-first rollout
I would not begin by asking, “Which agent should we buy?” I would begin by picking one repeated workflow where the pain is visible.
Good candidates have a clear input, a repeatable output, a known reviewer, and a measurable handoff. Bad candidates depend on hidden politics, vague judgment, or permission to act before the review habit exists.
For example, I would start with “prepare the first draft of the weekly vendor-risk memo with sources attached.” I would not start with “handle vendor risk.” One has a harness. The other is a wish.
This is also why prompt libraries disappoint people. A prompt can improve one moment. A harness improves the path around the moment.
Failure criteria I would write down
Write the stop rules before the agent looks impressive.
- Stop if the agent cannot name the source of a recommendation.
- Stop if the agent needs a human to recheck every line.
- Stop if it asks for broader permissions to finish routine work.
- Stop if it changes production-facing material before review.
- Stop if the same exception appears three runs in a row.
- Stop if the owner cannot explain what the agent did.
These rules sound conservative. They are cheaper than cleaning up a confident mistake after the agent has access to real systems.
What I would build first
I would build a small harness document before changing the model. One page is enough:
- What work enters the agent.
- What the agent may read.
- What the agent may change.
- What evidence it must leave.
- Who reviews the result.
- What happens when the run blocks.
- Which repeated mistake becomes a new rule.
Then I would run the same workflow five times and compare review time, not vibes. If the fifth run still needs the same human explanation as the first run, the problem is not solved.
FAQ
Is harness engineering only for coding agents?
No. Coding agents make the problem visible because repositories, tests, and diffs are concrete. The same pattern applies to document work, browser research, customer support, sales operations, and internal reporting.
Does a better model reduce the need for a harness?
It reduces some mistakes, but it does not remove the operating problem. A better model still needs the right context, safe tools, evidence requirements, and a review path.
Should every workflow become an agent workflow?
No. If the task is rare, judgment-heavy, or politically sensitive, a checklist and a good human owner may be better. Agents earn their place when repetition, evidence, and handoff are clear.
What is the first sign that the harness is working?
The agent becomes less dramatic. It asks fewer setup questions, leaves clearer evidence, stops safely when blocked, and makes the reviewer’s job smaller instead of louder.
Final judgment
The next useful wave of AI agents will not be won only by larger models. It will be won by teams that know how to package work, limit tools, prove results, and turn repeated failures into reusable operating rules.
That is why harness engineering matters. It is not a decorative layer around the model. It is the difference between an impressive run and a work system that can be trusted again tomorrow.
Workflow path
Where this guide fits
Use this section to connect the guide you are reading with the broader workflow it supports.
A path for comparing automation platforms, app builders, agent builders, bookkeeping tools, and general AI assistants.
Open workflow path- Best fit
- teams deciding whether to buy a simple tool, build an internal workflow, or adopt a broader platform
- Not ideal if
- You need step-by-step setup instructions more than a decision framework.
Sources checked
Main public pages used to check reported facts, official documentation, policy background, product details, and claims that may change.
- Harness engineering OpenAI
- Harness engineering for coding agent users Martin Fowler
- The Anatomy of an Agent Harness LangChain
- What is an AI Harness? Databricks
- Building effective agents Anthropic
- Engineer photo Wikimedia Commons / Bench Accounting
