Engineering guide

AI Harness Engineering for Coding Agents

Build the delivery environment around a coding agent so its changes are scoped, testable, observable, and safe to review.

Published: Jul 14, 2026
Updated: Jul 15, 2026
Reading time: 10 min read

On this page

AI harness engineering is the practical design of the environment around a coding agent: its task contract, repository context, tools, execution boundaries, feedback, and release controls. It is not a standardized discipline and it is not a synonym for production LLMOps. The target is a reviewable software change before merge or deployment.

The same discipline scales the way a managed edge-orchestration layer does: once dozens of services sit behind one entry point, the guardrails around any automated change must be consistent and independently checkable, not reinvented per repository — the value sits in the contracts and controls between systems, the same principle Optimi applies to orchestrating best-of-breed providers across a customer's edge.

Overview

Outcome

A bounded agent run produces either a small Acme Shop checkout-tax patch with reproducible evidence, or a concise escalation that preserves the branch, failures, and decision needed from a human.

Overview

Prerequisites

Use a protected repository, isolated workspaces, a fast local check plus an authoritative CI gate, a path-policy checker, and a named reviewer for changes to checkout behavior.

What AI harness engineering controls

AI harness engineering treats four surfaces as separate design problems: the task contract, repository context, tool and execution boundaries, and evaluation. A well-written contract cannot compensate for an unreadable repository, nor a generous sandbox for a vague success criterion.

Run one bounded Acme Shop task

Use one scenario throughout the harness: Acme Shop's checkout-api must reject a tax calculation when the cart country is missing. The issue links a failing request and acceptance test. The agent may edit only services/checkout-api/** and its tests; it must not change payment, infrastructure, dependency locks, or deployment configuration.

The task contract is a delivery input, not merely a better prompt. It names the permitted outcome, evidence, non-goals, and escalation path. A repository policy and CI gate must independently reject changes outside those bounds.

A contract should also carry a change budget: a numeric ceiling on files touched and lines changed, set before the run starts. ACME-1842 is scoped to one file, so two files and 40 lines is generous, not tight — and because the CI gate enforces it, not the prompt, scope creep gets stopped rather than justified.

Acme Shop coding-agent loop

Task contract
Read repository map
Isolated patch
Fast checks
CI and review
Merge or escalation

The agent receives a bounded contract, works in an isolated checkout-api workspace, and returns evidence to a human-controlled merge gate. Prompt guidance explains the route; policy and CI enforce the boundary.

Task contract: ACME-1842

id: ACME-1842
repository: acme-shop
base_ref: main
allowed_paths:
- services/checkout-api/**
- services/checkout-api/tests/**
required_checks:
- bun run test:checkout
- bun run typecheck
change_budget:
max_files: 2
max_diff_lines: 40
non_goals:
- payment provider changes
- dependency or infrastructure changes
success: "Missing country returns TAX_COUNTRY_REQUIRED with a regression test"
escalate_when: "baseline fails or a change outside allowed_paths is needed"

Coding agent loop design: a short, deterministic route

Coding agent loop design decides what the agent does turn by turn, not just what it is allowed to touch. The root instructions should tell every contributor how to establish a clean baseline, locate the repository map, run the fast check, and report evidence. Local instructions belong only where rules differ, such as services/checkout-api/AGENTS.md naming its contract-test command and prohibiting fixture exports from production data.

The first loop should be deterministic:

Record branch, base commit, task ID, and harness version.
Read root and local instructions plus the checkout-api map.
Run the declared baseline test before editing.
Make the smallest allowed patch and focused regression test.
Run required local checks, then submit the diff and evidence to CI and review.
Cap tool-call retries and output size; a command failing the same way twice is a stop signal, not a retry cue.

Do not replace this loop with "review your own work." A model critique may be useful context, but it is not a verifier. OpenAI's harness engineering discussion and Anthropic's long-running-agent guidance both emphasize shaping work, state, and feedback around the agent rather than relying on model capability alone.

Two failure modes belong to the loop itself: unbounded tool output that gives the agent more to lose track of, not more information, and silent retries where a failing command gets rerun with cosmetic changes instead of stopping. Treat a repeated identical failure as an automatic stop, the same way a failed baseline is.

Representative run output

[ACME-1842] baseline: PASS (42 checkout tests)
[ACME-1842] changed: services/checkout-api/src/tax.ts
[ACME-1842] changed: services/checkout-api/tests/tax.test.ts
[ACME-1842] test:checkout: PASS (43 tests)
[ACME-1842] typecheck: PASS
[ACME-1842] path-policy: PASS
[ACME-1842] disposition: ready for CI review

Separate guidance from controls

Prompt and instruction text can tell an agent not to read .env, call a production API, or merge a branch. They cannot enforce that behavior. The harness must apply filesystem allowlists, default-deny egress, short-lived identities, branch protections, command limits, and CI review rules outside the model context.

For Acme Shop, a path-policy service can reject an attempted edit to infra/ even if the task wording is ambiguous. Similarly, a sandbox can deny an outbound request to an unapproved host even if a README tells the agent to install a package from it, and a CI gate can reject a diff over ACME-1842's change budget regardless of how the agent justifies it. These controls reduce authority; they do not prove that a platform is impossible to misconfigure.

A coding-agent run stops when evidence does not meet the contract

A harness treats failures and denied actions as deliberate stop states, not prompts to retry around a control.

Download:PNG SVG

Handle work that outgrows one session

Not every task fits ACME-1842's shape. ACME-1843, VAT-inclusive pricing across five checkout services, is too large for one context window and too interdependent for a single allowlist — run as one oversized session, it loses the agent's own reasoning and leaves review with an unexplained diff.

Split it into session-scoped subtasks and keep state outside the model's context, following Anthropic's long-running-agent pattern: an initializer builds a task list, and each session reads git history and a progress note before acting. Adapted to Acme Shop: a tax-rollout.json list, one entry per service; each session claims one entry and commits on exit. The change budget and allowlist still apply per session; git history and the note, not memory, are the source of truth a disagreeing session must escalate against.

Validate, stop, and recover

Validate the contract at three levels:

Positive validation: a request without country returns 400 and TAX_COUNTRY_REQUIRED; the focused test and type check pass; only allowed paths changed.
Negative validation: a test for a valid French cart still returns the expected tax quote; the path-policy checker rejects an attempted infra/ edit.
Failure validation: simulate a failing baseline or blocked package host. The run must stop, label the blocker, retain the unmerged diff and test output, and request the named escalation rather than retrying around the control.

Recovery is deliberate: restore the base worktree, attach the failed command and trace ID to ACME-1842, and have the task owner either repair the baseline, widen the contract through review, or split the work. Do not grant a broader token or disable a gate to turn an ambiguous run into a pass.

Troubleshooting

Symptom	Likely cause	Evidence to collect	Safe recovery
Baseline checkout test fails	Pre-existing regression or undeclared service dependency	Base commit, command, failure ID	Stop; owner confirms baseline or repairs it before rerun.
Diff includes `infra/`	Task was underspecified or agent followed unrelated context	Path-policy result and diff	Reject the diff; create a separately reviewed infrastructure task.
Test passes locally but fails CI	Environment or authoritative gate differs	Image/version, CI log, artifact link	Reproduce in the declared CI image; update instructions only after confirming the cause.
Agent exhausts retries	Flaky test or missing evidence	Retry count and each distinct failure	Stop at budget; assign a flake owner or provide the missing fixture.
Review cannot reconstruct the change	Handoff omitted commands or risk	Task ID, changed paths, check results	Request a concise evidence report before merge.
Diff exceeds the change budget	Task under-scoped or agent bundled an unrelated fix	Budget limits, files/lines changed	Reject the diff; split the extra change into its own reviewed task.
Next session contradicts prior work	Progress note or task-list entry stale or ignored	Git log, progress note, task-list state	Stop; reconcile the note with git history before resuming.
Grader disagrees with a manual transcript read	Grader too rigid (exact-match) or too lenient (LLM judge drift)	Transcript, grader output, expected outcome	Fix the grader, not the task; re-run the affected batch.

Operate the harness as delivery infrastructure

Version the task template, policy, tools, image, and evaluation suite. Measure completion and regression rates by task class, reviewer rework, policy denials, time to a validated patch, and cost per accepted change. Segment by repository, harness version, model, and tool set; an aggregate pass rate can hide a critical regression in checkout.

The evaluation suite deserves the same rigor as the harness it grades. Start small — twenty to fifty tasks drawn from real Acme Shop incidents outscore a large synthetic set, since each already has a known-correct outcome. Grade what the run produced, not the exact commands used to get there; agents routinely find a valid route the eval author did not anticipate. Keep the grader under review too — a literal one can fail a correct result over formatting, and a permissive judge can pass a wrong one.

One evidence plane across every repository running an agent

A managed orchestration layer running coding-agent harnesses across many services hits this at scale first: each repository's contract and CI gate can be reasonable alone yet produce evidence in incompatible shapes, hiding a fleet-wide regression until production. Correlating task IDs, check results, and escalations through one observability plane — the discipline Optimi applies to routing and security decisions through MYO — turns "this run passed" into comparable, provable evidence across the fleet.

Start with read-only investigation or a local patch, then allow branch or pull-request creation only when the task class has repeatable evidence. Passing tests do not authorize merges, credential rotation, production changes, or external contact.

Authoritative references

Bring reviewable coding-agent delivery to every repository

Talk to Optimi about orchestrating the task contracts and MYO-observed evidence that keep agent-written changes fast to ship, safe to trust, and visible to every reviewer — consistently, across every repository and provider.

Discuss engineering delivery architecture