AI Harness Engineering for Coding Agents

Scope: software delivery, not a standardized discipline

In this guide, AI Harness Engineering means the practical engineering of the environment around coding agents: repository instructions, tools, execution boundaries, task contracts, feedback loops, and release controls. It concerns agents that inspect, change, test, and propose software changes. It is not an established standardized discipline, and it is not a synonym for production LLMOps. Production LLMOps focuses on operating model-backed applications in production; a coding-agent harness focuses on making software-delivery work reliable before a change is merged or deployed.

The phrase is useful because model capability alone is not a delivery system. The same model can produce a low-risk patch in one repository and an unsafe, untestable change in another when its available context, permissions, validation, or stop conditions differ. OpenAI's February 2026 discussion of harness engineering emphasizes shaping the work around Codex; Anthropic's work on long-running agents similarly shows the value of clean state, incremental work, and explicit handoff artifacts.

Define a task contract before granting tools

A useful task is a bounded contract, not just a request such as "fix the failing tests." Give the agent a problem statement, allowed directories, non-goals, expected behavior, required checks, and the escalation path for ambiguity. Link the issue, design decision, or incident evidence rather than pasting an unbounded history into its context.

For each task, decide the highest permitted outcome:

• Read-only investigation: findings and reproduction steps, with no writes.
• Local patch: a diff plus tests, with human approval before merge.
• Branch or pull request: a CI-validated proposal, still subject to repository protection rules.
• Automated merge or deployment: only for a narrow, repeatedly evaluated change class with explicit rollback and ownership.

Start with the least autonomous option that can complete the work. A passing test is evidence about behavior, not blanket authorization to merge, rotate credentials, alter infrastructure, or contact external systems.

Make repository context navigable

Coding agents need a path to relevant facts, not a complete repository dump. Place stable instructions close to the work: a root contributor guide, directory-level conventions where they genuinely differ, and commands that explain how to validate a change. Keep architecture decisions, dependency boundaries, service ownership, and rollout requirements discoverable through short documents and predictable names.

An effective starting sequence is deterministic: identify the working tree and current branch, read repository instructions, inspect recent changes and the task's area, run a fast baseline check, then select the smallest relevant files. This reduces wasted context and reveals a pre-existing failure before an agent attributes it to its own patch.

Repository context should answer these questions without inference:

• Which command is the fast local check, and which is the authoritative CI check?
• Which files are generated, secret-bearing, vendor-managed, or off-limits?
• What user-visible behavior and compatibility constraints matter?
• Who owns the component and what evidence is required for a risky change?
• How can a later agent or reviewer reconstruct the decision from the branch and logs?

Anthropic's long-running-agent guidance uses structured progress artifacts and incremental changes to carry state across sessions. Adopt that idea selectively: use issue links, commit history, test reports, and concise handoff notes as durable state. Do not let free-form agent memory become the sole record of a production-affecting decision.

Build deterministic feedback loops

Agents improve when the environment returns specific, repeatable evidence. Prefer a small sequence of fast, deterministic checks: formatting, type checking, focused unit tests, static analysis, build validation, and a targeted integration or browser test when the task requires it. Return concise failures with paths, commands, and stable error identifiers.

Avoid a loop that only asks the model to "review its work." It can be useful as an additional check, but it is not a substitute for a verifier. For coding tasks, make the acceptance criteria executable where possible:

1. Reproduce the stated failure or demonstrate the missing behavior.
2. Make the smallest change that satisfies the contract.
3. Run the required checks in a clean, declared environment.
4. Compare the diff against allowed paths and prohibited changes.
5. Record commands, outcomes, and unresolved risk for a reviewer.

When a check is slow, provide a deterministic fast subset for iteration and reserve the full suite for the gate. Random samples can be useful for coverage, but seed and record them so a regression can be reproduced.

Design bounded tools and permissions

Tool definitions are part of the agent-computer interface. A tool should expose the smallest useful action, validate its inputs, return machine-readable failures, and make unsafe actions hard to express. For example, a deployment inspection tool can return release state without accepting arbitrary shell; a repository-write tool can reject paths outside the task worktree.

Separate read, write, execute, network, and approval capabilities. The agent that can inspect an incident need not have production shell access. The agent that can edit an application repository need not have access to cloud credentials. OWASP's LLM and GenAI guidance is relevant here: prompt injection, sensitive-information disclosure, supply-chain risks, improper output handling, and excessive agency become concrete delivery risks when untrusted repository content can influence a privileged agent.

Use a policy layer outside model instructions for controls that must hold. Prompt text can guide behavior; it cannot enforce filesystem isolation, network egress, branch protection, or credential scope.

Set explicit budgets and stopping rules

Every run consumes delivery capacity. Define budgets for wall-clock latency, model and tool cost, context size, retries, command duration, and concurrent agents. Budgets make failure modes visible: a broad repository search can exhaust context, a flaky integration test can consume the entire latency budget, and unbounded retries can create cost without new evidence.

Useful stop conditions include a fixed number of failed validation attempts, an unresolvable policy denial, a missing dependency, a changed baseline, or a required human decision. On stop, preserve the diff, test output, relevant trace identifiers, and a short explanation of what blocked progress. The correct result is sometimes an actionable handoff, not a forced patch.

Operate a reviewable delivery loop

For most teams, a safe initial loop is: dispatch one bounded task in an isolated workspace, require deterministic validation, open a reviewable pull request, and collect evidence before merge. Expand autonomy only after the same class of task has demonstrated stable evaluation performance and low operational risk.

Measure the delivery system rather than celebrating raw agent activity. Track task completion and regression rates by task class, reviewer rework, test flakiness, time to a validated patch, merge conflicts, policy denials, cost per accepted change, and the fraction of runs that end with a useful escalation. Segment these metrics by repository, harness version, model, tool set, and evaluation suite; an aggregate score can hide a dangerous regression in a critical service.

Implementation checklist

• Publish concise repository instructions and keep validation commands runnable.
• Create task contracts with allowed scope, evidence, non-goals, and escalation rules.
• Isolate workspaces and give every run a unique branch, trace, and artifact location.
• Enforce filesystem, network, credential, and merge permissions outside prompts.
• Run fast deterministic checks during iteration and authoritative checks before acceptance.
• Bound latency, cost, context, retries, and concurrency; preserve evidence when a run stops.
• Review harness changes as carefully as the code changes they enable.

Authoritative references

• OpenAI: Harness engineering: leveraging Codex in an agent-first world, February 11, 2026
• Anthropic: Effective harnesses for long-running agents
• Anthropic: Demystifying evals for AI agents
• OWASP Top 10 for LLM and GenAI
• OpenTelemetry semantic conventions for Generative AI