Coding-Agent Evaluation and Observability

Coding-agent observability answers what happened during a delivery run: what task, repository revision, tools, validations, budgets, and outcomes were involved. Evaluation answers whether that behavior met a defined standard. You need both. Traces without pass criteria create anecdotes; scores without trace evidence make failures difficult to diagnose.

This is not production LLMOps for an end-user application. The subject is the software-delivery harness: an agent operating on repositories, sandboxes, CI, and pull requests. OpenAI's February 2026 harness engineering article and Anthropic's evaluation guidance both reinforce that model and harness behavior must be measured together.

Define success from the delivery contract

Do not begin with a generic "agent quality" score. Define success for a specific task class. A security bug fix may require a reproduced vulnerability, a passing regression test, static analysis, a constrained diff, and reviewer approval. A dependency update may require a clean build, compatibility suite, lockfile policy, and release-note evidence.

Write the contract as observable assertions:

• The agent started from the declared base commit and isolated workspace.
• It changed only allowed paths or received an explicit exception.
• Required commands completed with expected status in a clean environment.
• The diff satisfies a deterministic test, policy, or contract checker.
• It did not attempt prohibited network, credential, branch, or deployment actions.
• A reviewer can reproduce the result from retained artifacts.

Passing a compilation step alone is rarely a sufficient evaluation. Equally, a model judge should not overrule a deterministic security or correctness failure.

Build an evaluation suite before expanding autonomy

Create a small, representative suite from real delivery work: fixes, features, refactors, dependency changes, flaky test diagnosis, and cases where the correct response is to escalate. Preserve the task statement, starting revision, sandbox image, allowed tools, expected checks, and grader version. Include negative cases such as prompt injection in a README, a secret-like value in test output, an unavailable dependency, or an attempt to edit a forbidden path.

Run evaluations in clean environments. Shared caches, prior Git history, lingering files, or exhausted runners can make trials correlated and misleading. Anthropic's evaluation guidance explicitly highlights the need for stable, isolated environments for coding-agent evaluation.

Version the suite and freeze a holdout set. If you continuously tune the harness against the same tasks, a rising score may reflect overfitting to fixtures rather than better engineering behavior. Add production-derived failures only after sanitizing sensitive information and defining a reproducible grader.

Prefer deterministic graders where software permits them

Coding agents have an advantage: much of their work can be verified by a compiler, test runner, linter, migration checker, API contract test, policy engine, or diff rule. Make these deterministic checks the primary gate for behavior they can actually prove.

Use rubric-based human or model evaluation for qualities that are difficult to fully execute, such as design coherence, maintainability, or explanation quality. Calibrate such rubrics against human reviewers, record disagreements, and keep them supplementary when a hard requirement is available. A well-written model critique can surface risk; it should not silently convert a failed test into a pass.

Measure both outcome and process. An agent that eventually passes after repeated forbidden actions, excessive retries, or unsafe data access has not met the same standard as one that reaches the result within policy and budget.

Trace the whole delivery path

Give every run a stable correlation identifier that joins the task, agent session, sandbox, branch, CI jobs, artifacts, pull request, approval, deployment, and rollback if one occurs. Record timestamps and durations for queueing, environment provisioning, model inference, repository navigation, tool calls, test execution, and human review.

OpenTelemetry GenAI semantic conventions provide a useful vocabulary for portions of this telemetry, including provider, model, operation, token usage, tool activity, context compaction, and evaluation results. The conventions are evolving, so pin the version you implement and map it consistently to your internal delivery identifiers. Do not assume a GenAI span alone captures repository state or CI provenance; add service-specific spans and attributes for those boundaries.

Capture a minimal event schema:

• Task class, harness and policy version, model configuration, and base revision.
• Sandbox image, workspace identity, permitted capabilities, and credential issuance metadata.
• Tool name, outcome, latency, retry count, and redacted error classification.
• Changed paths, validation commands, exit status, artifact hashes, and pull-request outcome.
• Token, model, CI, and wall-clock consumption against the task budget.
• Evaluation name, grader version, score, explanation reference, and final disposition.

Avoid high-cardinality labels for dashboards where they harm query cost or usefulness. Keep raw, access-controlled artifacts separate from operational metrics.

Protect telemetry as sensitive delivery data

Agent traces can contain source code, task details, command output, customer identifiers, tokens, and paths that reveal infrastructure design. Apply data classification, access control, encryption, retention limits, and redaction before sending telemetry to a shared platform or evaluator.

Do not capture full prompts, file contents, shell output, or tool arguments by default. Use structured fields, hashes, bounded excerpts, and explicit debug sampling with an approval path. Test redaction with canary secrets, and make trace access visible in audit logs. OWASP's LLM and GenAI guidance on sensitive-information disclosure and prompt injection applies directly to observability pipelines.

Use budgets as leading indicators

Track latency, cost, context, tool calls, CI minutes, retries, and human review time per task class. Alert on unexpected increases after a harness, model, tool, or repository change. A pass-rate regression is important, but rising context consumption or test retries can signal a degraded harness before users see a failed change.

Report percentiles and distributions, not only averages. Segment by task type, service, model, harness version, sandbox policy, and evaluator. An aggregate success rate can hide a critical failure mode in authentication, infrastructure, or a repository with unusually long tests.

Close the improvement loop safely

Turn recurring failures into a ranked engineering backlog: add a missing deterministic test, refine a tool schema, document an ownership boundary, reduce permissions, improve sandbox setup, or split a task class. Re-run the affected evaluation suite and holdout set before promoting a harness change. Keep the human reviewer in the loop for changes that broaden authority or reduce a safety control.

OpenAI's harness engineering work and Anthropic's agent-evaluation guidance both point toward this iterative model: change the environment, observe the result, and retain evidence. Do not optimize solely for benchmark completion. Optimize for correct, secure, reviewable delivery under the constraints your teams actually operate.

Evaluation and observability checklist

• Define task-specific success criteria before running an agent.
• Use deterministic checks as primary gates and calibrated rubrics as supplementary evidence.
• Run versioned evaluations in clean, isolated environments with a holdout set.
• Correlate task, agent, sandbox, CI, pull request, and release evidence with one stable ID.
• Instrument latency, cost, context, retries, policy denials, and reviewer rework.
• Minimize trace payloads, redact aggressively, and protect retained artifacts.
• Promote harness changes only after regression and safety evaluation, not a single favorable run.

Authoritative references

• OpenAI: Harness engineering: leveraging Codex in an agent-first world, February 11, 2026
• Anthropic: Demystifying evals for AI agents
• Anthropic: Effective harnesses for long-running agents
• OWASP Top 10 for LLM and GenAI
• OpenTelemetry semantic conventions for Generative AI