Self-Optimizing Coding Agent Harnesses with SuperQode

Shashikant Jagtap

Coding agents have become a practical part of software development workflows. Tools such as Codex-style agents, Claude Code-style agents, OpenCode-style agents, and IDE-integrated assistants are increasingly good at reading repositories, editing files, running tests, and explaining changes. As these systems mature, the technical question shifts from whether an agent can generate code to whether a team can define, measure, control, and improve the system around that agent.

That system is the harness. A harness determines which context the model receives, which tools it can use, what permissions are available, how approvals are handled, which evals are run, what memory persists, and which candidate changes are accepted or rejected. In production software work, the harness is often the difference between a useful coding assistant and an unreliable automation surface.

This article demonstrates a self-optimizing harness workflow using SuperQode. The workflow is implemented in SuperQode 0.2.8 and is available as a runnable demo repository: superqode-harness-optimization-demo. The SuperQode documentation is available at superagenticai.github.io/superqode.

Background: Harness Engineering

Lilian Weng’s article, Harness Engineering for Self-Improvement, provides a useful technical frame for this work. The post describes the harness as the deploy-time system around a model: tools, memory, context, workflow, permissions, artifacts, and evaluation. It also discusses self-improvement loops where failures are mined, candidate changes are proposed with bounded context, and validation gates prevent regressions.

SuperQode applies that pattern to coding-agent harnesses. The goal is not to replace the developer’s preferred coding agent. The goal is to provide a framework for defining and improving the harness around coding agents in a way that is measurable, auditable, and repeatable.

SuperQode as a Harness Optimization Framework

SuperQode treats the harness as a first-class artifact. A SuperQode HarnessSpec is a YAML file that describes model policy, runtime, tools, sandbox behavior, approvals, workflow, checks, context, memory, and optimization policy. Because this contract is explicit, it can be inspected, versioned, evaluated, and improved.

The self-optimization workflow in SuperQode is built around five capabilities:

  • Failure mining: convert test, eval, or benchmark output into structured failure records.
  • Logbook memory: persist recurring harness failure patterns in a file-backed logbook.
  • Bounded improvement evidence: export a candidate-improvement project with relevant failures, eval tasks, logbook entries, and surface boundaries.
  • Candidate audit: compare a proposed harness against the baseline and reject unsafe or unvalidated changes.
  • Candidate ledger: preserve accepted and rejected attempts so future optimization does not repeat failed edits.

The important design decision is that candidate generation and candidate acceptance are separate concerns. An LLM or meta-harness can propose a change, but SuperQode’s audit layer decides whether the change is safe to accept.

Demo Repository

The demo repository contains a deterministic example that does not require an API key:

git clone https://github.com/SuperagenticAI/superqode-harness-optimization-demo.git
cd superqode-harness-optimization-demo
./demo/run-demo.sh

The demo uses the published SuperQode package:

uvx superqode==0.2.8

The script writes generated artifacts into _run/. It uses a fixture eval result and passes --allow-ungated during candidate audit so the loop can run offline. In a production setting, the audit step should use a live held-out eval result with --require-heldout.

Demo File Layout

The repository is intentionally small:

superqode-harness-optimization-demo/
  README.md
  REPORT.md
  demo/
    harness.yaml
    tasks.yaml
    eval-failed.json
    candidate-safe.yaml
    candidate-unsafe.yaml
    run-demo.sh
    expected-output.md
  artifacts/
    failures.example.json
    trace-evidence.example.md
    candidates.example.jsonl

The main files are:

demo/harness.yaml,  Baseline SuperQode harness with optimization boundaries.

demo/tasks.yamlHeld-in and held-out eval task split.

demo/eval-failed.jsonFixture eval failure used for weakness mining.

demo/candidate-safe.yamlA narrow candidate that edits only the context surface.

demo/candidate-unsafe.yamlA candidate that widens shell permission and disables checks.

demo/run-demo.shEnd-to-end command script.

The Baseline Harness

The demo starts with a small HarnessSpec:

name: demo-harness
inherits: no-tool
context:
  instruction_files: [AGENTS.md]
checks:
  enabled: true
  fail_on_error: true
optimization:
  enabled: true
  editable_surfaces: [context, workflow, model_policy, agents.tools]
  protected_surfaces: [execution_policy, checks, approvals, sandbox]
  heldout_fraction: 0.3
  max_candidate_edits: 3

This harness declares which surfaces may be optimized and which surfaces require stricter review. context, workflow, model_policy, and agents.tools are editable. execution_policy, checks, approvals, and sandbox are protected. The maximum candidate edit count is three.

This policy is central to the workflow. A harness optimizer should be allowed to improve the mechanism, but it should not silently weaken permissions, disable checks, or bypass the evaluation layer.

Held-In and Held-Out Eval Tasks

The demo task file defines one held-in task and one held-out task:

tasks:
  - id: train-readme
    split: held-in
    prompt: explain the project
  - id: gate-safety
    split: held-out
    prompt: explain the safety policy

The held-in split represents tasks that can be used to diagnose and improve known weaknesses. The held-out split represents tasks reserved for validation. SuperQode supports this directly:

uvx superqode==0.2.8 harness eval \
  --spec harness.yaml \
  --tasks tasks.yaml \
  --split held-out \
  --json

The offline demo runs this in dry mode. A production workflow should add --live and use the result as an audit gate.

Step 1: Mine a Structured Failure

The demo includes a fixture eval result where the harness fails because it did not include enough project context:

{
  "split": "held-in",
  "live": true,
  "variants": [
    {
      "harness": "demo-harness",
      "spec": "harness.yaml",
      "regressions_vs_baseline": ["train-readme"],
      "tasks": [
        {
          "id": "train-readme",
          "status": "failed",
          "reason": "missing project instructions",
          "failure_digest": {
            "failure_category": "context_error",
            "dimension": {
              "id": "D2",
              "label": "context assembly",
              "field": "context"
            },
            "evidence": ["AGENTS.md was not enough context"]
          }
        }
      ]
    }
  ]
}

Run failure mining:

uvx superqode==0.2.8 harness mine-failures \
  --eval-result eval-failed.json \
  --output failures.json

Expected output:

Failure mining: 1 failure(s)
Dimensions: D2=1
  fail_0001: train-readme D2:context missing project instructions
Wrote: failures.json

The resulting failure record contains the task id, failure category, harness dimension, source reference, symptom, evidence, and suggested surfaces. This is more useful than a free-form error message because it can feed both memory and candidate generation.

Step 2: Persist the Failure in the Logbook

The next step updates the self-improvement logbook:

uvx superqode==0.2.8 harness logbook update \
  --from-failures failures.json

uvx superqode==0.2.8 harness logbook show

Expected output:

Self-improvement logbook: 1 failure pattern(s)
  fp_0001: count=1 confidence=low status=active D2:context missing project instructions

The logbook is stored under:

.superqode/self-improve/logbook/failure_patterns.yaml

This makes harness memory inspectable and versionable. A team can review what the harness has learned, prune low-signal memory, and preserve recurring failure patterns across optimization attempts.

Step 3: Export a Bounded Improvement Project

SuperQode can export a harness-improvement project that contains the baseline harness, eval task contract, trace evidence, and meta-harness configuration:

uvx superqode==0.2.8 harness improve \
  --spec harness.yaml \
  --tasks tasks.yaml \
  --from-failures failures.json \
  --project-dir improve-demo \
  --export-only

This creates:

improve-demo/
  baseline/
    harness.yaml
    eval-tasks.yaml
  metaharness.json
  tasks.json
  trace-evidence.md

The generated trace-evidence.md file contains the bounded context for candidate generation:

  • baseline harness snapshot
  • held-in and held-out task counts
  • mined failures
  • logbook memory
  • previous candidate attempts
  • editable surfaces
  • protected surfaces

This file is designed to provide enough information for a proposer to suggest a targeted harness edit, while keeping the evaluator and permission policy outside the proposal loop.

Step 4: Audit a Safe Candidate

The safe candidate makes one narrow change to the context configuration:

context:
  instruction_files: [AGENTS.md, SUPERQODE.md]

Audit it:

uvx superqode==0.2.8 harness audit-candidate \
  --base harness.yaml \
  --candidate candidate-safe.yaml \
  --tasks tasks.yaml \
  --allow-ungated \
  --record

Expected output:

Candidate audit: cand_1bf84ccaa1a6
Decision: accepted
Changed fields: 1
Changed surfaces: context
Recorded: .superqode/self-improve/candidates.jsonl

This candidate is accepted because it edits an allowed surface and stays within the configured edit budget.

Step 5: Audit an Unsafe Candidate

The unsafe candidate changes the harness flavor, enables shell access, and disables checks:

name: demo-harness
flavor: coding
execution_policy:
  allow_shell: true
checks:
  enabled: false

Audit it:

uvx superqode==0.2.8 harness audit-candidate \
  --base harness.yaml \
  --candidate candidate-unsafe.yaml \
  --tasks tasks.yaml \
  --allow-ungated \
  --record

Expected output:

Candidate audit: cand_bbe95738c3d7
Decision: rejected
Changed fields: 14
Violations:
  - protected_surface_change
  - out_of_scope_change
  - candidate_edit_limit
  - permission_widening
  - check_weakening

This candidate is rejected because it changes protected surfaces, exceeds the configured edit budget, widens shell permission, moves to a less restrictive sandbox, disables checks, and stops failing on check errors.

This is the core control point. The optimizer can explore candidate harness changes, but the acceptance layer prevents broad or unsafe changes from being applied silently.

Step 6: Inspect the Candidate Ledger

The final step inspects the candidate ledger:

uvx superqode==0.2.8 harness candidates list

Expected output:

candidate       decision  surfaces  violations
cand_1bf84ccaa1a6 accepted  context  -
cand_bbe95738c3d7 rejected  agents,approvals,checks,context,description,execution_policy,flavor,inherits,metadata,optimization,sandbox  protected_surface_change,out_of_scope_change,candidate_edit_limit,permission_widening,permission_widening,check_weakening,check_weakening

The ledger is stored at:

.superqode/self-improve/candidates.jsonl

The accepted candidate and rejected candidate are both preserved. This gives the harness optimizer negative result memory, which is important for avoiding repeated attempts that already failed audit.

Production Validation

The demo uses --allow-ungated so it can run without a live model. A production workflow should replace that with a held-out eval gate:

uvx superqode==0.2.8 harness eval \
  --spec harness.yaml \
  --variant candidate.yaml \
  --tasks tasks.yaml \
  --split held-out \
  --live \
  --json > heldout.json

uvx superqode==0.2.8 harness audit-candidate \
  --base harness.yaml \
  --candidate candidate.yaml \
  --tasks tasks.yaml \
  --eval-result heldout.json \
  --require-heldout \
  --record

This gives the full production loop:

  1. mine known weaknesses from evals, tests, or benchmark runs
  2. persist recurring patterns into the logbook
  3. export bounded improvement evidence
  4. generate candidate harness changes
  5. validate candidates against held-out tasks
  6. audit protected surfaces, permissions, checks, and edit scope
  7. record accepted and rejected candidates

Relationship to Existing Coding Agents

SuperQode does not need to replace an existing daily coding agent to be useful. Many teams will continue to use the agent interface that fits their editor, terminal, subscription, or model preference. SuperQode is positioned at a different layer: the harness layer.

A harness optimization framework helps answer questions that a single coding agent usually does not answer on its own:

  • Which context policy works best for this repository?
  • Which workflow reduces repeated failures?
  • Which model and tool combination reduces cost or latency?
  • Did a candidate harness edit widen permissions?
  • Did it disable checks?
  • Did it regress held-out tasks?
  • Have we already rejected this type of edit?

As coding agents become more common, harness ownership becomes a practical engineering concern. Teams need the ability to define and improve the harness, not only consume a fixed harness from a vendor.

Try the Demo

Install and check SuperQode:

uvx superqode==0.2.8 --version

Run the demo:

git clone https://github.com/SuperagenticAI/superqode-harness-optimization-demo.git
cd superqode-harness-optimization-demo
./demo/run-demo.sh

The final summary should be:

failures=1 candidates=2 accepted=1 rejected=1

That result demonstrates the smallest complete harness self-optimization loop: a failure is mined, harness memory is updated, improvement evidence is generated, a narrow candidate is accepted, an unsafe candidate is rejected, and both outcomes are preserved for future optimization.

Github Repo here.

Final Note on Future of Harnesses

Harness Engineering concepts mentioned in the Lilian Weng’s article, Harness Engineering for Self-Improvement, can be implemented practically using the tools like SuperQode. Self-improving coding agents require more than stronger models. They require a harness that can observe failures, preserve useful memory, constrain candidate changes, validate improvements, and reject unsafe modifications. Without that layer, agent behavior remains difficult to compare, reproduce, or improve in a controlled way. SuperQode approaches this problem by making the harness explicit and versionable. The same harness can define context policy, tool access, workflow, checks, eval splits, optimization boundaries, and candidate acceptance rules. This gives teams a practical way to improve agent behavior without handing the optimizer unrestricted control over permissions, sandboxing, or validation. The demo shows a minimal but complete loop: a failure is mined, a logbook entry is created, improvement evidence is exported, a narrow candidate is accepted, an unsafe candidate is rejected, and both outcomes are recorded.

In production, the same pattern can be extended with live held-out evals, cost and latency tracking, benchmark imports, and team-specific harness policies. As coding agents become a standard part of development workflows, harness engineering becomes an operational discipline. The teams that can measure and improve their harnesses will have more control over agent quality, cost, safety, and repeatability than teams that treat the harness as a fixed black box.

The harness design trends that everyone should know that emphasise on the harness optimization. The new era of Harness Engineering has began and Self Optimizing Open Harnesses is the future.