Harness²: Optimize AI Brokers With out Altering Fashions

When an AI agent fails to finish a process, upgrading the mannequin can appear to be the plain subsequent step. But an agent’s efficiency additionally depends upon the way it receives directions, makes use of instruments, and manages its work. Understanding these components can reveal alternatives to enhance reliability with the mannequin already in place.

Google Analysis’s Harness² examines that surrounding setup and its position in agent efficiency. I put in and ran the venture to discover the thought firsthand. On this article, I have a look at the analysis, the implementation, and what builders can be taught from it when constructing their very own brokers.

No single harness wins in all places!

A harness shapes how an AI agent behaves via its prompts, instruments, reminiscence, and management loop. However completely different harnesses excel at completely different duties. Claude Code performs higher on coding and spreadsheets, whereas Codex leads on net duties.

The takeaway: A hard and fast harness leaves efficiency on the desk.

Harness² goals to shut that hole by adapting at check time even for duties its designers by no means anticipated.

Why Self-Enhancing Prompts Fail?

Letting an agent rewrite its personal system immediate sounds promising. In observe, it could actually make efficiency worse.

The issue? Each added instruction comes at a price.

System prompts are processed repeatedly all through a process. As they develop, they eat extra tokens, leaving much less room for the precise work.

And when the token price range runs out, the agent could fail earlier than ending.

What it edits as a substitute

Harness² exposes eight parts, most of which keep dormant till one thing triggers them. That’s the level: they value nothing when unused, not like immediate textual content.

The excellence that issues: a rule in a immediate is recommendation. A rule within the management loop is enforced.

Two Fixes That Really Work

1. Stop countless analysis: The agent constructed a watchdog to trace its steps and power it to start out writing earlier than exhausting its token price range.
2. Catch incomplete spreadsheets: It blocked process completion till a script verified that no formulation had been lacking.

Why it really works: Not like immediate directions, these safeguards are enforced by code not simply the mannequin’s willingness to comply with them.

Suggest, Probe, Compose

The laborious drawback is realizing whether or not an edit helped. On a benchmark you will have a solution key. On an actual process you don’t.

The sign comes from variations between executions, not from a rating in opposition to a recognized reply. That’s what lets it run on duties it has by no means seen.

The probe step just isn’t ceremony. With out it, efficiency dropped by about 10%.

Letting the agent edit its personal harness and easily trusting the edit was examined, and it was worse than not enhancing in any respect.

Analysis

What the analysis reported

Examined on three skilled benchmarks throughout 4 model-harness configurations, utilizing GLM-5 and Gemini 3.5 Flash. A single step improved on the bottom harness in each configuration tried.

Two Outcomes Value Noticing

• Reminiscence isn’t all the time higher. ReasoningBank underperformed even the baseline on coding duties, suggesting that storing recollections alone could not enhance an agent’s efficiency.
• Construct costly, run low-cost. A harness developed with an costly mannequin additionally labored with a less expensive one probably lowering deployment prices.

Implementation

Putting in it

Linux, Python 3.12, uv. That is what I ran.

Instructions

git clone --depth 1 https://github.com/google-research/harness2.git
cd harness2

uv venv --python 3.12 .venv
. .venv/bin/activate
uv pip set up -e '.[render]'

Output:

• Repository is 15 MB. The set up pulled 8 packages: lxml, openpyxl, pillow, python-pptx, typing-extensions, xlsxwriter and dependencies.
• pyproject pins Python to >=3.12,<3.14. Earlier variations fail on the venv step.
• Seven CLI instructions are put in: harness2, harness2-demo, harness2-setup, harness2-check, harness2-judge, harness2-report, harness2-bridge.

Working the demo

No API keys, no cloud account. It runs each recursion modes with deterministic editors.

Command

harness2-demo --output runs/demo --k 2

Output:

The demo finishing each accounting duties.

  [accounting] 1/2 okay           ledger-1
  [accounting] 2/2 okay           ledger-2
Native demo: 2 duties accomplished; 0 failed. runs/demo/demo.json

What the run produced

A 2-task run writes 2,360 information holding about 265 KB between them. How a lot disk that truly occupies depends upon the filesystem: ext4 gave 13 MB as a result of every tiny file claims a 4 KB block, whereas APFS on macOS packed the identical run into 0.3 MB. Value realizing earlier than a benchmark run with a whole bunch of duties.

The place the 28 executions got here from

Measured from the run. Parallel retains express baselines; sequential doesn’t.

8 and 6 variants throughout 2 duties every provides 16 plus 12, which is the 28 the abstract reported. The names map onto the algorithm: inv for invariant edits, dom for domain-specific edits, agg or c for compositions.

The analysis places the associated fee at 4 to twenty full agent runs per process. My run landed at 6 to eight with okay=2, which sits on the backside of that vary.

What the free demo hides

That is the discovering I’ve not seen said anyplace else, and it issues when you plan to judge the venture by working the demo.

Measured byte counts from the demo run.

• All eight parts are scaffolded, however guardrails, long-term reminiscence and power steering had been written empty.
• 4 extra exist solely as placeholder READMEs: plugins, scripts, expertise, subagents.
• Solely two information carry content material: a 56-byte system immediate and a 243-byte script, an identical throughout each duties and each modes.
• Execution traces are 75 bytes; solutions are 13 bytes.

The demo proves the plumbing works. It doesn’t display the strategy.

The recursion loop runs, probes fireplace, compositions are produced, the library is written and expertise is recorded. However the deterministic editors by no means generate actual content material, so no harness is meaningfully rewritten. To see that you simply want mannequin credentials and an actual benchmark.

What the library does report

Every process writes an EXPERIENCE.md, and its headings present what the second loop retains:

# ledger-2
## Why this was chosen
## What every edit did
## Transferable lesson

That third heading is the entire level of harnessing-level recursion. My run produced 16 of those throughout modes and duties.

Inspecting a run

Run directories are massive and deeply nested. This pulls out the numbers that matter. Not a part of the repository.

inspect_run.py

"""
Examine a harness2 run listing. Written for this text, not a part of the repo.
Utilization: python inspect_run.py runs/demo
"""

import json
import pathlib
import sys
import collections


root = pathlib.Path(
    sys.argv[1] if len(sys.argv) > 1 else "runs/demo"
)


# --- headline numbers from the run abstract -------------------------

abstract = root / "demo.json"

if abstract.exists():
    d = json.masses(abstract.read_text())

    print("RUN SUMMARY")

    for okay in (
        "area",
        "n_tasks",
        "performed",
        "failed",
        "execution_count",
        "editor_calls"
    ):
        if okay in d:
            print(f"  {okay:<16} {d[k]}")

    print()


# --- what number of artefacts did a run really produce? ----------------

information = [
    p for p in root.rglob("*")
    if p.is_file()
]

total_mb = sum(
    p.stat().st_size for p in information
) / 1024**2

print(
    f"ARTEFACTS: {len(information)} information, {total_mb:.1f} MBn"
)


# --- execution variants, grouped by mode ---------------------------

print("EXECUTIONS PER TASK")

for mode_dir in sorted((root / "executions").glob("*")):

    duties = sorted(
        p for p in mode_dir.glob("*")
        if p.is_dir()
    )

    if not duties:
        proceed

    variants = sorted(
        p.title for p in duties[0].glob("*")
        if p.is_dir()
    )

    print(
        f"  {mode_dir.title:<16} "
        f"{len(variants)} variants: {', '.be part of(variants)}"
    )

print()


# --- which harness parts really received content material? ----------------

print("HARNESS COMPONENTS (bytes written)")

sizes = collections.defaultdict(set)

for p in root.rglob("harness/**/*"):

    if p.is_file():

        key = (
            str(p.relative_to(p.mother and father[1]))
            if p.dad or mum.title != "harness"
            else p.title
        )

        sizes[key].add(p.stat().st_size)


for title, vals in sorted(sizes.objects()):

    v = sorted(vals)

    flag = "  <- EMPTY" if v == [0] else ""

    size_display = (
        v[0]
        if len(v) == 1
        else f"{v[0]}-{v[-1]}"
    )

    print(
        f"  {title:<34} {size_display:>8}{flag}"
    )

print()


# --- the harnessing-level reminiscence -----------------------------------

exp = checklist(root.rglob("EXPERIENCE.md"))

if exp:

    print(
        f"EXPERIENCE.md x{len(exp)}, part headings:"
    )

    for line in exp[0].read_text().splitlines():

        if line.startswith("#"):

            print("  ", line)

Output:

One gotcha price realizing. Re-running the identical output listing skips duties that already completed, which is the supposed resume behaviour, however it additionally rewrites demo.json with execution_count and editor_calls at zero. The artefacts from the primary run survive, so the variant counts and element sizes above are nonetheless correct. Level –output at a contemporary listing if you need the counts.

RUN SUMMARY
  area           accounting
  n_tasks          2
  performed             2
  failed           0
  execution_count  28
  editor_calls     32

EXECUTIONS PER TASK
  demo-oc-agg2     8 variants: agg1, agg2, base1, base2, dom1, dom2, inv1, inv2
  demo-oc-seq2     6 variants: c1, c2, dom1, dom2, inv1, inv2

HARNESS COMPONENTS (bytes written)
  LongTermMEMORY.md                         0  <- EMPTY
  guardrails.md                             0  <- EMPTY
  plugin/README.md                        154
  scripts/README.md                       111
  scripts/summarize.py                    243
  expertise/README.md                        175
  sub_agents/README.md                    177
  systemprompt.md                          56
  tool_descriptions/tool_guidance.md        0  <- EMPTY

Shifting to an actual benchmark

Mannequin-backed runs want Node.js 20+, the Google Cloud CLI and Vertex AI credentials. LAB additionally wants Pandoc and Poppler; WorkBuddy wants Docker.

Instructions

cp .env.instance .env            # add your Vertex AI venture and mannequin IDs
gcloud auth application-default login

harness2-setup lab
harness2-check --bench lab

harness2 --bench lab --k 1 --mode parallel 
  --only-domains antitrust-competition --max-tasks 1

harness2-judge  --bench lab --k 1 --mode parallel
harness2-report --bench lab --k 1 --mode parallel

• Judging and reporting should match the refinement run on benchmark, executor, subset, okay, mode and seed, or the numbers won’t line up.
• Add --dry-run to preview process choice with out spending something.
• Begin with --max-tasks 1. Every process fires a number of executions.
• Use --substrate cx for the Codex executor as a substitute of the default OpenCode.

Conclusion

Harness² is a working implementation of a transparent argument. Maintain the weights fastened, optimise the scaffolding, and use contrasts between runs as a substitute of ground-truth labels to determine what to maintain.

The analysis behind it’s extra attention-grabbing than the headline numbers. Immediate-stuffing made issues worse. Skipping validation made issues worse. A competing reminiscence baseline made coding worse. These adverse outcomes are what make the constructive ones credible.

The set up is clear and the demo runs in seconds with out credentials, which makes it low-cost to confirm the equipment. Simply don’t mistake that demo for the strategy: the parts it fills in are stubs, and the half the paper is about solely exhibits up as soon as a mannequin is wired in.

Learn extra: Easy methods to Use Jev AI: A Fingers-On Information to TypeSafe’s System One Mannequin

Incessantly Requested Questions

Hello , I’m Sree Vamsi a passionate Information Science fanatic presently working at Analytics Vidhya. My journey into knowledge science started with a curiosity for uncovering insights from advanced knowledge and has developed into constructing end-to-end Generative AI functions, RAG pipelines, agentic AI workflows, and multi-agent methods that clear up real-world enterprise issues.