Harness Bench

A compact scorer and methodology starter for reviewing normalized run artifacts from agent harnesses.

The current implementation does not run harnesses, guarantee equivalent conditions, or produce a defensible public leaderboard. Its value is narrower: it makes task, rubric, run, cost, latency, and result assumptions explicit enough to inspect and extend.

python -m pip install -e .
harness-bench score examples/sample_run.json

The scorer reads a structured artifact and applies the repository’s current weighted rubric. It can support repeatable demonstrations and schema-level review. It does not independently verify that:

• the task was executed from the correct starting state;
• two harnesses had equivalent model, context, tools, permissions, or confirmation rules;
• the supplied latency, cost, or quality values were measured correctly;
• the evaluator was valid or blind to harness identity;
• missing and failed runs were retained;
• the sample supports a broader claim.

Those responsibilities belong to the experiment and evidence package around the scorer.

A harness comparison changes more than an interface. Results may depend on:

• harness and version;
• model and provider version;
• system, project, and user instructions;
• repository and retrieval context;
• tools and effective permissions;
• approval and confirmation policy;
• memory, cache, and session behavior;
• network, filesystem, and execution environment;
• task fixture and reset procedure;
• evaluator and rubric;
• run count and operator intervention.

When several variables change, the result may still be useful, but the claim must be limited accordingly.

See methodology/comparability-contract.md.

A benchmark task should define:

• immutable starting state;
• user request;
• allowed and prohibited actions;
• expected artifact or state change;
• success and failure rules;
• timeout and resource budget;
• evidence required for scoring;
• cleanup and reset;
• known ambiguity and valid alternative solutions.

A prose prompt alone is not a reproducible task.

Do not rely on one total score. Report the components and critical failures.

A weighted score should never compensate for a critical prohibited action or silently dropped failure.

Comparisons should record whether systems received equivalent:

• file, network, connector, and code-execution authority;
• tools and tool versions;
• data and repository context;
• tests and reference outputs;
• hidden or provider-managed instructions where known;
• confirmation and human-approval behavior;
• persistent memory and prior context.

A harness with broader authority or richer context is not necessarily reasoning better; it may be receiving a different treatment.

For repeated evaluation:

• predefine run count and task set;
• reset fixtures and relevant state;
• record sampling / seed policy where available;
• randomize or counterbalance order when services can drift;
• retain setup failures, timeouts, refusals, and invalid outputs;
• separate harness retries from operator rescue;
• preserve raw artifacts and manifests.

Repeated attempts on one task are clustered observations and should not be presented as broad independent evidence.

Prefer executable checks for observable state. When judgment is required:

• define the rubric before seeing outputs;
• blind reviewers to harness identity where feasible;
• randomize presentation order;
• report reviewer disagreement and adjudication;
• calibrate model judges against human review;
• record judge model, prompt, and version.

A model judge is part of the measurement system and can favor style, verbosity, order, or model family.

Specify whether measurements include:

• setup and environment preparation;
• confirmations and clarification;
• retries and recovery;
• failed runs;
• cached versus uncached calls;
• tool and external-service cost;
• human review time;
• pricing date.

Cost per successful task is useful only alongside quality, constraint, and recovery outcomes.

The current repository supports the first two levels when the surrounding run records are complete. It does not support general provider or product rankings.

The most valuable next work is:

1. versioned system-under-test and task manifests;
2. repeated-run ingestion with missing/failure preservation;
3. paired task-level comparison rather than only aggregate ranking;
4. executable constraint and artifact checks;
5. variance and uncertainty reporting;
6. blinded review workflow for subjective rubrics;
7. synthetic task packs with reset scripts and known alternative solutions.

A larger leaderboard without these controls would make the repository look more impressive while making its conclusions less trustworthy.

Only publish synthetic tasks, fictional outputs, and sanitized run artifacts. Do not publish private prompts, proprietary benchmark tasks, customer or employer data, confidential tool output, credentials, connector details, internal cost accounts, or comparison claims not supported by the recorded method.

This is a working prototype scorer plus a comparability methodology. It is not an independent benchmark lab, certified evaluation suite, provider endorsement, or definitive ranking of models, IDEs, or agent harnesses.

Maintained by Sima Bagheri.