Coding Agent Cost Benchmark

Compare coding agents on the metric that matters for self-hosting: cost per solved task.

The benchmark runs the same contamination-free DeepSWE tasks through each agent setup, grades every attempt with the task's own tests, and produces a report comparing success rate and cost. It's built to weigh self-hosted GLM-5.2 (on Modal, 8×B200) against Claude Opus across GLM's reasoning tiers.

Three studies, each 4 runs × the same 33 DeepSWE v1.1 tasks (132 attempts per setup). The 33 in each folder name is that task count — these are not benchmark versions, and all three ran the identical task list (tasks.txt is in every folder alongside per-run CSVs, billing, and the report):

Headline, cheapest to most expensive per completed task, under the observed concurrency:

Every self-hosted setting beat both Opus models on cost per completed task — when the endpoint stays busy (a GLM task alone on the GPU is ~$9.61). K3 at low reasoning solved the most tasks of anything tested, for ~6× less than Opus. Full numbers and method in each folder.

Self-hosted rows are the real Modal bill split across overlapping tasks by concurrency; Opus rows are Claude Code's own per-token cost. GLM ran on 8×B200, K3 on B300s (its default serving config spilled to a second container under our load).

The three GLM tiers are the only ones GLM-5.2's chat template distinguishes: reasoning_effort is either high or max (max = the default when unset), plus enable_thinking:false for no reasoning. opencode can't set chat_template_kwargs itself, so a small reasoning-proxy sidecar injects the tier and forwards to the endpoint (see Reasoning tiers).

1. Provision the GLM auto-endpoint on Modal — once — with ./setup_auto_endpoint.sh. (GLM setups only.)
2. Run the benchmark — one docker run — which writes raw per-run output + manifest.csv to runs/<run-id>/.
3. Report — python3 benchmark_progress_report.py runs/<run-id> — locally, producing progress_report.html + CSVs.

The GLM setups send inference to a GLM-5.2-FP8 auto-endpoint on Modal (8×B200). Provision it once:

pip install modal && modal setup          # authenticate the Modal CLI (writes ~/.modal.toml)
./setup_auto_endpoint.sh                  # idempotent: create the endpoint only if missing

setup_auto_endpoint.sh:

• ensures the Modal CLI is installed and authenticated (modal setup);
• enforces exactly one proxy token (wk-…/ws-…) that matches your .env — if none exists it creates one and prints the pair to copy into .env, then you re-run;
• creates the auto-endpoint for zai-org/GLM-5.2-FP8, reusing the pre-downloaded weights volume (glm-5-2-weights) so there's no multi-hundred-GB re-download;
• waits for provisioning (8×B200 cold start ≈ a few minutes);
• confirms MODAL_ENDPOINT — copy the endpoint URL from the Modal dashboard and use <url>/v1.

It's safe to re-run any time (reuses an existing endpoint + volume). Options: --name, --model, --volume, --wait-tries (see ./setup_auto_endpoint.sh --help). An Opus-only run needs no Modal endpoint — only ANTHROPIC_API_KEY.

pier drives the host Docker daemon (via the mounted socket) to build and run each task's own container, inject the agent CLI into it, and grade it. The image ships only python + the Docker CLI + pier + the orchestrator — the agent CLIs (opencode / claude-code) are installed by pier inside each task container.

docker pull ghcr.io/agencyenterprise/coding-agent-cost-bench:latest

DIR="$PWD/runs"                          # host-aligned dir: holds runs + the pier job tree
docker run --rm -p 80:80 \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -v "$DIR:$DIR" -e OUT_DIR="$DIR" \
  -e HOST_IP="$(hostname -I | awk '{print $1}')" \
  --env-file .env \
  ghcr.io/agencyenterprise/coding-agent-cost-bench:latest \
  --setups glm-default,glm-high,glm-nothink,opus \
  --tasks ytt-jsonpath-query-api \
  --runs 4 --jobs 10

Raw per-run output lands under runs/<run-id>/: one folder per run (<setup>__<task>__runN/ with output.log, reward.json, usage.json, model.patch) plus manifest.csv. Turn these into a report in step 3.

Why the host-aligned $DIR: pier runs inside this container but tells the host daemon to bind-mount its job tree (under $DIR/<run-id>/pier-jobs) into each task container. Those paths must exist at the same location inside and outside the container (docker-out-of-docker), so mounting $DIR to the same path and pointing OUT_DIR at it keeps the run and the job tree aligned in one mount.

Reporting is a local post-processing step — it reads the raw run folders, and pulls your real Modal bill (which needs your Modal account token), so it's kept out of the benchmark image:

python3 benchmark_progress_report.py runs/<run-id>     # → progress_report.html + CSVs
python3 verify_report.py             runs/<run-id>     # correctness gate (recommended)

benchmark_progress_report.py is self-contained (stdlib + modal for the bill). --no-billing skips the Modal bill pull and falls back to the modeled GPU rate. verify_report.py re-derives every number from the raw files and fails loudly on any mismatch.

Pulling from a remote box? ./pull_run_from_server.sh [RUN_ID] SSH-pulls the run dir and runs the report in one step. Configure REMOTE / REMOTE_RUNS in .env.

• progress_report.html — pass@k (task solved in ≥1 run) + pass@1 (per-run) success, and cost per solved task, per setup
• per_run.csv — one row per run: orchestration_s (full job) ⊇ session_s (agent session) ⊇ generation_s (GPU/API generation = cost basis), plus billed_usd/sole_usd, tokens, steps
• summary.csv — per-setup rollup
• deepswe_task_difficulty.csv — per-task complexity + pass rate
• billing.json — the actual Modal endpoint bill over the GLM-active window (unless --no-billing)

All from .env (--env-file); nothing is baked into the image (.env is git- and docker-ignored). Copy .env.example and fill in:

Create the proxy token with modal workspace proxy-tokens create (or let setup_auto_endpoint.sh print one). Create the account token at modal.com/settings/tokens or modal token new.

reasoning_proxy.py --router runs as a sidecar on port 80 inside the container. For glm-high and glm-nothink, opencode's baseURL points at http://$HOST_IP/<tier>/v1; the proxy strips the /<tier> prefix, injects the matching chat_template_kwargs (reasoning_effort:high or enable_thinking:false), and forwards to MODAL_ENDPOINT. glm-default talks to the endpoint directly (no proxy). A single sidecar serves every tier by URL path because the task egress proxy only permits ports 80/443.

• Claude Opus — priced from reported token usage.
• GLM (self-hosted) — the actual Modal bill for the auto-endpoint over the GLM-active window (billing.json), split across GLM runs by concurrency: each generating second is shared among the runs in flight then, so the per-setup totals reconcile to the real bill (never double-counted for parallelism). Without billing.json it falls back to a modeled hourly rate.

The report normalizes everything to cost per solved task, making the API model and the self-hosted tiers directly comparable.

Dockerfile                    one-`docker run` benchmark image (orchestrator + sidecar)
entrypoint.sh                 starts the reasoning-proxy sidecar, then the orchestrator
run_deepswe.py                orchestrator: fans pier over (setup × task × run) → raw runs + manifest.csv
reasoning_proxy.py            GLM reasoning-tier sidecar (router mode)
setup_auto_endpoint.sh        provision the GLM-5.2 Modal auto-endpoint (idempotent)
benchmark_progress_report.py  the report (run locally): pass@k / pass@1 + cost, real bill split by concurrency
verify_report.py              correctness gate for the report's numbers
pull_run_from_server.sh       pull a run from the server, then run the report
create_result.sh              name a run (runs/<name>/, local) and freeze it into results/<name>/
results/<name>/               committed result: task list + CSVs + billing + rendered report
runs/                         local scratch — raw per-run runs (gitignored)