bauplan agent parse-error study

Supporting evidence for BauplanLabs/bauplan#369 — "client.run() project-file parse errors carry no diagnostic detail".

Shared so the report can be verified rather than taken on trust. Every number below is derived from the traces in this repo by the included script:

python scripts/parse_error_report.py

No figure in this README is hand-typed.

runs/       15 per-turn agent traces (JSONL), one line per event,
            grouped by study run. Each run dir carries its own
            contract.yaml and meta.json.
contracts/  the study contracts — exact goals, models, cells, injected files
scripts/    the derivation script

Traces record every tool_call (with full arguments), every tool_result (with the verbatim string returned to the model), the turn number, and the execution mode.

Scope. These are the 3 study runs in which the error occurred, selected from a larger corpus of 67 real-mode traces across 13 runs. The 10 runs with zero occurrences are omitted for size — so the ratio below is within these runs, not a base rate. Corpus-wide it was 3 traces out of 67.

15 real-mode traces here. 3 hit failed to parse project file, 15 occurrences total. The worst single run hit it 11 times.

The string returned to the agent, in full:

{"status": "error", "error": "failed to parse project file"}

That is the entire diagnostic. No line number, no offending key, no parser detail. The model cannot tell which part of bauplan_project.yml was rejected, so its only available recovery is to rewrite the whole file and retry — which is the 11× loop above.

cell_1_skills_full injects Bauplan's own shipped skills wholesale into the system prompt, cloned from github.com/BauplanLabs/bauplan-skills:

• CLAUDE.md
• plugins/bauplan/skills/bauplan-data-pipeline/SKILL.md

The error occurs with those skills fully loaded. See contracts/bauplan_poc_skills_full.yaml.

The task in these contracts requires the agent to author a pipeline from scratch — "You must write your own pipeline code — nothing is pre-built for you" — for TPC-H Q3, on its own branch, then merge to a sandbox branch.

If your usual path starts from an existing project file or a guided workflow, the model never synthesises bauplan_project.yml cold, and this error class would not surface. That is a real difference in exercise conditions, and it is stated here rather than left for you to find.

1. failed to parse project file vs Failed to parse SQL. These are different failures with different causes. This repo also contains 3 traces with 5 SQL-parse occurrences — not part of issue #369, reported only so the numbers reconcile. An earlier internal count merged the two classes and overstated #369 by 5×; the script now reports them separately and refuses to combine them.

2. Real vs mock execution. The harness that produced these traces can run against a mock target. A mock target cannot produce a real parse error, so including mock runs would dilute any failure rate toward zero. Only real-mode traces were exported here, and scripts/parse_error_report.py refuses to run if it finds any non-real trace.

• Verification, not re-execution. The engine that produced these traces (harness-bench) is not public. The contracts are included so the exact goals, models, and cells are on the record and the runs can be repeated on request.
• Sanitisation. Local absolute paths were replaced with ~ before publishing. Trace and tool-call IDs are unaltered so the traces stay internally consistent — note that Gemini traces embed an opaque __thought__ reasoning signature inside tool_call_id. It is provider data, not a credential.
• Small n. 3 affected traces out of 67 corpus-wide. This documents that the failure mode exists, not how often it occurs in normal use.
• Trace format. Each tool_call is followed by its tool_result sharing the same turn and tool_call_id. The tool name is repeated on both so a result record is interpretable on its own — that repetition is deliberate, not duplicated data.

Ramona C. Truta — github.com/rctruta