test(runner): record portable suite observations - #46

Summary

The portable runner executes fixed, Bash 3.2-compatible waves, but it records no per-suite timing evidence. This PR adds an opt-in --summary PATH JSON report and has CI keep one per interpreter. The scheduler and the pass criteria are unchanged.

What changed

• scripts/run-tests.bash --summary PATH writes:
  • per suite: path, passed / failed / skipped (OS rule), whole-second duration, child exit status and reason;
  • per run: target and host OS, $BASH_VERSION, source HEAD (null for exported trees), dirty and jobs.
• The file is written atomically (mktemp + mv) even when a child fails. Missing or corrupt status, duration or log evidence fails closed and is recorded with a reason.
• JSON strings are quoted with Bash builtins only (no jq/Python), byte-wise under LC_ALL=C.
• Discovery uses git ls-files -z, so quoted non-ASCII paths are handled. Exported tarballs fall back to printf '%s\0' tests/*.bash.
• CI writes test-summary-current.json and, on macOS, test-summary-bash32.json, and uploads them with if: always() as suite-observations-<os>.
• CONTRIBUTING.md, CHANGELOG.md, and new invariants in tests/runner.bash and tests/workflows.bash.

Why

Before proposing any change to the wave scheduler, we need matching per-suite observations, from the same source, fixtures, runner and interpreter class. These are coarse measurements, not a claimed saving.

How it was tested

• Runner fixtures cover:
  • serial and parallel outcomes, all selected logs after child failures, signals and literal names;
  • quoted and backslash JSON strings, timings and duplicate requests;
  • exported sources, missing or corrupt markers, and summary write failures;
  • (new) relative summary paths.
• Independent review checks:
  • json_string round-trips UTF-8 (café, 日本), ", \, TAB and \x01 exactly through Python's json.loads, under both C and C.UTF-8;
  • the summary arrays are only written in the parent shell.
• Full suite on head 386ddb2, as a non-root user, with --summary: 31/31 passed. The generated JSON parsed, with head 386ddb2, dirty: false and 31 suites.
• ShellCheck 0.11.0, shfmt 3.13.1 (-i 2 -ci -bn), bash -n and tests/workflows.bash are clean.
• Hosted CI on 386ddb2: all 10 jobs and all 11 checks green, including native macOS Bash 3.2 and Windows Bash/PowerShell. Previous qualification: run 37375586671 on 4f03846.

Review follow-ups

• 386ddb2 fix(runner): resolve a relative --summary path against the caller. The runner cds to the repository root before writing, so --summary out.json used to land in the checkout (as an untracked file) instead of next to the caller. A relative path is now anchored to $PWD before the cd; /… and Windows C:/… / C:\… paths are kept as given. The new regression in tests/runner.bash fails without the fix.

Decisions taken

• Durations stay at whole-second SECONDS resolution, to stay portable to Bash 3.2. No sub-second clock dependency was added.

Known limitations & follow-ups

• dirty uses git diff --quiet HEAD --, which ignores untracked files. Untracked suites can't run anyway (discovery uses ls-files), but untracked fixtures could still change a result.
• No scheduler change, threshold relaxation or quantified CI saving is claimed.

Merge order

#46 → #47, then #45. #47 is stacked on this branch: merge this first, then retarget #47 to main.

• ci: cancel superseded review runs #45 edits the same spots in CHANGELOG.md and CONTRIBUTING.md (and touches ci.yml and tests/workflows.bash in other hunks).
• git merge-tree shows only text conflicts in those two docs. Whichever lands later resolves them at merge time by keeping both entries.
• Independent of fix(project): resolve Go pins past unrelated tool manifests #44, refactor(exec): share the user-command execution contract #48, fix(sudo): keep line endings when replaying captured command output #49 and build(deps): bump github/codeql-action/upload-sarif from 4.37.9 to 4.38.2 #41.