ci: spike -- Backend Tests shard 1 on the CodeBuild runner as an unprivileged user - #10419
ci: spike -- Backend Tests shard 1 on the CodeBuild runner as an unprivileged user#10419CrysisDeu wants to merge 1 commit into
Conversation
…unprivileged user
The CodeBuild-hosted runner executes jobs as root; the first attempt to
route the backend shards there failed 37-42 tests per shard on exactly
that (read-only refusals, root-owned-ancestor checks, PermissionError
semantics, and the code's own "provider execution is disabled for a root
gateway"), plus two image differences: jq 1.6 where the repo's jq
programs need 1.7, and a setup-python toolcache whose libpython needs
LD_LIBRARY_PATH -- which the tests strip when they spawn a sanitized
subprocess (rc 127).
New composite action .github/actions/run-as-runner: on a root runner it
creates `runner` (uid 1001, the hosted runner's user), hands it the
workspace and job temp, registers the toolcache lib dirs with ldconfig,
installs jq 1.7.1 and gh 2.100.0 (sha256-pinned), and installs
/usr/local/bin/ci-shell -- a `shell:` program that runs a step's script
as `runner` via `runuser -m` with the environment preserved. On
ubuntu-latest ci-shell is a plain bash passthrough, so the test steps
declare `shell: /usr/local/bin/ci-shell {0}` once and behave identically
on both runners. Steps that write $GITHUB_OUTPUT keep the default shell.
Spike scope: only `Backend Tests (3.12, 1)` is routed
(`matrix.group == 1`); shards 2-4 stay on ubuntu-latest and exercise the
passthrough. A diagnostics step on the CodeBuild shard prints the
effective user, libpython resolution, jq/gh versions, workspace owner and
whether `unshare NEWNS` is available. If shard 1 is green all four move
in a follow-up; if not, the remaining reds name what a custom image must
carry.
Design Review (Fable 5) — ✅ PASSDesign-level review of Design-Verdict: PASS A properly scoped spike: one shard, passthrough-identical on hosted runners, pinned binaries, and a one-line rollback — the right shape for de-risking the runner migration. [DESIGN-REVIEWED] 1746f11 |
Opus 4.8 Review — ✅ no blocking findingsReviewed Review detailsNo findings. The sole candidate (ci-shell hardcoding [OPUS-REVIEWED] 1746f11 Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
First Principles Review (Fable 5) — 🟡 CONCERNSPremise-level review of First-Principles-Verdict: CONCERNS docs/ci/ci-and-reviews.md still pins "those shards move only once a custom image … exists" — this PR ships the opposite for shard 1 and leaves that record standing. Not justified as shipped
What this change shipsInventory (7 items) — 6 justifiedIntent: get the backend test shards off the congested hosted queue by proving one shard runs green on the CodeBuild runner — an ADDITION (spike).
Watch
Subtractions
[FIRST-PRINCIPLES-REVIEWED] 1746f11 |
GPT 5.6 Review — 🔴 changes requested (blocking)GPT 5.6 found at least one blocking issue that must be resolved before merging This comment is updated in place on each push. BLOCKING -- .github/actions/run-as-runner/action.yml:27 -- jq 1.7.1 checksum is incorrect Adjudication (Opus 4.8) — is blocking on each finding proportionate?Network access to verify the published checksum is blocked in this environment, so I cannot independently confirm whether the hash is right or wrong. But that does not matter for the FLAG bar, because the triggering condition is not rare. Conditions confirmed from code opened this run:
Recovery path: none — a checksum mismatch fails the step every run until the constant is corrected; shard 1 never runs (self-reported, and consistent with the Rarity: none available. The failure path is the ordinary CodeBuild root-runner path, hit on every shard-1 execution, not an extreme or contradictory condition combination. A FLAG would require a rarity argument I cannot construct, and I could not verify the checksum this run — the FLAG record cannot be completed. Per the fenced contract, that is UPHOLD-FENCED. [ADJUDICATION] 1746f11 total=0 uphold=0 downgrade=0 [ADJUDICATION-FENCED] 1746f11 fenced=1 flagged=0 False positive or not applicable? A repository writer can comment: |
Spike result —
|
| check | result |
|---|---|
| effective user in test steps | uid=1002(runner) gid=1002(runner) (uid 1001 was already taken in standard:7.0; the action fell back to the next free uid — harmless) |
HOME / USER |
/home/runner / runner |
| python | toolcache 3.12.14; libpython resolves without LD_LIBRARY_PATH: True (the ldconfig step works — the earlier rc 127 class is gone) |
| jq / gh / git | jq-1.7.1 (installed by the action), gh 2.99.0 (already in the image), git 2.55.0 |
| workspace / temp owner | runner / runner |
unshare --mount --map-root-user |
Operation not permitted — unprivileged user namespaces are unavailable in the CodeBuild container. This settles two inventory rows: backend-test-sandbox and the e2e precondition cannot run there. |
hosted shards 2–4 through the same ci-shell (passthrough) |
shards 2 and 4 green, 3 still running at the time of writing — the wrapper is inert on ubuntu-latest |
Queue → start 204 s (20:40:02 → 20:43:26; longer than the 15–21 s seen on every other build so far — worth watching whether large builds provision slower), execution 42.1 min (hosted shard 1 on the last main run: 41 min), cost ≈ $0.84.
The 4 remaining failures
| test | reason from the log | class |
|---|---|---|
test/test_cli_server_more_coverage.py::TestStopViaService::test_explicit_port_bypasses_the_service |
`lsof` not found — cannot look up the gateway process on port 8123 |
env-missing (lsof is on ubuntu-latest, not on standard:7.0) |
test/test_app_backend.py::TestBootSpawnLatency::test_survival_check_exits_early_for_a_healthy_child |
"healthy child burned 1.60s of a 1.60s budget" — platform_compat.listening_pid_tool_available() is false without lsof, so the survival poll can never confirm the child and sleeps the whole window |
env-missing (same lsof) |
test/test_chat_send_agent_model_default.py::TestEagerSpawnDefaultModel::test_eager_session_starts_on_the_global_default |
Expected get_or_create to have been awaited once. Awaited 0 times. |
other / undetermined — not seen in the root run; _eager_spawn returned before creating the session; needs one rerun to tell deterministic-on-this-runner from flake |
test/test_chat_send_agent_model_default.py::TestEagerSpawnDefaultModel::test_eager_resolve_runs_off_the_loop_and_survives_stop_iteration |
same assertion | other / undetermined (same pair) |
No still-root-semantics failures remain: every read-only-refusal, root-owned-ancestor, PermissionError and "provider execution is disabled for a root gateway" test from the root run passes as runner.
What this means for wave 3
- Add
lsofto therun-as-runnertoolchain step (apt-get install -y lsof, or the ubuntu package pin) → 2 of 4 go away by construction. - The two
TestEagerSpawnDefaultModelfailures need one more run to classify; if they repeat only on CodeBuild, the cause is in_eager_spawn's early-return gates, not in the runner user model. - The sudo/custom-image question is answered: no custom image needed for the user model. (Whether CodeBuild honours a custom image's
USERremains untested — no image was involved.)
Summary
Spike, not the migration. Routes exactly one backend shard —
Backend Tests (3.12, 1)— to the CodeBuild-hosted runner (matrix.group == 1), running the test steps as an unprivileged user through a new composite action. Shards 2–4 stay onubuntu-latestand exercise the passthrough path of the same action, so the diff is a single set of steps for both runners. If shard 1 is green here, a follow-up PR moves all four; if it is not, the remaining failures tell us what a custom runner image would have to carry.Why
The pilot's first attempt to route the backend shards (#10328, revision 1) failed 37–42 tests per shard. Causes, all environment, none code:
runner(uid 1001). The suite asserts permission semantics root does not have —test_app_scaffold.py::TestWriteContainment::test_a_read_only_file_site_is_refused(write through 0o444 →DID NOT RAISE),test_governance_distribution.py::…agent_writable…(root owns every chain),test_computer_use_enable_state.py::…test_unreadable_file_is_disabled(root reads chmod 000) — andgithub_runner.validate_provider_executablerefuses root outright ("provider execution is disabled for a root gateway"), which takestest_github_runner.py::TestResolveGh::*,test_issue_radar_gh_bin.py::*andtest_source_providers.py::…strict_mode…with it.standard:7.0vs 1.7 on the hosted image —test_issue_triage_workflow.py(7 tests) uses 1.7 syntax.LD_LIBRARY_PATH. setup-python's downloaded toolcache build links libpython dynamically; the tests that spawn a sanitized subprocess (test_dev_fleet_backend_only_sync_skip.py, 5 tests) gotpython: error while loading shared libraries: libpython3.12.so.1.0(rc 127).AWS documents no non-root mode for the CodeBuild GitHub Actions runner, so this closes the gap inside the job instead of with a custom image.
What changed
.github/actions/run-as-runner/action.yml(new composite action). On a root runner:useradd -m -u 1001 runner,chown -R runner: $GITHUB_WORKSPACE $RUNNER_TEMP, register$RUNNER_TOOL_CACHE/Python/*/x64/libwithldconfig, install jq 1.7.1 and gh 2.100.0 (sha256-pinned release binaries), and install/usr/local/bin/ci-shell=runuser -m -u runner -- env HOME=/home/runner USER=runner LOGNAME=runner bash -eo pipefail "$1". On a non-root runner: install ci-shell as a bash passthrough and stop. Everything before it (checkout, setup-python, setup-uv,uv pip install --system) still runs as whatever user the runner is — that is fine, the tool cache only needs to be readable.ci.yml/backend-test:runs-onismatrix.group == 1 && needs.changes.outputs.linux_runner_large || 'ubuntu-latest'; theRun testsandStage shard coverage datasteps declareshell: /usr/local/bin/ci-shell {0}; a diagnostics step (CodeBuild only,runner.environment == 'self-hosted') prints the effective user, libpython resolution, jq/gh/git versions, workspace ownership, and whetherunshare NEWNSworks in the container (answers the e2e/sandbox question for the inventory too).Select test scopekeeps the default shell because it writes$GITHUB_OUTPUT.Not in this PR: docs (the pilot entry still says backend stays hosted — true until the follow-up), the other three shards.
How it was verified
Static only, per repo convention: YAML parses,
bash -non the action's script, brand gate clean againstorigin/main. The CI run on this PR is the experiment; result goes in a comment.Rollback
Delete the
runs-onexpression back toubuntu-latest; ci-shell is a passthrough there, so the action and theshell:lines are inert.UX CONCERNS
none (CI-only)