Skip to content

test(benchmarks): dedicated prewarmed sidecar per bench op#200

Merged
NathanFlurry merged 1 commit into
mainfrom
stack/test-benchmarks-dedicated-prewarmed-sidecar-per-bench-op-klmrulns
Jul 2, 2026
Merged

test(benchmarks): dedicated prewarmed sidecar per bench op#200
NathanFlurry merged 1 commit into
mainfrom
stack/test-benchmarks-dedicated-prewarmed-sidecar-per-bench-op-klmrulns

Conversation

@NathanFlurry

Copy link
Copy Markdown
Member

The latency matrix shared one VM across every op, making results
order-dependent (identical code drifted 2-3x across runs), letting the first
op eat one-time costs (isolate creation, bridge snapshot, wasm compilation),
and never recording WHICH sidecar binary was measured.

  • Per-op VM: guest/wasm/vmCmd lanes get a fresh, prewarmed, disposed-after VM
    (host lanes never touch a VM and run up front). Ops that need special VM
    options (external listeners' loopback-exempt ports) declare a prepareVm hook.
  • Documented warmup contract in prewarmBenchVm: trivial guest exec (isolate +
    snapshot), one native-baseline wasm exec (module compile), one command exec
    for command ops — before the op's own discarded warmup iterations.
    BENCH_COLD=1 skips it explicitly (loudly); BENCH_SHARED_VM=1 keeps the old
    engine as an escape hatch.
  • Binary provenance: every run prints and records sidecar path, profile,
    mtime, size in results JSON; run-benchmarks.sh builds release and pins it
    via the new resolveNodeRuntimeSidecarBinary hook.

Order-independence: tcp_echo solo vs in-family now 1.5% apart (was 2-3x).

The latency matrix shared one VM across every op, making results
order-dependent (identical code drifted 2-3x across runs), letting the first
op eat one-time costs (isolate creation, bridge snapshot, wasm compilation),
and never recording WHICH sidecar binary was measured.

- Per-op VM: guest/wasm/vmCmd lanes get a fresh, prewarmed, disposed-after VM
  (host lanes never touch a VM and run up front). Ops that need special VM
  options (external listeners' loopback-exempt ports) declare a prepareVm hook.
- Documented warmup contract in prewarmBenchVm: trivial guest exec (isolate +
  snapshot), one native-baseline wasm exec (module compile), one command exec
  for command ops — before the op's own discarded warmup iterations.
  BENCH_COLD=1 skips it explicitly (loudly); BENCH_SHARED_VM=1 keeps the old
  engine as an escape hatch.
- Binary provenance: every run prints and records sidecar path, profile,
  mtime, size in results JSON; run-benchmarks.sh builds release and pins it
  via the new resolveNodeRuntimeSidecarBinary hook.

Order-independence: tcp_echo solo vs in-family now 1.5% apart (was 2-3x).
@NathanFlurry

Copy link
Copy Markdown
Member Author

Stack for rivet-dev/secure-exec

Get stack: forklift get 200
Push local edits: forklift submit
Merge when ready: forklift merge 200

@railway-app
railway-app Bot temporarily deployed to secure-exec / secure-exec-pr-200 July 2, 2026 11:36 Destroyed
@NathanFlurry
NathanFlurry merged commit 85e82f0 into main Jul 2, 2026
0 of 2 checks passed
@NathanFlurry
NathanFlurry deleted the stack/test-benchmarks-dedicated-prewarmed-sidecar-per-bench-op-klmrulns branch July 2, 2026 11:36
@railway-app
railway-app Bot temporarily deployed to secure-exec / preview July 2, 2026 11:36 Inactive
@railway-app
railway-app Bot temporarily deployed to secure-exec / production July 2, 2026 11:36 Inactive
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant