Skip to content

fix(gateway): preserve Control UI turns across restarts - #106151

Merged
steipete merged 15 commits into
mainfrom
fix/sqlite-restart-resume
Jul 13, 2026
Merged

steipete merged 15 commits into
mainfrom
fix/sqlite-restart-resume

Conversation

@steipete

@steipete steipete commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

Related: #87808

AI-assisted implementation and review.

What Problem This Solves

Fixes an issue where a Control UI user turn accepted immediately before a Gateway restart could disappear from the conversation without a final reply or a visible terminal outcome. The client received an acknowledgement, but the Gateway did not yet own enough durable state to resume that accepted work after startup.

Why This Change Was Made

Control UI admission now records the user transcript turn and its restart-recovery claim atomically before acknowledging the send. Startup recovery adopts that durable claim through a single-use, identity-bound runtime handoff, so the normal agent path executes the accepted turn exactly once without persisting the user message twice.

The canonical state remains in the existing per-agent openclaw-agent.sqlite database and its session_entries rows. Recovery uses indexed lifecycle status and claim fields; it does not add an unsent side table, JSON sidecar, dual-write path, or runtime file fallback. Terminal cancellation/failure state is retained so completed or intentionally stopped work is not replayed, while safe unadopted claims can be retried.

Automatic replay is intentionally limited to plain Control UI user turns whose work can be reconstructed safely. Command/hook turns and other unsafe admission shapes retain explicit terminal behavior instead of being guessed or replayed.

User Impact

Users can restart the Gateway during a supported in-flight Control UI turn and expect the accepted turn to remain visible and resume once after reconnect. This removes a message-loss window without creating a second queue or a separate source of session truth.

The change also consolidates restart ownership into focused session, Gateway, and agent helpers, deletes duplicated lookup/persistence policy, and documents the recovery contract and boundaries.

Evidence

  • Live Crabbox user-path scenario: gateway-restart-inflight-run passed with a real Gateway process restart and mock OpenAI provider (run_20624fe8f162).
  • Focused matrix: 1,297 tests passed across 13 session, Gateway, agent, protocol, runtime-config, and reply-runner files (run_7089b0036e9d).
  • Exact-head follow-up after the adjacent upstream transcript/agent/auto-reply cleanup: core TypeScript graph plus 359 storage/admission/Gateway/recovery tests passed (run_a669fe9550cd).
  • Complete changed-surface gate passed: formatting, generated SQLite schema baseline, Plugin SDK baseline, production/core-test/UI TypeScript graphs, core/scripts lint, import-cycle and database-first storage guards, and selected platform tests (run_7341b9ec5383). SwiftLint remained owned by macOS CI because the Crabbox host was Linux.
  • Last rebased base: 6df498c3cd1000b3f40f19cbd9cf63d6a1fb0c65. git range-diff reports all 12 commits patch-identical across the final rebases; git diff --check passes.

@openclaw-barnacle openclaw-barnacle Bot added docs Improvements or additions to documentation app: web-ui App: web-ui gateway Gateway runtime agents Agent runtime and tooling size: XL maintainer Maintainer-authored PR labels Jul 13, 2026
@steipete
steipete force-pushed the fix/sqlite-restart-resume branch 2 times, most recently from 49451e0 to 98db2ce Compare July 13, 2026 07:40
@steipete steipete self-assigned this Jul 13, 2026
@steipete
steipete marked this pull request as ready for review July 13, 2026 08:00

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 98db2ceb43

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread CHANGELOG.md Outdated
@@ -52,6 +52,7 @@ Docs: https://docs.openclaw.ai
- **Plugin SDK API baseline:** cover every public entrypoint, preserve complete declaration shapes without source-line churn, and run baseline and export-surface guards from changed-file validation.
- **Official plugin beta compatibility:** keep the exact beta.5 session-store helper imports working over SQLite through the documented deprecation window, preventing installed Codex and Feishu plugins from failing during package acceptance and upgrades. (#105287) Thanks @vincentkoc.
- **SQLite terminal session recovery:** track physical transcript mutation time in the agent database so killed or timed-out main sessions rotate when transcript writes outlive the registry update, while preserving legacy transcript mtimes during doctor import.
- **Control UI restart recovery:** commit eligible user turns and running-session metadata to SQLite before acknowledging them, resume interrupted work through one durable dispatch identity, suppress late outbox duplicates, and show an interruption notice in WebChat history when a turn cannot be resumed. (#95141) Thanks @openperf for the WebChat notice fallback contribution.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Remove the non-release changelog entry

This change adds a regular fix entry to CHANGELOG.md, but the root AGENTS.md states that CHANGELOG.md is release-only and that release generation owns it. Because the release flow derives changelog entries from merged PRs/direct commits and requires the release SHA delta to be changelog-only, keeping this manual entry in a non-release fix risks duplicate or stale release notes and breaks the release-process invariant; move this context to the commit/PR release-note text and let release generation add the changelog line.

Useful? React with 👍 / 👎.

@openclaw-barnacle openclaw-barnacle Bot added the scripts Repository scripts label Jul 13, 2026
@steipete
steipete force-pushed the fix/sqlite-restart-resume branch from 3f86158 to c8a2524 Compare July 13, 2026 08:38
@clawsweeper

clawsweeper Bot commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

Codex review: needs real behavior proof before merge. Reviewed July 13, 2026, 4:48 AM ET / 08:48 UTC.

Summary
This PR atomically persists accepted Control UI turns and restart-recovery claims, then resumes eligible interrupted work exactly once through session-bound Gateway and agent recovery paths.

PR surface: Source +1941, Tests +2614, Docs +12, Generated +6, Other +5. Total +4578 across 67 files.

Reproducibility: yes. at source level: the linked report describes the restart timing window, and the patch clearly moves acknowledgement behind durable transcript and claim admission. A failing current-main run was not independently established during this review.

Review metrics: 3 noteworthy metrics.

  • Stored schema evolution: v6→v7; 1 nullable column and 1 partial index added. Persistent per-agent databases change during upgrade, requiring migration review beyond ordinary green tests.
  • Backend protocol surface: 1 optional expected-session parameter added. The additive field must stay restricted to identity-bound backend continuations.
  • Durable recovery metadata: 3 new persisted ownership/idempotency fields. Request fingerprints, source-run ownership, and terminal tombstones become part of session-state compatibility.

Stored data model
Persistent data-model change detected: database schema: src/state/openclaw-agent-db-session-migrations.ts, database schema: src/state/openclaw-agent-db.test.ts, database schema: src/state/openclaw-agent-schema.generated.ts, database schema: src/state/openclaw-agent-schema.sql, serialized state: src/agents/command/session.ts, serialized state: src/agents/embedded-agent-runner/run/attempt.spawn-workspace.context-engine.test.ts, and 47 more. Migration or upgrade compatibility proof is recorded; maintainers should verify it before merge.

Merge readiness
Overall: 🦐 gold shrimp
Proof: 🦐 gold shrimp
Patch quality: 🐚 platinum hermit
Result: blocked until stronger real behavior proof is added.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

  • Attach a redacted Control UI transcript, recording, live output, or logs showing acceptance, Gateway restart, reconnect, and exactly-once completion.
  • [P1] Complete or rerun the required exact-head checks after the final force-pushed commit.

Proof guidance:

  • [P1] Needs stronger real behavior proof before merge: The PR names a live Crabbox restart run but does not expose an inspectable transcript, recording, log excerpt, or linked artifact showing the accepted turn remains visible and resumes once; add redacted evidence, then update the PR body for automatic re-review or ask a maintainer to comment @clawsweeper re-review.

Mantis proof suggestion
A short independent Web UI chat capture can directly prove the acknowledged-turn, restart, reconnect, and exactly-once resume sequence. A maintainer can ask Mantis to capture proof by posting this exact PR comment:

@openclaw-mantis web UI chat proof: reproduce a Gateway restart immediately after an accepted Control UI turn and capture redacted evidence that the turn remains visible and resumes exactly once.

Risk before merge

  • [P1] The v6-to-v7 startup migration scans and backfills every persisted session-entry row, so rollout behavior on large or unusual user databases needs explicit upgrade acceptance.
  • [P1] The recovery state machine is exact-once-sensitive: an ownership, handoff, or tombstone regression could replay a turn twice, suppress it, or associate it with stale session state despite otherwise green CI.
  • [P1] The new optional Gateway protocol field must remain backend-only; treating it as a general client session-selection parameter would weaken its identity and lifecycle assumptions.
  • [P1] The PR body names validation run IDs but does not include directly inspectable after-fix logs, transcript output, recording, or linked artifact demonstrating the visible restart outcome.

Maintainer options:

  1. Prove upgrade and visible recovery (recommended)
    Require an inspectable redacted restart transcript or recording plus exact-head migration checks, then land the coordinated state machine together.
  2. Accept the supplied run summaries
    Maintainers may accept the named Crabbox runs and automated migration coverage as sufficient evidence while explicitly owning the rollout risk.
  3. Defer the complete change
    If the current release window cannot absorb persistent-schema and exact-once risk, pause the whole PR rather than merging partial transcript or recovery pieces.

Next step before merge

  • [P1] A maintainer should review the persistent-schema and exact-once rollout risk and require directly inspectable after-fix behavior proof before merge.

Security
Cleared: No concrete security or supply-chain regression was found; the sensitive continuation parameter is guarded as backend-only and uses a single-use identity-bound handoff.

Review details

Best possible solution:

Merge the atomic admission, migration, recovery scan, identity-bound handoff, and terminal non-replay behavior as one coordinated change only after exact-head checks pass and visible proof demonstrates fresh-install, v6-upgrade, resumed-once, and stopped-never-replayed paths.

Do we have a high-confidence way to reproduce the issue?

Yes at source level: the linked report describes the restart timing window, and the patch clearly moves acknowledgement behind durable transcript and claim admission. A failing current-main run was not independently established during this review.

Is this the best way to solve the issue?

Yes. Keeping the transcript and recovery claim atomic in the canonical per-agent SQLite database and adopting it through a single-use session-bound handoff is safer than adding a second queue, sidecar, dual-write path, or runtime fallback.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against a857c4430049.

Label changes

Label changes:

  • add P1: The PR addresses a real user-facing workflow where an acknowledged Control UI turn can silently disappear during a Gateway restart.
  • add merge-risk: 🚨 compatibility: The PR bumps the persistent agent database schema and backfills existing session-entry rows during upgrade.
  • add merge-risk: 🚨 message-delivery: Incorrect recovery ownership could duplicate, suppress, or fail to surface an acknowledged Control UI turn.
  • add merge-risk: 🚨 session-state: The patch changes durable run claims, terminal tombstones, lifecycle status, and transcript ownership across restarts.
  • add rating: 🦐 gold shrimp: Overall readiness is 🦐 gold shrimp; proof is 🦐 gold shrimp and patch quality is 🐚 platinum hermit.
  • add status: 📣 needs proof: The PR needs real behavior proof before ClawSweeper can clear the contributor ask. Needs stronger real behavior proof before merge: The PR names a live Crabbox restart run but does not expose an inspectable transcript, recording, log excerpt, or linked artifact showing the accepted turn remains visible and resumes once; add redacted evidence, then update the PR body for automatic re-review or ask a maintainer to comment @clawsweeper re-review.

Label justifications:

  • P1: The PR addresses a real user-facing workflow where an acknowledged Control UI turn can silently disappear during a Gateway restart.
  • merge-risk: 🚨 compatibility: The PR bumps the persistent agent database schema and backfills existing session-entry rows during upgrade.
  • merge-risk: 🚨 message-delivery: Incorrect recovery ownership could duplicate, suppress, or fail to surface an acknowledged Control UI turn.
  • merge-risk: 🚨 session-state: The patch changes durable run claims, terminal tombstones, lifecycle status, and transcript ownership across restarts.
  • rating: 🦐 gold shrimp: Overall readiness is 🦐 gold shrimp; proof is 🦐 gold shrimp and patch quality is 🐚 platinum hermit.
  • status: 📣 needs proof: The PR needs real behavior proof before ClawSweeper can clear the contributor ask. Needs stronger real behavior proof before merge: The PR names a live Crabbox restart run but does not expose an inspectable transcript, recording, log excerpt, or linked artifact showing the accepted turn remains visible and resumes once; add redacted evidence, then update the PR body for automatic re-review or ask a maintainer to comment @clawsweeper re-review.
Evidence reviewed

PR surface:

Source +1941, Tests +2614, Docs +12, Generated +6, Other +5. Total +4578 across 67 files.

View PR surface stats
Area Files Added Removed Net
Source 48 3059 1118 +1941
Tests 13 2831 217 +2614
Docs 1 24 12 +12
Config 0 0 0 0
Generated 3 7 1 +6
Other 2 5 0 +5
Total 67 5926 1348 +4578

What I checked:

Likely related people:

  • steipete: The branch history shows sustained ownership of the coordinated Gateway, session, agent, migration, and recovery-state implementation across fourteen commits; current-main blame provenance could not be independently completed because read-only shell execution failed before startup. (role: recent restart-recovery area contributor; confidence: low; commits: 20fa93661075, fcc1f6809153, c8a2524a14da; files: src/gateway/server-methods/chat.ts, src/gateway/server-methods/chat-restart-recovery.ts, src/agents/main-session-restart-recovery.ts)
What the crustacean ranks mean
  • 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
  • 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
  • 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
  • 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
  • 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
  • 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
  • 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works
  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c8a2524a14

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +267 to +270
const dispatchResult = await callGateway<{ runId: string; status?: unknown }>({
method: "agent",
params: agentParams,
timeoutMs: 10_000,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Dispatch recovery calls as backend clients

When restart recovery resumes a durable Control UI turn, this agent RPC includes expectedExistingSessionId, but the call does not set mode: GATEWAY_CLIENT_MODES.BACKEND. callGateway defaults callers to CLI mode (src/gateway/call.ts:1014), and the agent handler rejects expectedExistingSessionId unless the client mode is BACKEND (src/gateway/server-methods/agent-expected-session.ts:29), so every automatic recovery dispatch through this path is rejected before it can resume the accepted turn.

Useful? React with 👍 / 👎.

@clawsweeper clawsweeper Bot added rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. P1 High-priority user-facing bug, regression, or broken workflow. merge-risk: 🚨 compatibility 🚨 May break existing users, config, migrations, defaults, or upgrade paths. merge-risk: 🚨 message-delivery 🚨 May drop, duplicate, misroute, suppress, or wrongly target messages. merge-risk: 🚨 session-state 🚨 May lose, corrupt, stale, or mis-associate session, agent, or context state. labels Jul 13, 2026
@steipete
steipete force-pushed the fix/sqlite-restart-resume branch from c8a2524 to 572925c Compare July 13, 2026 09:06
@steipete
steipete merged commit bafabba into main Jul 13, 2026
102 of 103 checks passed
@steipete
steipete deleted the fix/sqlite-restart-resume branch July 13, 2026 09:09
@steipete

Copy link
Copy Markdown
Contributor Author

Merged via squash.

github-actions Bot pushed a commit to Desicool/openclaw that referenced this pull request Jul 14, 2026
)

* fix(gateway): preserve Control UI turns across restart

* perf(sessions): index restart recovery state

* fix(sessions): avoid replaying pre-admitted turns

* fix(sessions): index per-entry recovery status

* fix(sessions): guard restart failure notices

* fix(sessions): preserve terminal claims across admissions

* fix(sessions): retry failed recovery notices

* fix(sessions): harden restart retry ownership

* fix(gateway): tighten restart recovery typing

* fix(sessions): close restart recovery races

* refactor(sessions): split restart recovery ownership

* chore(sessions): clean restart recovery lint

* chore: drop release-owned changelog entry

* fix(gateway): satisfy restart recovery guards

* test(gateway): cover Control UI durable admission
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agents Agent runtime and tooling app: web-ui App: web-ui docs Improvements or additions to documentation gateway Gateway runtime maintainer Maintainer-authored PR merge-risk: 🚨 compatibility 🚨 May break existing users, config, migrations, defaults, or upgrade paths. merge-risk: 🚨 message-delivery 🚨 May drop, duplicate, misroute, suppress, or wrongly target messages. merge-risk: 🚨 session-state 🚨 May lose, corrupt, stale, or mis-associate session, agent, or context state. P1 High-priority user-facing bug, regression, or broken workflow. rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. scripts Repository scripts size: XL status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant