Codex app-server client closes before long maintenance turn completes; Gateway restart also terminates active turn without graceful recovery #95547

Summary

During a long maintenance/update task initiated from a messaging channel, the active Codex-backed agent turn failed with:

codex app-server client closed before turn completed

This happened twice in the same maintenance flow:

1. During the original long update/check run.
2. During a later Gateway restart that was used to recover from a stuck/active memory reindex lock.

In both cases the Gateway itself was able to run afterward, but the user-facing turn was lost and the user only saw the generic failure message instead of a graceful status/follow-up.

Environment

• OpenClaw: 2026.6.9 (c645ec4)
• Codex CLI: 0.141.0
• Node.js: v24.16.0
• OS: Linux x86_64
• Channel type: messaging channel
• Gateway bind: loopback
• Runtime: Codex app-server via OpenClaw managed Codex project

What happened

A user request triggered a long-running maintenance task: OpenClaw update/checks, OS package checks, npm/pipx checks, Codex/Claude version checks, Gateway health checks, and memory checks.

The first long turn failed before a final response:

2026-06-21T07:33:36.896+00:00 warn agent/embedded {"subsystem":"agent/embedded"} {"threadId":"<thread-id>","turnId":"<turn-id>"} codex app-server client closed before turn completed
2026-06-21T07:33:37.015+00:00 error diagnostic {"subsystem":"diagnostic"} lane task error: lane=main durationMs=122307 error="Error: codex app-server client closed before turn completed"
2026-06-21T07:33:37.017+00:00 error diagnostic {"subsystem":"diagnostic"} lane task error: lane=session:<messaging-session> durationMs=122317 error="Error: codex app-server client closed before turn completed"
2026-06-21T07:33:37.033+00:00 error Embedded agent failed before reply: codex app-server client closed before turn completed

Later, memory search/status showed the semantic index was dirty and paused, and reindexing initially failed because a reindex lock was held:

2026-06-21T10:09:56.519+00:00 error Memory index failed: Memory reindex lock is held at <agent-state-dir>/openclaw-agent.sqlite.reindex-lock.sqlite; another reindex is active.
2026-06-21T10:09:57.253+00:00 info Memory index failed: Memory reindex lock is held at <agent-state-dir>/openclaw-agent.sqlite.reindex-lock.sqlite; another reindex is active.
Index error: Memory reindex lock is held at <agent-state-dir>/openclaw-agent.sqlite.reindex-lock.sqlite; another reindex is active.

A Gateway restart was performed to recover from the stuck/active lock. That restart also terminated the active Codex turn:

2026-06-21T10:14:40.097+00:00 info gateway {"subsystem":"gateway"} signal SIGTERM received
2026-06-21T10:14:40.100+00:00 info gateway {"subsystem":"gateway"} received SIGTERM; shutting down
2026-06-21T10:14:40.357+00:00 warn agent/embedded {"subsystem":"agent/embedded"} {"threadId":"<thread-id>","turnId":"<turn-id>"} codex app-server client closed before turn completed
2026-06-21T10:14:40.662+00:00 error diagnostic {"subsystem":"diagnostic"} lane task error: lane=main durationMs=540522 error="Error: codex app-server client closed before turn completed"
2026-06-21T10:14:40.663+00:00 error diagnostic {"subsystem":"diagnostic"} lane task error: lane=session:<messaging-session> durationMs=540526 error="Error: codex app-server client closed before turn completed"
2026-06-21T10:14:40.673+00:00 error Embedded agent failed before reply: codex app-server client closed before turn completed

The Gateway did come back successfully:

2026-06-21T10:14:47.913+00:00 info gateway {"subsystem":"gateway"} gateway ready

The memory index was later rebuilt successfully:

2026-06-21T10:30:29.683+00:00 info Memory index complete.

Expected behavior

For long-running Codex-backed turns, especially maintenance tasks:

• If codex app-server exits unexpectedly, OpenClaw should preserve enough task state to provide a useful follow-up or recovery message.
• If the Gateway is intentionally stopping/restarting, the active turn should ideally be marked as interrupted by Gateway shutdown, not reported as a generic Codex app-server failure.
• The user-facing message should distinguish between:
  • unexpected Codex app-server failure;
  • expected termination due to Gateway restart/SIGTERM;
  • replay-safe retry attempted/failed.
• If possible, long-running maintenance runs should be resumable or should emit a final synthesized status after Gateway returns.

Actual behavior

The user received generic failure messages like:

Codex app-server connection closed before this turn finished. OpenClaw retried once when the stdio turn was still replay-safe; please try again if this keeps happening.

The underlying Gateway was healthy afterward, but the original maintenance status was lost until the user manually asked for status again.

Impact

For maintenance/update operations, this makes the system look broken even when Gateway is still healthy or has restarted successfully. It also leaves the user without a reliable final report unless they manually ask again.

Notes

This may be two related issues:

1. Long Codex app-server turns can close before completion.
2. Gateway restart/SIGTERM during an active Codex-backed turn is surfaced as a generic app-server failure instead of a graceful interruption/recovery path.

The memory reindex lock issue may be separate, but it contributed to the recovery flow that triggered the second app-server interruption.