systrap: fail-stop sentry on stuck context - #14201

A systrap task can remain in `sleepOnState` forever when its stub stops responding. The existing path sends a single interrupt and logs repeated warnings, but it has no retry and no escape. Container teardown can then block indefinitely in `WaitExited`, leaving the sandbox and its control RPCs stuck.

We hit this in a production Kubernetes cluster. Multiple gVisor pods remained Terminating for days, on both release-20260427.0 and release-20260714.0. In one affected sandbox, both `runsc kill` and `runsc debug --stacks` hung while the sentry stayed alive. Systrap `[exe]` stubs and node load continued to accumulate even though CPU use remained low. Killing the sentry process tree cleared the pod and returned node load to normal immediately. Full forensics in #14408.

#14405 is an independent production report of the same hang, with the sentry stacks we could not capture: a task goroutine parked 7,818 minutes in `sleepOnState` on the ThreadContext.State futex after one missed SIGUSR1, `Kernel.Pause()` blocked behind it so the URPC handler for `runsc kill --all` never returns. On the shim side 45,646 goroutines piled up on `Init.mu` behind the hung kill, growing containerd and kubelet memory until node OOM.

The fix, staged from least to most disruptive:

1. Because #14405 shows a single interrupt can simply be lost, the interrupt is resent on every checkup wakeup instead of once. A lost signal recovers the workload with no visible damage.
2. If the stub stays unresponsive through repeated interrupts for the full stuck-context deadline (30 seconds), all goroutine stacks are dumped via `log.TracebackAll`. Stacks are unobtainable from outside a wedged sandbox, so the escalation self-documents what #14405 had to capture by hand.
3. Only the stuck subprocess is then killed, through the same path `NotifyInterrupt` already uses when a stub is gone (mark dead, `ContextStateUnexpectedDeath`, kill the syscall thread). Healthy subprocesses in the sandbox are unaffected, and the task goroutine returns instead of blocking `Kernel.Pause()` forever.

`sleepOnStateWithTimeout` returns a typed error and stays side-effect free; `sleepOnState` does the traceback and the kill. A stub that is already gone takes the quiet `NotifyInterrupt` path (warn + kill subprocess, no stack dump). Resent interrupts do not extend the stuck deadline; a regression test covers that.

Tests:

- `bazel test //pkg/sentry/platform/systrap:systrap_test`
- `bazel build //runsc:runsc`
- Stuck, stub-gone, repeated-interrupt, and recovered-context cases covered on the typed error value.

Fixes #14408. Related to #14405 and #12209.

Assisted-by: Claude Code
FUTURE_COPYBARA_INTEGRATE_REVIEW=#14201 from NahumLitvin:fix/systrap-stuck-context d119f5c
PiperOrigin-RevId: 974017442

A systrap task can remain in `sleepOnState` forever when its stub stops responding. The existing path sends a single interrupt and logs repeated warnings, but it has no retry and no escape. Container teardown can then block indefinitely in `WaitExited`, leaving the sandbox and its control RPCs stuck.

We hit this in a production Kubernetes cluster. Multiple gVisor pods remained Terminating for days, on both release-20260427.0 and release-20260714.0. In one affected sandbox, both `runsc kill` and `runsc debug --stacks` hung while the sentry stayed alive. Systrap `[exe]` stubs and node load continued to accumulate even though CPU use remained low. Killing the sentry process tree cleared the pod and returned node load to normal immediately. Full forensics in #14408.

#14405 is an independent production report of the same hang, with the sentry stacks we could not capture: a task goroutine parked 7,818 minutes in `sleepOnState` on the ThreadContext.State futex after one missed SIGUSR1, `Kernel.Pause()` blocked behind it so the URPC handler for `runsc kill --all` never returns. On the shim side 45,646 goroutines piled up on `Init.mu` behind the hung kill, growing containerd and kubelet memory until node OOM.

The fix, staged from least to most disruptive:

1. Because #14405 shows a single interrupt can simply be lost, the interrupt is resent on every checkup wakeup instead of once. A lost signal recovers the workload with no visible damage.
2. If the stub stays unresponsive through repeated interrupts for the full stuck-context deadline (30 seconds), all goroutine stacks are dumped via `log.TracebackAll`. Stacks are unobtainable from outside a wedged sandbox, so the escalation self-documents what #14405 had to capture by hand.
3. Only the stuck subprocess is then killed, through the same path `NotifyInterrupt` already uses when a stub is gone (mark dead, `ContextStateUnexpectedDeath`, kill the syscall thread). Healthy subprocesses in the sandbox are unaffected, and the task goroutine returns instead of blocking `Kernel.Pause()` forever.

`sleepOnStateWithTimeout` returns a typed error and stays side-effect free; `sleepOnState` does the traceback and the kill. A stub that is already gone takes the quiet `NotifyInterrupt` path (warn + kill subprocess, no stack dump). Resent interrupts do not extend the stuck deadline; a regression test covers that.

Tests:

- `bazel test //pkg/sentry/platform/systrap:systrap_test`
- `bazel build //runsc:runsc`
- Stuck, stub-gone, repeated-interrupt, and recovered-context cases covered on the typed error value.

Fixes #14408. Related to #14405 and #12209.

Assisted-by: Claude Code
FUTURE_COPYBARA_INTEGRATE_REVIEW=#14201 from NahumLitvin:fix/systrap-stuck-context d119f5c
PiperOrigin-RevId: 974017442

This is the shim half of #14548. When the sandbox control server stopped answering (#14405, #14408), each `runsc kill` ran with no limit and the shim held the container lock across the call. In one production dump 45,646 goroutines sat behind that lock, most of them cadvisor `Stats` calls whose clients had long since timed out, 610MB shim RSS after 5 days, and kubelet retried into it for days.

Two changes:

- `Kill`, `Stats` and `Status` each run under a 30s context, so a wedged sandbox releases `Init.mu` instead of holding it for good
- runsc commands get `WaitDelay`, so a killed command cannot leave `Wait` blocked on fds a grandchild inherited

The sentry-side fix for the original hang merged in #14201. This covers the shim for when a sandbox finds a new way to stop answering.

Not covered: `Init.Update` and `Init.Delete` hold `p.mu` across a runsc call with no bound too, same shape as `Status` and `Stats`. The pileup can come back through either. I left them out to keep this reviewable.

On the number: `kill`'s own retry loop already gives up after 1s, 10s under `FuseAbort`, so 30s sits well clear of anything a healthy sandbox does. It is there to catch the sandbox that never answers, not to set a latency target.

An earlier revision escalated to SIGKILL on the sandbox process once the 30s expired. That is out, per review: `killAllLocked` also runs from the exit handler when an init process exits, so the escalation would have taken healthy containers in the same sandbox with it. #14548 tracks it instead.

Tests: `init_kill_test.go` covers a wedged `runsc` for Kill, KillAll, Stats and Status, and checks that the sandbox process survives KillAll. `runsc_test.go` covers the fd case: a fake runsc that exits while a child still holds the pipe `cmdOutput` handed it, which leaves `Wait` stuck without `WaitDelay`.

Updates #14548

Assisted-by: Claude Code
FUTURE_COPYBARA_INTEGRATE_REVIEW=#14549 from NahumLitvin:fix/shim-bounded-kill 9d1aa3e
PiperOrigin-RevId: 987096412

This is the shim half of #14548. When the sandbox control server stopped answering (#14405, #14408), each `runsc kill` ran with no limit and the shim held the container lock across the call. In one production dump 45,646 goroutines sat behind that lock, most of them cadvisor `Stats` calls whose clients had long since timed out, 610MB shim RSS after 5 days, and kubelet retried into it for days.

Two changes:

- `Kill`, `Stats` and `Status` each run under a 30s context, so a wedged sandbox releases `Init.mu` instead of holding it for good
- runsc commands get `WaitDelay`, so a killed command cannot leave `Wait` blocked on fds a grandchild inherited

The sentry-side fix for the original hang merged in #14201. This covers the shim for when a sandbox finds a new way to stop answering.

Not covered: `Init.Update` and `Init.Delete` hold `p.mu` across a runsc call with no bound too, same shape as `Status` and `Stats`. The pileup can come back through either. I left them out to keep this reviewable.

On the number: `kill`'s own retry loop already gives up after 1s, 10s under `FuseAbort`, so 30s sits well clear of anything a healthy sandbox does. It is there to catch the sandbox that never answers, not to set a latency target.

An earlier revision escalated to SIGKILL on the sandbox process once the 30s expired. That is out, per review: `killAllLocked` also runs from the exit handler when an init process exits, so the escalation would have taken healthy containers in the same sandbox with it. #14548 tracks it instead.

Tests: `init_kill_test.go` covers a wedged `runsc` for Kill, KillAll, Stats and Status, and checks that the sandbox process survives KillAll. `runsc_test.go` covers the fd case: a fake runsc that exits while a child still holds the pipe `cmdOutput` handed it, which leaves `Wait` stuck without `WaitDelay`.

Updates #14548

Assisted-by: Claude Code
FUTURE_COPYBARA_INTEGRATE_REVIEW=#14549 from NahumLitvin:fix/shim-bounded-kill 9d1aa3e
PiperOrigin-RevId: 987096412

This is the shim half of #14548. When the sandbox control server stopped answering (#14405, #14408), each `runsc kill` ran with no limit and the shim held the container lock across the call. In one production dump 45,646 goroutines sat behind that lock, most of them cadvisor `Stats` calls whose clients had long since timed out, 610MB shim RSS after 5 days, and kubelet retried into it for days.

Two changes:

- `Kill`, `Stats` and `Status` each run under a 30s context, so a wedged sandbox releases `Init.mu` instead of holding it for good
- runsc commands get `WaitDelay`, so a killed command cannot leave `Wait` blocked on fds a grandchild inherited

The sentry-side fix for the original hang merged in #14201. This covers the shim for when a sandbox finds a new way to stop answering.

Not covered: `Init.Update` and `Init.Delete` hold `p.mu` across a runsc call with no bound too, same shape as `Status` and `Stats`. The pileup can come back through either. I left them out to keep this reviewable.

On the number: `kill`'s own retry loop already gives up after 1s, 10s under `FuseAbort`, so 30s sits well clear of anything a healthy sandbox does. It is there to catch the sandbox that never answers, not to set a latency target.

An earlier revision escalated to SIGKILL on the sandbox process once the 30s expired. That is out, per review: `killAllLocked` also runs from the exit handler when an init process exits, so the escalation would have taken healthy containers in the same sandbox with it. #14548 tracks it instead.

Tests: `init_kill_test.go` covers a wedged `runsc` for Kill, KillAll, Stats and Status, and checks that the sandbox process survives KillAll. `runsc_test.go` covers the fd case: a fake runsc that exits while a child still holds the pipe `cmdOutput` handed it, which leaves `Wait` stuck without `WaitDelay`.

Updates #14548

Assisted-by: Claude Code
FUTURE_COPYBARA_INTEGRATE_REVIEW=#14549 from NahumLitvin:fix/shim-bounded-kill 9d1aa3e
PiperOrigin-RevId: 987096412

This is the shim half of #14548. When the sandbox control server stopped answering (#14405, #14408), each `runsc kill` ran with no limit and the shim held the container lock across the call. In one production dump 45,646 goroutines sat behind that lock, most of them cadvisor `Stats` calls whose clients had long since timed out, 610MB shim RSS after 5 days, and kubelet retried into it for days.

Two changes:

- `Kill`, `Stats` and `Status` each run under a 30s context, so a wedged sandbox releases `Init.mu` instead of holding it for good
- runsc commands get `WaitDelay`, so a killed command cannot leave `Wait` blocked on fds a grandchild inherited

The sentry-side fix for the original hang merged in #14201. This covers the shim for when a sandbox finds a new way to stop answering.

Not covered: `Init.Update` and `Init.Delete` hold `p.mu` across a runsc call with no bound too, same shape as `Status` and `Stats`. The pileup can come back through either. I left them out to keep this reviewable.

On the number: `kill`'s own retry loop already gives up after 1s, 10s under `FuseAbort`, so 30s sits well clear of anything a healthy sandbox does. It is there to catch the sandbox that never answers, not to set a latency target.

An earlier revision escalated to SIGKILL on the sandbox process once the 30s expired. That is out, per review: `killAllLocked` also runs from the exit handler when an init process exits, so the escalation would have taken healthy containers in the same sandbox with it. #14548 tracks it instead.

Tests: `init_kill_test.go` covers a wedged `runsc` for Kill, KillAll, Stats and Status, and checks that the sandbox process survives KillAll. `runsc_test.go` covers the fd case: a fake runsc that exits while a child still holds the pipe `cmdOutput` handed it, which leaves `Wait` stuck without `WaitDelay`.

Updates #14548

Assisted-by: Claude Code
FUTURE_COPYBARA_INTEGRATE_REVIEW=#14549 from NahumLitvin:fix/shim-bounded-kill 9d1aa3e
PiperOrigin-RevId: 987096412

This is the shim half of #14548. When the sandbox control server stopped answering (#14405, #14408), each `runsc kill` ran with no limit and the shim held the container lock across the call. In one production dump 45,646 goroutines sat behind that lock, most of them cadvisor `Stats` calls whose clients had long since timed out, 610MB shim RSS after 5 days, and kubelet retried into it for days.

Two changes:

- `Kill`, `Stats` and `Status` each run under a 30s context, so a wedged sandbox releases `Init.mu` instead of holding it for good
- runsc commands get `WaitDelay`, so a killed command cannot leave `Wait` blocked on fds a grandchild inherited

The sentry-side fix for the original hang merged in #14201. This covers the shim for when a sandbox finds a new way to stop answering.

Not covered: `Init.Update` and `Init.Delete` hold `p.mu` across a runsc call with no bound too, same shape as `Status` and `Stats`. The pileup can come back through either. I left them out to keep this reviewable.

On the number: `kill`'s own retry loop already gives up after 1s, 10s under `FuseAbort`, so 30s sits well clear of anything a healthy sandbox does. It is there to catch the sandbox that never answers, not to set a latency target.

An earlier revision escalated to SIGKILL on the sandbox process once the 30s expired. That is out, per review: `killAllLocked` also runs from the exit handler when an init process exits, so the escalation would have taken healthy containers in the same sandbox with it. #14548 tracks it instead.

Tests: `init_kill_test.go` covers a wedged `runsc` for Kill, KillAll, Stats and Status, and checks that the sandbox process survives KillAll. `runsc_test.go` covers the fd case: a fake runsc that exits while a child still holds the pipe `cmdOutput` handed it, which leaves `Wait` stuck without `WaitDelay`.

Updates #14548

Assisted-by: Claude Code
FUTURE_COPYBARA_INTEGRATE_REVIEW=#14549 from NahumLitvin:fix/shim-bounded-kill 9d1aa3e
PiperOrigin-RevId: 987096412

sleepOnStateWithTimeout only resends the interrupt when other contexts
are waiting in the queue. If the sentry interrupts a running context
(signal delivery, kill) and that one signal is lost while the queue is
empty, nothing resends it and the stuck deadline never starts, so the
task goroutine waits indefinitely. The fix in google#14201 only covered the
busy-queue case.

Track a sentry-private interruptPending flag, set in NotifyInterrupt and
cleared with the shared Interrupt flag, and resend while it is set. The
shared Interrupt field is not used for this because the stub can write
it. An idle context with nothing pending still waits as before.

The stuck-context traceback now also logs the context's shared state
and the stub thread's sysmsg state, error and line, so the next report
shows whether the stub ever saw the signal.

Both gaps were pointed out by Mor Kalfon reading google#14201.

Assisted-by: Claude Code
Signed-off-by: Nahum Litvin <nahuml@wix.com>

sleepOnStateWithTimeout only resends the interrupt when other contexts
are waiting in the queue. If the sentry interrupts a running context
(signal delivery, kill) and that one signal is lost while the queue is
empty, nothing resends it and the stuck deadline never starts, so the
task goroutine waits indefinitely. The fix in google#14201 only covered the
busy-queue case.

Track a sentry-private interruptRequested flag, set in NotifyInterrupt and
cleared with the shared Interrupt flag, and resend while it is set. The
shared Interrupt field is not used for this because the stub can write
it. An idle context with nothing pending still waits as before.

The stuck-context traceback now also logs the context's shared state
and the stub thread's sysmsg state, error and line, so the next report
shows where the stub thread was when it stopped answering.

Both gaps were pointed out by Mor Kalfon reading google#14201.

Assisted-by: Claude Code
Signed-off-by: Nahum Litvin <nahuml@wix.com>

sleepOnStateWithTimeout only resends the interrupt when other contexts
are waiting in the queue, and the stuck deadline only fires after a
resend. If the sentry interrupts a running context (signal delivery,
kill) and that one signal is lost while the queue is empty, nothing
resends it and the task goroutine waits indefinitely. google#14201 only
covered the busy-queue case.

Track a sentry-private interruptRequested flag, set in NotifyInterrupt
and cleared with the shared Interrupt flag, and resend while it is set.
The shared Interrupt field is not used for this because the stub can
write it. A context running with nothing requested still waits as
before.

Pointed out by Mor Kalfon reading google#14201.

Assisted-by: Claude Code
Signed-off-by: Nahum Litvin <nahuml@wix.com>

The stuck-context traceback only printed the context ID, so a report
showed the sentry side and nothing about the stub that stopped
answering. Also log the context's shared state and the stub thread's
sysmsg state, error and line, read atomically.

Pointed out by Mor Kalfon reading google#14201.

Assisted-by: Claude Code
Signed-off-by: Nahum Litvin <nahuml@wix.com>

google#14201 kills the subprocess when a context stays stuck through the 30s
deadline. A stub in a slow host page fault cannot take the interrupt
signal until the fault returns, so it looks stuck while it is healthy.
On a host with a slow FUSE server a fault can take more than 30s, and
the kill takes down a subprocess that would have recovered. The kill
does not help a lost signal either: the resend every checkup is what
recovers it. Log the stack dump once when the deadline passes, then keep
waiting and resending, with a warning every 30s after that. A stub
thread that no longer exists is still killed.

sleepOnStateWithTimeout only resends the interrupt when other contexts
are waiting in the queue. If the sentry interrupts a running context
and that one signal is lost while the queue is empty, nothing resends
it. Track a sentry-private interruptRequested flag, set in
NotifyInterrupt and cleared with the shared Interrupt flag, and resend
while it is set. The shared Interrupt field is not used for this because
the stub can write it.

The stuck-context traceback only printed the context ID. Also log the
context's shared state and the stub thread's sysmsg state, error and
line, read atomically.

The dispatcher only moves waiting contexts to the slow path after two
deep sleep timeouts without processing anything, and any completion
resets that timer. Under steady churn a context whose stub stopped
answering is polled forever and never reaches sleepOnStateWithTimeout,
where interrupts are resent. Hand a context to the slow path once it has
waited longer than contextPreemptTimeout (10ms).

The empty-queue resend and the stub-state logging were pointed out by
Mor Kalfon reading google#14201.

Assisted-by: Claude Code
Signed-off-by: Nahum Litvin <nahuml@wix.com>