Skip to content

fix(directory): cancel snapshot pulls from departed silos - #10318

Merged
ReubenBond merged 3 commits into
dotnet:mainfrom
ReubenBond:reubenbond-investigate-grain-directory-hang
Aug 1, 2026
Merged

ReubenBond merged 3 commits into
dotnet:mainfrom
ReubenBond:reubenbond-investigate-grain-directory-hang

Conversation

@ReubenBond

@ReubenBond ReubenBond commented Aug 1, 2026

Copy link
Copy Markdown
Member

During an in-place rolling upgrade, a distributed directory partition could begin pulling a snapshot from the retiring owner while that silo transitioned from ShuttingDown to Dead. Snapshot pulls were the one per-member directory operation not bound to the membership-driven cancellation token, so the acquisition range lock remained held until the 30-second messaging timeout. Local lookups for that range queued behind the lock and timed out as a result.

This change plumbs the shared member cancellation token through GetSnapshotAsync and range waits. When the previous owner departs, the snapshot pull now stops immediately and the existing recovery path takes over. Callee shutdown remains linked into the wait, and diagnostics distinguish an unavailable owner from an invalid snapshot.

Microsoft Reviewers: Open in CodeFlow

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 6b23a814-ee05-48e1-bd7b-9b304a409b25
Copilot AI review requested due to automatic review settings August 1, 2026 01:20

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR improves Orleans distributed grain directory resilience during rolling upgrades by ensuring snapshot-pull operations are canceled promptly when the previous owner silo departs, preventing prolonged range-lock holds and downstream lookup timeouts.

Changes:

  • Plumbs a membership-driven CancellationToken through IGrainDirectoryPartition.GetSnapshotAsync to allow snapshot pulls to cancel when the previous owner becomes unavailable.
  • Updates range-wait logic (WaitForRange) to support cancellation during lock waits and view refresh.
  • Improves diagnostics to distinguish “previous owner unavailable” from “invalid/missing snapshot”.
Show a summary per file
File Description
src/Orleans.Runtime/GrainDirectory/LocalGrainDirectoryCompatibility.cs Updates the compatibility system target to accept and honor a cancellation token for snapshot requests.
src/Orleans.Runtime/GrainDirectory/IGrainDirectoryPartition.cs Extends the partition system-target contract to include an optional CancellationToken for snapshot pulls.
src/Orleans.Runtime/GrainDirectory/GrainDirectoryPartition.cs Links caller cancellation into snapshot acquisition and range waits; routes snapshot pulls via member-cancellable invocation and refines warning logs.

Copilot's findings

  • Files reviewed: 3/3 changed files
  • Comments generated: 1

Comment thread src/Orleans.Runtime/GrainDirectory/GrainDirectoryPartition.cs
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 6b23a814-ee05-48e1-bd7b-9b304a409b25
Copilot AI review requested due to automatic review settings August 1, 2026 02:46

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot's findings

Suppressed comments (2)

src/Orleans.Runtime/GrainDirectory/GrainDirectoryPartition.cs:109

  • This post-wait cancellation check only observes ShutdownToken. If the caller-provided cancellationToken is canceled after WaitForRange completes (but before snapshot construction), this method will continue building the snapshot unnecessarily. Since you already created a linked token, use that for the final ThrowIfCancellationRequested check.
        ShutdownToken.ThrowIfCancellationRequested();

src/Orleans.Runtime/GrainDirectory/GrainDirectoryPartition.cs:582

  • This snapshot pull is no longer bound to the local partition ShutdownToken. Previously the call used WaitAsync(ShutdownToken), ensuring the range-lock acquisition path could abort promptly during local shutdown; now InvokeOnClusterMember waits solely on the remote member token and can block until the messaging timeout if this silo is stopping.
                var snapshot = await InvokeOnClusterMember(
                    previousOwner,
                    cancellationToken => partition.GetSnapshotAsync(
                        current.Version,
                        previousVersion,
  • Files reviewed: 5/5 changed files
  • Comments generated: 0 new

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 6b23a814-ee05-48e1-bd7b-9b304a409b25
Copilot AI review requested due to automatic review settings August 1, 2026 04:27

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot's findings

Suppressed comments (1)

src/Orleans.Runtime/GrainDirectory/DistributedGrainDirectory.cs:190

  • InvokeAsync links the per-request token with GetClusterMemberCancellationToken(owner). If the token cache has not yet been updated for the selected owner (e.g., after a RefreshViewAsync returns a newer view but before ProcessMembershipUpdates calls _clusterMemberCancellationTokens.Update for that snapshot), GetToken returns a pre-canceled token and the call is immediately canceled/retried (and can fail after MaxAttempts). Consider lazily seeding/updating the token cache when the latest membership snapshot still considers the owner invokable.
                RequestContext.Set("gid", partitionReference.GetGrainId());
                using var requestCts = CancellationTokenSource.CreateLinkedTokenSource(
                    cancellationToken,
                    GetClusterMemberCancellationToken(owner));
                invokeResult = await func(partitionReference, view.Version, state, requestCts.Token);
  • Files reviewed: 5/5 changed files
  • Comments generated: 0 new

@ReubenBond
ReubenBond merged commit c925aa7 into dotnet:main Aug 1, 2026
66 of 67 checks passed
@ReubenBond
ReubenBond deleted the reubenbond-investigate-grain-directory-hang branch August 1, 2026 06:01
This was referenced Aug 28, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants