fix(directory): cancel snapshot pulls from departed silos - #10318
Merged
ReubenBond merged 3 commits intoAug 1, 2026
Merged
ReubenBond merged 3 commits into
ReubenBond merged 3 commits into
Conversation
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6b23a814-ee05-48e1-bd7b-9b304a409b25
Contributor
There was a problem hiding this comment.
Pull request overview
This PR improves Orleans distributed grain directory resilience during rolling upgrades by ensuring snapshot-pull operations are canceled promptly when the previous owner silo departs, preventing prolonged range-lock holds and downstream lookup timeouts.
Changes:
- Plumbs a membership-driven
CancellationTokenthroughIGrainDirectoryPartition.GetSnapshotAsyncto allow snapshot pulls to cancel when the previous owner becomes unavailable. - Updates range-wait logic (
WaitForRange) to support cancellation during lock waits and view refresh. - Improves diagnostics to distinguish “previous owner unavailable” from “invalid/missing snapshot”.
Show a summary per file
| File | Description |
|---|---|
| src/Orleans.Runtime/GrainDirectory/LocalGrainDirectoryCompatibility.cs | Updates the compatibility system target to accept and honor a cancellation token for snapshot requests. |
| src/Orleans.Runtime/GrainDirectory/IGrainDirectoryPartition.cs | Extends the partition system-target contract to include an optional CancellationToken for snapshot pulls. |
| src/Orleans.Runtime/GrainDirectory/GrainDirectoryPartition.cs | Links caller cancellation into snapshot acquisition and range waits; routes snapshot pulls via member-cancellable invocation and refines warning logs. |
Copilot's findings
- Files reviewed: 3/3 changed files
- Comments generated: 1
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6b23a814-ee05-48e1-bd7b-9b304a409b25
Contributor
There was a problem hiding this comment.
Copilot's findings
Suppressed comments (2)
src/Orleans.Runtime/GrainDirectory/GrainDirectoryPartition.cs:109
- This post-wait cancellation check only observes ShutdownToken. If the caller-provided cancellationToken is canceled after WaitForRange completes (but before snapshot construction), this method will continue building the snapshot unnecessarily. Since you already created a linked token, use that for the final ThrowIfCancellationRequested check.
ShutdownToken.ThrowIfCancellationRequested();
src/Orleans.Runtime/GrainDirectory/GrainDirectoryPartition.cs:582
- This snapshot pull is no longer bound to the local partition ShutdownToken. Previously the call used WaitAsync(ShutdownToken), ensuring the range-lock acquisition path could abort promptly during local shutdown; now InvokeOnClusterMember waits solely on the remote member token and can block until the messaging timeout if this silo is stopping.
var snapshot = await InvokeOnClusterMember(
previousOwner,
cancellationToken => partition.GetSnapshotAsync(
current.Version,
previousVersion,
- Files reviewed: 5/5 changed files
- Comments generated: 0 new
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6b23a814-ee05-48e1-bd7b-9b304a409b25
Contributor
There was a problem hiding this comment.
Copilot's findings
Suppressed comments (1)
src/Orleans.Runtime/GrainDirectory/DistributedGrainDirectory.cs:190
- InvokeAsync links the per-request token with GetClusterMemberCancellationToken(owner). If the token cache has not yet been updated for the selected owner (e.g., after a RefreshViewAsync returns a newer view but before ProcessMembershipUpdates calls _clusterMemberCancellationTokens.Update for that snapshot), GetToken returns a pre-canceled token and the call is immediately canceled/retried (and can fail after MaxAttempts). Consider lazily seeding/updating the token cache when the latest membership snapshot still considers the owner invokable.
RequestContext.Set("gid", partitionReference.GetGrainId());
using var requestCts = CancellationTokenSource.CreateLinkedTokenSource(
cancellationToken,
GetClusterMemberCancellationToken(owner));
invokeResult = await func(partitionReference, view.Version, state, requestCts.Token);
- Files reviewed: 5/5 changed files
- Comments generated: 0 new
This was referenced Aug 28, 2026
Merged
This was referenced Aug 31, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
During an in-place rolling upgrade, a distributed directory partition could begin pulling a snapshot from the retiring owner while that silo transitioned from ShuttingDown to Dead. Snapshot pulls were the one per-member directory operation not bound to the membership-driven cancellation token, so the acquisition range lock remained held until the 30-second messaging timeout. Local lookups for that range queued behind the lock and timed out as a result.
This change plumbs the shared member cancellation token through GetSnapshotAsync and range waits. When the previous owner departs, the snapshot pull now stops immediately and the existing recovery path takes over. Callee shutdown remains linked into the wait, and diagnostics distinguish an unavailable owner from an invalid snapshot.
Microsoft Reviewers: Open in CodeFlow