fix(directory): wait for active membership during startup - #10613
Merged
ReubenBond merged 4 commits intoAug 18, 2026
Merged
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
This PR addresses a startup race in the distributed grain directory where directory membership can temporarily lag behind cluster membership (showing a joining-only/empty directory view even though the silo is already active), causing transient null owner/registration results and invalid activations. It updates the runtime to wait for an owner-bearing directory view when cluster membership still has an active member, and adds a deterministic regression test which simulates the lagging membership view.
Changes:
- Add
GetViewWithOwnerAsyncto wait for a directory membership view which can resolve an owner when cluster membership still contains an active silo. - Refactor
DirectoryMembershipServiceto expose an internalPublishMembershipUpdatehelper (used by the service loop and the new test). - Add a new BVT test which forces a joining-only directory membership view and verifies owner resolution resumes once an active view is published.
Show a summary per file
| File | Description |
|---|---|
| test/Orleans.GrainDirectory.Tests/GrainDirectory/DistributedGrainDirectoryTests.cs | Adds a deterministic regression test for directory membership lag during startup. |
| src/Orleans.Runtime/GrainDirectory/DistributedGrainDirectory.cs | Waits for an owner-bearing directory membership view when cluster membership still has active members; adds test hook. |
| src/Orleans.Runtime/GrainDirectory/DirectoryMembershipService.cs | Extracts membership update publishing into an internal method to support deterministic testing. |
Review details
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
- Files reviewed: 3/3 changed files
- Comments generated: 1
- Review effort level: Lite
Contributor
There was a problem hiding this comment.
Review details
Suppressed comments (1)
Previously missed (1) — in code that hasn't changed since the last review.
src/Orleans.Runtime/GrainDirectory/DistributedGrainDirectory.cs:253
GetViewWithOwnerAsync(...)only returns afterview.TryGetOwner(grainId, ...)succeeds (it’s part of that helper’s loop condition), so thisTryGetOwner/throw is effectively an invariant check and should be unreachable. Consider refactoringGetViewWithOwnerAsyncto return(view, owner, partitionReference)so the owner is computed once and the invariant is expressed in one place (avoids duplicate ring lookups and removes an unreachable branch here).
view = resolvedView;
if (!view.TryGetOwner(grainId, out var owner, out var partitionReference))
{
throw new InvalidOperationException($"Directory membership view {view.Version} does not have an owner for grain '{grainId}'.");
- Files reviewed: 3/3 changed files
- Comments generated: 0 new
- Review effort level: Lite
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: f2fdec36-8ebf-4708-9061-b5e3a1055d4c
ReubenBond
force-pushed
the
rb-fix-startup-task-directory-race
branch
from
August 17, 2026 22:08
8c4dcfc to
b931fda
Compare
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: f2fdec36-8ebf-4708-9061-b5e3a1055d4c
This was referenced Aug 28, 2026
Merged
This was referenced Aug 31, 2026
This was referenced Sep 7, 2026
Open
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #10600
Problem
Startup tasks run after cluster membership reports the silo as active, but the distributed grain directory can briefly lag on a joining-only view with no active directory owners. That transient view was treated as terminal, returning a null registration result and invalidating the new activation.
Solution
Wait for the directory view to catch up when the latest cluster membership still contains an active member. Preserve the existing terminal behavior when the cluster truly has no active members, including shutdown. Add a deterministic regression which holds directory membership on a joining-only view and verifies owner resolution resumes after the active view is published.
Rationale
This restores the readiness invariant at the directory operation boundary instead of masking the failure with startup-task retries or delays, while retaining prompt shutdown behavior.
Microsoft Reviewers: Open in CodeFlow