test: observe grain directory convergence in liveness tests - #10299
Merged
Merged
Conversation
Liveness_Grain_4_Kill_Silo_1_With_Timers (and related hard-kill liveness tests) can spuriously fail with a TimeoutException right after killing a silo. WaitForLivenessToStabilizeAsync computes a wait based on ProbeTimeout * NumMissedProbesLimit, but ClusterMembershipOptions. ExtendProbeTimeoutDuringDegradation (enabled by default) adaptively extends probe timeouts under local health degradation, which is common on loaded CI machines. This makes the actual failure detection + directory convergence time exceed the test's precomputed wait, so the very next grain call can hit the 30s response timeout before the cluster finishes reacting to the kill. Retry transient TimeoutExceptions for the post-recovery traffic in Do_Liveness_OracleTest_2, bounded by a 60s window, so the test tolerates slow convergence without weakening its assertions.
|
Azure Pipelines: There may be pipelines that require an authorized user to comment /azp run to run. |
Contributor
There was a problem hiding this comment.
Pull request overview
This PR improves the reliability of Orleans liveness/membership tests by retrying post-recovery grain calls which can transiently time out after a hard silo kill (where failure detection and directory convergence are timing-sensitive under CI load).
Changes:
- Wrap post-recovery grain traffic in
Do_Liveness_OracleTest_2with a retry-on-timeout helper. - Add
SendTrafficWithRetryto tolerate transientTimeoutExceptions for a bounded period after kill/restart. - Add
System.Diagnosticsusage for retry window timing.
Show a summary per file
| File | Description |
|---|---|
| test/Orleans.Runtime.Tests/MembershipTests/LivenessTests.cs | Adds bounded retry logic around post-recovery grain calls to reduce spurious liveness test failures under load. |
Copilot's findings
Comments suppressed due to low confidence (1)
test/Orleans.Runtime.Tests/MembershipTests/LivenessTests.cs:186
- To support a shared retry window across all post-recovery grain calls,
SendTrafficWithRetryshould accept the caller'sStopwatch/window instead of starting a new one per call. This prevents multiplying the retry budget by the number of grains.
private async Task SendTrafficWithRetry(long key, bool startTimers = false)
{
var stopwatch = Stopwatch.StartNew();
var retryWindow = TimeSpan.FromSeconds(60);
while (true)
- Files reviewed: 1/1 changed files
- Comments generated: 1
Wait for LocalGrainDirectory to apply the membership version observed by each active silo before liveness stabilization completes. The version signal is published after stale directory and cache entries are removed, closing the race between silo-status convergence and grain routing without polling, sleeps, or call retries. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 965b2b7b-61b4-474d-ac47-bec64d97ff6c
Contributor
There was a problem hiding this comment.
Copilot's findings
Comments suppressed due to low confidence (1)
src/Orleans.TestingHost/LivenessStabilizationHelper.cs:56
- WaitForExpectedActiveSilosAsync can now wait up to
timeoutfor ISiloStatusOracle convergence and then wait up to another fulltimeoutfor grain-directory cleanup, so this method can exceed itstimeoutparameter. Iftimeoutis intended as an overall bound (as implied by callers), consider using the remaining time budget for the grain-directory phase.
waitTasks = testHooks.Select(hooks => hooks.WaitForGrainDirectoryMembershipVersion(timeout));
results = await Task.WhenAll(waitTasks).WaitAsync(timeout);
return results.All(static result => result);
- Files reviewed: 4/4 changed files
- Comments generated: 3
Emit membership-version-applied diagnostics after local directory cleanup and after distributed partition transitions complete. Test hooks subscribe to the aggregate directory event using a check-subscribe-check pattern instead of calling a waiter on LocalGrainDirectory. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 965b2b7b-61b4-474d-ac47-bec64d97ff6c
Contributor
There was a problem hiding this comment.
Copilot's findings
Comments suppressed due to low confidence (2)
src/Orleans.TestingHost/LivenessStabilizationHelper.cs:29
WaitForLivenessToStabilizeAsynclogs that it is waiting "up to" the provided timeout, but this helper no longer tracks elapsed time. Since subsequent waits use the fulltimeoutagain, the overall stabilization wait can exceed the caller-provided budget, slowing/derailing tests.
public static async Task<bool> WaitForExpectedActiveSilosAndGatewaysAsync(
IReadOnlyCollection<SiloHandle> activeSilos,
IReadOnlyCollection<ITestHooks> testHooks,
GatewayManager gatewayManager,
TimeSpan timeout)
{
ArgumentNullException.ThrowIfNull(gatewayManager);
if (!await WaitForExpectedActiveSilosAsync(activeSilos, testHooks, timeout))
{
return false;
}
return await WaitForExpectedActiveGatewaysAsync(activeSilos, gatewayManager, timeout);
}
src/Orleans.TestingHost/LivenessStabilizationHelper.cs:56
- This method now performs two separate waits (liveness oracle convergence, then grain-directory convergence) each with the full
timeout, so it can take up to ~2× the intended stabilization budget. Consider accounting for elapsed time before starting the second phase so the method honors the provided overall timeout.
try
{
var waitTasks = testHooks.Select(hooks => hooks.WaitForActiveSilos(expectedActiveSilos, timeout));
var results = await Task.WhenAll(waitTasks).WaitAsync(timeout);
if (!results.All(static result => result))
{
return false;
}
waitTasks = testHooks.Select(hooks => hooks.WaitForGrainDirectoryMembershipVersion(timeout));
results = await Task.WhenAll(waitTasks).WaitAsync(timeout);
return results.All(static result => result);
- Files reviewed: 7/7 changed files
- Comments generated: 2
Keep grain directory runtime behavior unchanged and emit only local-applied and partition-observed diagnostics. In-process test clusters subscribe before silo startup and track membership versions plus existing range-operation lifetimes externally, allowing liveness stabilization to await directory convergence without runtime task tracking or state. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 965b2b7b-61b4-474d-ac47-bec64d97ff6c
Contributor
There was a problem hiding this comment.
Copilot's findings
Comments suppressed due to low confidence (2)
src/Orleans.TestingHost/LivenessStabilizationHelper.cs:33
timeoutis reused for the silo-status wait, grain-directory convergence wait, and gateway convergence wait, so the overall stabilization can exceed the intended timeout. Track elapsed time and pass the remaining time to each subsequent stage.
if (waitForGrainDirectoryConvergence is not null
&& !await waitForGrainDirectoryConvergence(timeout))
{
return false;
}
src/Orleans.TestingHost/LivenessStabilizationHelper.cs:49
- When
activeSilos.Count == 0, this helper returns immediately without waiting, even though callers/logging treat it as a stabilization wait. This was previously a delay for the full timeout; consider keeping that behavior to avoid surprising fast-fail paths.
if (activeSilos.Count == 0)
{
return false;
}
- Files reviewed: 7/7 changed files
- Comments generated: 0 new
Use a single timeout budget across silo-status, grain-directory, and gateway convergence, and preserve the existing fallback delay when no active silos are available. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 965b2b7b-61b4-474d-ac47-bec64d97ff6c
Do not await runtime grain-directory diagnostics when InProcessTestCluster uses its custom InProcessGrainDirectory, which does not emit those events. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 965b2b7b-61b4-474d-ac47-bec64d97ff6c
Only enable directory convergence observation when each silo uses the built-in local or distributed directory. External default directory implementations do not emit the required diagnostics, so retain the existing liveness stabilization path for those clusters. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 965b2b7b-61b4-474d-ac47-bec64d97ff6c
This was referenced Aug 28, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Liveness_Grain_4_Kill_Silo_1_With_Timerscan fail immediately afterWaitForLivenessToStabilizeAsyncreturns because membership-status convergence and grain-directory convergence are handled by separate asynchronous consumers.The existing stabilization helper waits until each silo's
ISiloStatusOracleexcludes the killed silo. The grain directory independently consumes membership changes to remove registrations and cache entries for activations on dead silos. The status event can therefore complete the helper just before the directory applies the same membership version, allowing the next grain call to route to a stale activation and time out.This change keeps convergence tracking entirely outside the runtime directory implementations:
LocalGrainDirectoryemits a diagnostic after applying a membership snapshot and purging stale directory/cache entries.GrainDirectoryPartitionemits a diagnostic after observing a membership view; its existing range-operation start/completion diagnostics describe asynchronous range transitions.GrainDirectoryObserver, owned by in-process test clusters and subscribed before silos start, tracks those events and pending range operations externally.DistributedGrainDirectoryretains its original behavior and has no additional state or task tracking. There are no fixed delays, polling loops, or grain-call retries.