perf(reminders): drop reminders that are not due soon - #10259
Conversation
|
Azure Pipelines: There may be pipelines that require an authorized user to comment /azp run to run. |
|
Azure Pipelines: There may be pipelines that require an authorized user to comment /azp run to run. |
There was a problem hiding this comment.
Pull request overview
This PR addresses a regression where reminders with very long due times/periods (beyond the runtime timer limit) could silently stop firing and only surface as a fault during shutdown. The fix introduces a configurable “loading window” so far-future reminders stay storage-only until they approach execution, and replaces unbounded delays with bounded absolute waiting to avoid Task.Delay overflow.
Changes:
- Add
ReminderOptions.ReminderLoadingWindowand updateLocalReminderServiceto load/evict reminders based on that window while preserving local mutations against stale table refreshes. - Introduce
TimeProvider.DelayUntilAsync(chunked waiting) andDateTime.AddClampedto safely compute absolute deadlines without exceeding timer limits. - Expand reminder test coverage, including distant reminders, stale-refresh scenarios, and timer-delay boundary cases.
Show a summary per file
| File | Description |
|---|---|
| test/Orleans.Reminders.Tests/TimerTests/ReminderTests_TableGrain.cs | Adds functional coverage for loading-window behavior, eviction, and stale refresh reconciliation. |
| test/Orleans.Reminders.Tests/TimerTests/LocalReminderServiceTests.cs | Updates unit tests to cover new tick calculation and new options validation. |
| test/Orleans.Reminders.Tests/TimerTests/ControllableReminderTable.cs | Adds a controllable reminder table wrapper to simulate blocked/stale range reads. |
| test/Orleans.Core.Tests/General/TimeProviderExtensionsTests.cs | Adds tests for DelayUntilAsync boundary and cancellation behavior beyond max timer delay. |
| test/Orleans.Core.Tests/General/DateTimeExtensionsTests.cs | Adds a test ensuring AddClamped preserves DateTimeKind. |
| test/Grains/TestInternalGrains/ReminderTestGrain2.cs | Adds an overload to start reminders with explicit due time/period for new tests. |
| test/Grains/TestGrainInterfaces/IReminderTestGrain2.cs | Adds the new overload to the test grain interface. |
| src/Orleans.Runtime/Silo/Silo.cs | Switches to new TimeSpan Min/Max extensions usage. |
| src/Orleans.Runtime/Core/InsideRuntimeClient.cs | Switches to new TimeSpan Min/Max extensions usage. |
| src/Orleans.Reminders/ReminderService/LocalReminderService.cs | Core logic changes: loading window, bounded absolute waits, and refresh-vs-local reconciliation. |
| src/Orleans.Reminders/Options/ReminderOptions.cs | Adds ReminderLoadingWindow and validates refresh period/loading window constraints. |
| src/Orleans.Reminders/Diagnostics/ReminderEvents.cs | Adds OutsideLoadingWindow stop reason for instrumentation/diagnostics. |
| src/Orleans.Reminders/Constants/ReminderOptionsDefaults.cs | Adds default loading window (2× refresh period). |
| src/Orleans.Core/Utils/TimeSpanExtensions.cs | Introduces TimeSpan.Min/Max extension methods (replacement for StandardExtensions). |
| src/Orleans.Core/Utils/TimeProviderExtensions.cs | Adds DelayUntilAsync which chunks long waits to avoid Task.Delay limits. |
| src/Orleans.Core/Utils/StandardExtensions.cs | Removes the old static Min/Max helpers. |
| src/Orleans.Core/Utils/DateTimeExtensions.cs | Adds AddClamped to avoid DateTime overflow when computing future deadlines. |
| src/Orleans.Core/Runtime/OutsideRuntimeClient.cs | Switches to new TimeSpan Min/Max extensions usage. |
| src/Orleans.Core/Messaging/GatewayManager.cs | Switches to new TimeSpan Min/Max extensions usage. |
| src/Orleans.Core/Manifest/ClientClusterManifestProvider.cs | Switches to new TimeSpan Min/Max extensions usage. |
| src/Orleans.Core/Diagnostics/Metrics/ReminderInstruments.cs | Improves gauge description to reflect tombstones being included. |
| src/Orleans.Core/Async/AsyncExecutorWithRetries.cs | Switches to new TimeSpan Min/Max extensions usage. |
| src/api/Orleans.Reminders/Orleans.Reminders.cs | Updates public API surface for the new option and stop reason enum value. |
Copilot's findings
- Files reviewed: 23/23 changed files
- Comments generated: 3
8d9bc90 to
5b7652b
Compare
5b7652b to
598305b
Compare
There was a problem hiding this comment.
Copilot's findings
Suppressed comments (1)
src/Orleans.Reminders/ReminderService/LocalReminderService.cs:852
CalculateNextTickTimeassumesentry.StartAt.KindisUtcviaDebug.Assertand also returnsentry.StartAtdirectly whennow <= entry.StartAt. Some reminder storage providers can round-tripStartAtasDateTimeKind.Unspecified(the code below already treats reminder timestamps as UTC even if the kind is lost), which makes the assert incorrect in Debug builds and can propagateUnspecifiedinto computed tick times (breaking the laterpreviousTickTime.Kind == Utcassert). NormalizeStartAtto UTC ticks before using/returning it.
internal static DateTime CalculateNextTickTime(ReminderEntry entry, DateTime now)
{
ArgumentNullException.ThrowIfNull(entry);
Debug.Assert(entry.StartAt.Kind == DateTimeKind.Utc);
Debug.Assert(now.Kind == DateTimeKind.Utc);
if (entry.Period <= TimeSpan.Zero)
{
throw new ArgumentOutOfRangeException(nameof(entry), entry.Period, "Reminder period must be greater than zero.");
}
if (now <= entry.StartAt)
{
return entry.StartAt;
}
- Files reviewed: 12/12 changed files
- Comments generated: 0 new
There was a problem hiding this comment.
Copilot's findings
Suppressed comments (1)
src/Orleans.Reminders/ReminderService/LocalReminderService.cs:843
CalculateNextTickTimeasserts thatentry.StartAt.Kind == DateTimeKind.Utc, but reminder storage providers can legitimately round-tripDateTimevalues asUnspecified(while still representing UTC). This makes the invariant too strict (and contradicts the later comment which says timestamps represent UTC even if kind is lost), causing debug builds/tests to fail unexpectedly.
internal static DateTime CalculateNextTickTime(ReminderEntry entry, DateTime now)
{
ArgumentNullException.ThrowIfNull(entry);
Debug.Assert(entry.StartAt.Kind == DateTimeKind.Utc);
Debug.Assert(now.Kind == DateTimeKind.Utc);
- Files reviewed: 13/13 changed files
- Comments generated: 1
There was a problem hiding this comment.
Copilot's findings
Suppressed comments (1)
src/Orleans.Reminders/ReminderService/LocalReminderService.cs:2
using Microsoft.CodeAnalysis;appears to be unused in this file and will trigger CS8019 (unused using directive) under warnings-as-errors. Remove it to keep the build clean.
using Microsoft.CodeAnalysis;
- Files reviewed: 13/13 changed files
- Comments generated: 0 new
ce886b0 to
d8cd4be
Compare
There was a problem hiding this comment.
Copilot's findings
Suppressed comments (2)
src/Orleans.Reminders/ReminderService/LocalReminderService.cs:5
using Microsoft.CodeAnalysis;is unused in this file and will trigger an unnecessary-using warning (and likely fail the build if warnings are treated as errors). Please remove it.
using System.Diagnostics;
using Microsoft.CodeAnalysis;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Options;
src/Orleans.Reminders/ReminderService/LocalReminderService.cs:845
ReminderEntry.StartAtis documented/handled elsewhere as “UTC even if a storage provider loses DateTimeKind”, butCalculateNextTickTimecurrently assumesDateTimeKind.Utc(Debug.Assert) and usesnow <= entry.StartAt, which can perform kind conversions and miscompute comparisons ifStartAt.KindisUnspecified/Local. NormalizeStartAt(andnow) viaDateTime.SpecifyKind(..., Utc)and compare using ticks to avoid unintended kind conversions.
ArgumentNullException.ThrowIfNull(entry);
Debug.Assert(entry.StartAt.Kind == DateTimeKind.Utc);
Debug.Assert(now.Kind == DateTimeKind.Utc);
if (entry.Period <= TimeSpan.Zero)
{
- Files reviewed: 13/13 changed files
- Comments generated: 0 new
There was a problem hiding this comment.
Copilot's findings
Suppressed comments (1)
src/Orleans.Reminders/ReminderService/LocalReminderService.cs:5
using Microsoft.CodeAnalysis;is unused in this file (it’s the only occurrence), which will trigger CS8019 under warnings-as-errors. Please remove it to keep the build clean.
using System.Diagnostics;
using Microsoft.CodeAnalysis;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Options;
- Files reviewed: 13/13 changed files
- Comments generated: 0 new
Treat reminder StartAt ticks as UTC when storage providers discard DateTimeKind, preventing Debug assertions from aborting reminder refreshes. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: ae5923e3-d0e0-4d87-b14f-cb7b19fe3584
Advance fake time until each expected activation and tick count so replayed diagnostic events cannot make the loading-window test race. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: ae5923e3-d0e0-4d87-b14f-cb7b19fe3584
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: ae5923e3-d0e0-4d87-b14f-cb7b19fe3584
9de1e0a to
4573ee5
Compare
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: ae5923e3-d0e0-4d87-b14f-cb7b19fe3584
There was a problem hiding this comment.
Copilot's findings
Suppressed comments (2)
src/Orleans.Reminders/ReminderService/LocalReminderService.cs:1122
TryStopOutsideLoadingWindowcancels_scheduleChangedCancellationbut does not dispose it. Since evictions can happen frequently with a small loading window, disposing here helps avoid CTS resource buildup.
_shared.LogDebugStoppingReminder(entry, ReminderEvents.LocalReminderStopReason.OutsideLoadingWindow);
_stopCancellation.Cancel();
scheduleChangedCancellation.Cancel();
return true;
src/Orleans.Reminders/ReminderService/LocalReminderService.cs:1099
StopAsynccancels the current_scheduleChangedCancellationbut never disposes it. Since this CTS is replaced/disposed onUpdate, leaving it undisposed on stop can accumulate unmanaged resources (eg, registrations) across many reminders.
Dispose the captured CTS after cancelling it, similar to Update.
This issue also appears on line 1119 of the same file.
if (entry is not null)
{
_shared.LogDebugStoppingReminder(entry, reason);
}
_stopCancellation.Cancel();
scheduleChangedCancellation.Cancel();
return runTask ?? Task.CompletedTask;
- Files reviewed: 13/13 changed files
- Comments generated: 0 new
Fixes #10249.
Orleans currently keeps every owned reminder scheduled in memory, including reminders whose next tick is months away. This change introduces a configurable loading window so distant reminders remain only in durable reminder storage until a refresh observes that their next tick is approaching.
Reminder refreshes and local writes are sequence-ordered, with temporary tombstones preventing stale snapshots from restoring updated or unregistered schedules. Reminder option validation ensures the loading window is at least as long as the refresh period. If storage cannot be refreshed for longer than the configured window, affected reminders resume on their persisted cadence after storage recovers, potentially skipping missed occurrences.