Skip to content

perf(reminders): drop reminders that are not due soon - #10259

Merged
ReubenBond merged 18 commits into
dotnet:mainfrom
ReubenBond:reubenbond-fix-long-due-reminders
Jul 31, 2026
Merged

ReubenBond merged 18 commits into
dotnet:mainfrom
ReubenBond:reubenbond-fix-long-due-reminders

Conversation

@ReubenBond

@ReubenBond ReubenBond commented Jul 13, 2026

Copy link
Copy Markdown
Member

Fixes #10249.

Orleans currently keeps every owned reminder scheduled in memory, including reminders whose next tick is months away. This change introduces a configurable loading window so distant reminders remain only in durable reminder storage until a refresh observes that their next tick is approaching.

Reminder refreshes and local writes are sequence-ordered, with temporary tombstones preventing stale snapshots from restoring updated or unregistered schedules. Reminder option validation ensures the loading window is at least as long as the refresh period. If storage cannot be refreshed for longer than the configured window, affected reminders resume on their persisted cadence after storage recovers, potentially skipping missed occurrences.

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

@ReubenBond
ReubenBond marked this pull request as ready for review July 13, 2026 19:11
@ReubenBond
ReubenBond requested a review from Copilot July 13, 2026 19:11
@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR addresses a regression where reminders with very long due times/periods (beyond the runtime timer limit) could silently stop firing and only surface as a fault during shutdown. The fix introduces a configurable “loading window” so far-future reminders stay storage-only until they approach execution, and replaces unbounded delays with bounded absolute waiting to avoid Task.Delay overflow.

Changes:

  • Add ReminderOptions.ReminderLoadingWindow and update LocalReminderService to load/evict reminders based on that window while preserving local mutations against stale table refreshes.
  • Introduce TimeProvider.DelayUntilAsync (chunked waiting) and DateTime.AddClamped to safely compute absolute deadlines without exceeding timer limits.
  • Expand reminder test coverage, including distant reminders, stale-refresh scenarios, and timer-delay boundary cases.
Show a summary per file
File Description
test/Orleans.Reminders.Tests/TimerTests/ReminderTests_TableGrain.cs Adds functional coverage for loading-window behavior, eviction, and stale refresh reconciliation.
test/Orleans.Reminders.Tests/TimerTests/LocalReminderServiceTests.cs Updates unit tests to cover new tick calculation and new options validation.
test/Orleans.Reminders.Tests/TimerTests/ControllableReminderTable.cs Adds a controllable reminder table wrapper to simulate blocked/stale range reads.
test/Orleans.Core.Tests/General/TimeProviderExtensionsTests.cs Adds tests for DelayUntilAsync boundary and cancellation behavior beyond max timer delay.
test/Orleans.Core.Tests/General/DateTimeExtensionsTests.cs Adds a test ensuring AddClamped preserves DateTimeKind.
test/Grains/TestInternalGrains/ReminderTestGrain2.cs Adds an overload to start reminders with explicit due time/period for new tests.
test/Grains/TestGrainInterfaces/IReminderTestGrain2.cs Adds the new overload to the test grain interface.
src/Orleans.Runtime/Silo/Silo.cs Switches to new TimeSpan Min/Max extensions usage.
src/Orleans.Runtime/Core/InsideRuntimeClient.cs Switches to new TimeSpan Min/Max extensions usage.
src/Orleans.Reminders/ReminderService/LocalReminderService.cs Core logic changes: loading window, bounded absolute waits, and refresh-vs-local reconciliation.
src/Orleans.Reminders/Options/ReminderOptions.cs Adds ReminderLoadingWindow and validates refresh period/loading window constraints.
src/Orleans.Reminders/Diagnostics/ReminderEvents.cs Adds OutsideLoadingWindow stop reason for instrumentation/diagnostics.
src/Orleans.Reminders/Constants/ReminderOptionsDefaults.cs Adds default loading window (2× refresh period).
src/Orleans.Core/Utils/TimeSpanExtensions.cs Introduces TimeSpan.Min/Max extension methods (replacement for StandardExtensions).
src/Orleans.Core/Utils/TimeProviderExtensions.cs Adds DelayUntilAsync which chunks long waits to avoid Task.Delay limits.
src/Orleans.Core/Utils/StandardExtensions.cs Removes the old static Min/Max helpers.
src/Orleans.Core/Utils/DateTimeExtensions.cs Adds AddClamped to avoid DateTime overflow when computing future deadlines.
src/Orleans.Core/Runtime/OutsideRuntimeClient.cs Switches to new TimeSpan Min/Max extensions usage.
src/Orleans.Core/Messaging/GatewayManager.cs Switches to new TimeSpan Min/Max extensions usage.
src/Orleans.Core/Manifest/ClientClusterManifestProvider.cs Switches to new TimeSpan Min/Max extensions usage.
src/Orleans.Core/Diagnostics/Metrics/ReminderInstruments.cs Improves gauge description to reflect tombstones being included.
src/Orleans.Core/Async/AsyncExecutorWithRetries.cs Switches to new TimeSpan Min/Max extensions usage.
src/api/Orleans.Reminders/Orleans.Reminders.cs Updates public API surface for the new option and stop reason enum value.

Copilot's findings

  • Files reviewed: 23/23 changed files
  • Comments generated: 3

Comment thread src/Orleans.Reminders/ReminderService/LocalReminderService.cs
Comment thread src/Orleans.Core/Messaging/GatewayManager.cs Outdated
Comment thread src/Orleans.Core/Utils/TimeSpanExtensions.cs Outdated
@ReubenBond ReubenBond changed the title Fix reminders with long due times perf(reminders): drop reminders that are not due soon Jul 13, 2026
@ReubenBond
ReubenBond force-pushed the reubenbond-fix-long-due-reminders branch from 8d9bc90 to 5b7652b Compare July 15, 2026 14:13
Copilot AI review requested due to automatic review settings July 31, 2026 16:18
@ReubenBond
ReubenBond force-pushed the reubenbond-fix-long-due-reminders branch from 5b7652b to 598305b Compare July 31, 2026 16:18

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot's findings

Suppressed comments (1)

src/Orleans.Reminders/ReminderService/LocalReminderService.cs:852

  • CalculateNextTickTime assumes entry.StartAt.Kind is Utc via Debug.Assert and also returns entry.StartAt directly when now <= entry.StartAt. Some reminder storage providers can round-trip StartAt as DateTimeKind.Unspecified (the code below already treats reminder timestamps as UTC even if the kind is lost), which makes the assert incorrect in Debug builds and can propagate Unspecified into computed tick times (breaking the later previousTickTime.Kind == Utc assert). Normalize StartAt to UTC ticks before using/returning it.
        internal static DateTime CalculateNextTickTime(ReminderEntry entry, DateTime now)
        {
            ArgumentNullException.ThrowIfNull(entry);
            Debug.Assert(entry.StartAt.Kind == DateTimeKind.Utc);
            Debug.Assert(now.Kind == DateTimeKind.Utc);
            if (entry.Period <= TimeSpan.Zero)
            {
                throw new ArgumentOutOfRangeException(nameof(entry), entry.Period, "Reminder period must be greater than zero.");
            }

            if (now <= entry.StartAt)
            {
                return entry.StartAt;
            }
  • Files reviewed: 12/12 changed files
  • Comments generated: 0 new

Copilot AI review requested due to automatic review settings July 31, 2026 18:04

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot's findings

Suppressed comments (1)

src/Orleans.Reminders/ReminderService/LocalReminderService.cs:843

  • CalculateNextTickTime asserts that entry.StartAt.Kind == DateTimeKind.Utc, but reminder storage providers can legitimately round-trip DateTime values as Unspecified (while still representing UTC). This makes the invariant too strict (and contradicts the later comment which says timestamps represent UTC even if kind is lost), causing debug builds/tests to fail unexpectedly.
        internal static DateTime CalculateNextTickTime(ReminderEntry entry, DateTime now)
        {
            ArgumentNullException.ThrowIfNull(entry);
            Debug.Assert(entry.StartAt.Kind == DateTimeKind.Utc);
            Debug.Assert(now.Kind == DateTimeKind.Utc);
  • Files reviewed: 13/13 changed files
  • Comments generated: 1

Comment thread src/Orleans.Reminders/ReminderService/LocalReminderService.cs
Copilot AI review requested due to automatic review settings July 31, 2026 19:08

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot's findings

Suppressed comments (1)

src/Orleans.Reminders/ReminderService/LocalReminderService.cs:2

  • using Microsoft.CodeAnalysis; appears to be unused in this file and will trigger CS8019 (unused using directive) under warnings-as-errors. Remove it to keep the build clean.
using Microsoft.CodeAnalysis;
  • Files reviewed: 13/13 changed files
  • Comments generated: 0 new

Copilot AI review requested due to automatic review settings July 31, 2026 19:16

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot's findings

  • Files reviewed: 13/13 changed files
  • Comments generated: 1

Comment thread src/Orleans.Reminders/ReminderService/LocalReminderService.cs
Copilot AI review requested due to automatic review settings July 31, 2026 19:43
@ReubenBond
ReubenBond force-pushed the reubenbond-fix-long-due-reminders branch from ce886b0 to d8cd4be Compare July 31, 2026 19:43

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot's findings

Suppressed comments (2)

src/Orleans.Reminders/ReminderService/LocalReminderService.cs:5

  • using Microsoft.CodeAnalysis; is unused in this file and will trigger an unnecessary-using warning (and likely fail the build if warnings are treated as errors). Please remove it.
using System.Diagnostics;
using Microsoft.CodeAnalysis;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Options;

src/Orleans.Reminders/ReminderService/LocalReminderService.cs:845

  • ReminderEntry.StartAt is documented/handled elsewhere as “UTC even if a storage provider loses DateTimeKind”, but CalculateNextTickTime currently assumes DateTimeKind.Utc (Debug.Assert) and uses now <= entry.StartAt, which can perform kind conversions and miscompute comparisons if StartAt.Kind is Unspecified/Local. Normalize StartAt (and now) via DateTime.SpecifyKind(..., Utc) and compare using ticks to avoid unintended kind conversions.
            ArgumentNullException.ThrowIfNull(entry);
            Debug.Assert(entry.StartAt.Kind == DateTimeKind.Utc);
            Debug.Assert(now.Kind == DateTimeKind.Utc);
            if (entry.Period <= TimeSpan.Zero)
            {
  • Files reviewed: 13/13 changed files
  • Comments generated: 0 new

Copilot AI review requested due to automatic review settings July 31, 2026 19:57

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot's findings

Suppressed comments (1)

src/Orleans.Reminders/ReminderService/LocalReminderService.cs:5

  • using Microsoft.CodeAnalysis; is unused in this file (it’s the only occurrence), which will trigger CS8019 under warnings-as-errors. Please remove it to keep the build clean.
using System.Diagnostics;
using Microsoft.CodeAnalysis;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Options;
  • Files reviewed: 13/13 changed files
  • Comments generated: 0 new

Copilot AI review requested due to automatic review settings July 31, 2026 20:25
ReubenBond and others added 3 commits July 31, 2026 15:31
Treat reminder StartAt ticks as UTC when storage providers discard DateTimeKind, preventing Debug assertions from aborting reminder refreshes.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: ae5923e3-d0e0-4d87-b14f-cb7b19fe3584
Advance fake time until each expected activation and tick count so replayed diagnostic events cannot make the loading-window test race.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: ae5923e3-d0e0-4d87-b14f-cb7b19fe3584
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: ae5923e3-d0e0-4d87-b14f-cb7b19fe3584
@ReubenBond
ReubenBond force-pushed the reubenbond-fix-long-due-reminders branch from 9de1e0a to 4573ee5 Compare July 31, 2026 22:31
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: ae5923e3-d0e0-4d87-b14f-cb7b19fe3584
Copilot AI review requested due to automatic review settings July 31, 2026 22:57

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot's findings

Suppressed comments (2)

src/Orleans.Reminders/ReminderService/LocalReminderService.cs:1122

  • TryStopOutsideLoadingWindow cancels _scheduleChangedCancellation but does not dispose it. Since evictions can happen frequently with a small loading window, disposing here helps avoid CTS resource buildup.
                _shared.LogDebugStoppingReminder(entry, ReminderEvents.LocalReminderStopReason.OutsideLoadingWindow);
                _stopCancellation.Cancel();
                scheduleChangedCancellation.Cancel();
                return true;

src/Orleans.Reminders/ReminderService/LocalReminderService.cs:1099

  • StopAsync cancels the current _scheduleChangedCancellation but never disposes it. Since this CTS is replaced/disposed on Update, leaving it undisposed on stop can accumulate unmanaged resources (eg, registrations) across many reminders.

Dispose the captured CTS after cancelling it, similar to Update.

This issue also appears on line 1119 of the same file.

                if (entry is not null)
                {
                    _shared.LogDebugStoppingReminder(entry, reason);
                }
                _stopCancellation.Cancel();
                scheduleChangedCancellation.Cancel();
                return runTask ?? Task.CompletedTask;
  • Files reviewed: 13/13 changed files
  • Comments generated: 0 new

@ReubenBond
ReubenBond merged commit 3b2f171 into dotnet:main Jul 31, 2026
67 checks passed
@ReubenBond
ReubenBond deleted the reubenbond-fix-long-due-reminders branch July 31, 2026 23:24
This was referenced Aug 28, 2026
@github-actions github-actions Bot locked and limited conversation to collaborators Aug 31, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Reminders with due times over ~49.7 days silently never fire and fault the reminder service on shutdown (10.2.0 regression via #10040)

2 participants