test(reminders): add provider conformance TestKit - #10785

src/Orleans.Reminders.TestKit/ReminderTableCapabilities.cs:69

• The capability documentation says that upsert is blind in every built-in provider, but Cosmos and Firestore pass the supplied ETag as an IfMatchEtag/update precondition and support conditional upserts. This guidance is incorrect and conflicts with the capability the class exposes; document the portable default separately from the providers which can opt in.

    /// Default is <see langword="false"/>. Upsert is a blind write in every built-in Orleans reminder provider;
    /// <see cref="IReminderTable.RemoveRow"/> is the only conditional operation in the contract. Providers which
    /// implement conditional upsert may opt in, which enables the stale-ETag rejection guarantee.

src/Orleans.Reminders.TestKit/ReminderTableTestRunner.cs:1415

• If an upsert succeeds for one or two entries and a later upsert throws, this method exits before returning a RangeFixture, so none of the already-created rows can reach the callers' finally/CleanupAsync blocks. That leaves durable test data behind in external reminder stores, especially when failures are injected; clean up the entries created so far before rethrowing.

            var entry = NewEntry(grainId, name, BaseTime.AddMinutes(items.Count), TimeSpan.FromMinutes(items.Count + 1));
            var etag = await UpsertAsync(entry, guarantee);
            items.Add(new RangeItem(entry, hash, etag));

src/Orleans.Reminders.TestKit/ReminderTableTestRunner.cs:1098

• The isolated table is started here but is never stopped or cleared, including when the subsequent upsert/read assertions throw. Any provider enabling cross-table isolation can therefore leak a remote table's rows or lifecycle resources across tests. Wrap the isolated-table exercise in cleanup which stops/clears it without masking the original failure.

        using var cancellation = new CancellationTokenSource(TimeSpan.FromMinutes(1));
        await isolated!.StartAsync(cancellation.Token);

        var grainId = NewGrainId("isolation");
        var entry = NewEntry(grainId, "isolation", BaseTime.AddMinutes(13), TimeSpan.FromMinutes(9));

src/Orleans.Reminders.TestKit/ReminderTableTestRunner.cs:1540

• The parameter list, opening brace, and body of CompleteUpsertBatchAsync are indented one level deeper than the other methods in this class (for example, AssertExactEntries immediately above), while the declaration and closing brace remain at class-member indentation. Please align the continuation lines and body with the surrounding method style.

            List<(ReminderEntry Entry, Task<string?> Upsert)> pending,
            List<(ReminderEntry Entry, string ETag)> created,
            string guarantee)
        {

src/Orleans.Reminders.TestKit/ReminderTableTestRunner.cs:645

• This guarantee claims to validate movement between loading windows and a grain-scoped read, but ReadRows(0, 0) is only a full-table hash-range enumeration; it has no loading-window or grain parameter. Consequently this test cannot detect a provider/service bug in due-window membership or grain-scoped enumeration, while its name and documentation imply that it does. Either rename/narrow the guarantee to schedule round-tripping or exercise the service's actual loading-window and grain-scoped paths.

        var rows = RequireRows(Guarantee, "ReadRows(0, 0)", await ReminderTable.ReadRows(0, 0));
        AssertExactEntries(
            Guarantee,
            "ReadRows(0, 0)",
            [ReminderTableEntrySnapshot.Create(outside, outsideETag, Capabilities.SupportsSubSecondPrecision)],
            rows.Reminders);

test/Orleans.Runtime.Internal.Tests/RemindersTest/ReminderTableTestsBase.cs:88

• These wrapper tests now invoke conformance methods which are also public [Fact] methods inherited by every provider test class below. For example, the six methods called here are each discovered and run again, doubling external-provider reads/writes and increasing the chance of costly or flaky runs. Remove the duplicate provider wrappers or stop exposing the same scenarios as inherited facts while retaining the desired test categorization.
  test/Orleans.Runtime.Internal.Tests/RemindersTest/ReminderTableTestsBase.cs:65
• This integration always supplies ReminderTableCapabilities.Portable, which disables conditional-upsert coverage for every provider. The shared suite therefore never exercises stale-ETag rejection for providers such as Cosmos and Firestore, even though their implementations support conditional writes and their existing provider tests cover that behavior. Add a provider-specific capability hook and opt those providers in so the common suite actually runs every guarantee each provider supports.

src/Orleans.Reminders.TestKit/IdealizedReminderTable.cs:400

• FreezeReads promises that reads stay on the captured snapshot while writes continue to apply, but clearing _frozenReads here ends the freeze as a side effect of a write. If TestOnlyClearTable is called while a freeze handle is active, subsequent writes become visible before that handle is disposed. Keep the freeze active and clear its snapshot so cleanup returns an empty stale view.

            _frozenReads = null;

src/Orleans.Reminders.TestKit/README.md:91

• This adoption example uses SupportsConcurrentUpserts, but ReminderTableCapabilities exposes SupportsConcurrentOperations instead. Copying the documented snippet therefore does not compile; update the example to use the actual capability property.

for simultaneous writers sets `SupportsConcurrentUpserts = false` on its

src/Orleans.Reminders.TestKit/README.md:75

• The fixture exposes an overridable Capabilities property, but this example constructs a new Portable capability set instead, so any capability overrides on MyReminderTableFixture are silently ignored. Pass the fixture's declared capabilities to the runner so the provider's manifest actually controls which guarantees run.

        : base(fixture.ReminderTable, ReminderTableCapabilities.Portable("MyProvider"))

src/Orleans.Reminders.TestKit/ReminderTableTestFixture.cs:112

• DeployAsync() runs LocalReminderService's lifecycle callback, which already calls IReminderTable.StartAsync for every silo. Calling StartAsync again here is not harmless: providers such as Redis and Cosmos allocate/replace their client on initialization, so this fixture can leak or replace a connection on every setup. Resolve and assign the table without reinitializing it.

        var reminderTable = ResolveReminderTable(cluster.Silos[0].ServiceProvider);
        using var cancellation = new CancellationTokenSource(TimeSpan.FromMinutes(1));
        await reminderTable.StartAsync(cancellation.Token).ConfigureAwait(false);
        _reminderTable = reminderTable;

src/Orleans.Reminders.TestKit/ReminderTableTestFixture.cs:107

• _cluster is assigned only after DeployAsync completes, so any deployment, table-resolution, or table-start failure leaves a built cluster unreachable by DisposeAsync. External-provider startup failures can therefore leak the in-process silos. Record the cluster immediately after Build() so the disposal path can clean up partial initialization.

        var cluster = builder.Build();
        await cluster.DeployAsync().ConfigureAwait(false);
        _cluster = cluster;

src/Orleans.Reminders.TestKit/ReminderTableTestRunner.cs:1541

• If one task in this batch fails, Task.WhenAll throws before any successful results are appended to created. Those successful upserts remain in the provider, and the outer cleanup only removes created, so a transient or conformance failure can contaminate subsequent tests which share this table. Track completed successful tasks before propagating the batch failure.

            var etags = await Task.WhenAll(pending.Select(item => item.Upsert));

src/Orleans.Reminders.TestKit/ReminderTableTestRunner.cs:941

• If any concurrent upsert faults, this method exits before the read/removal path and has no finally cleanup, leaving successful rows in the shared provider table. The next conformance fact can then see those rows and report a misleading failure. Track successful operations and remove them in a finally block (or isolate/clear this test's data on failure).

        var etags = await Task.WhenAll(Enumerable.Range(0, count).Select(index =>
            ReminderTable.UpsertRow(NewEntry(grainId, "concurrent-upsert", BaseTime.AddSeconds(index), TimeSpan.FromMinutes(1)))));

src/Orleans.Reminders.TestKit/ReminderTableTestRunner.cs:1415

• When setup fails after one or more earlier upserts, this method throws without returning a RangeFixture, so its caller has no way to run the fixture's cleanup. The successfully created range rows remain in the shared provider table and can affect later exact-range assertions. Wrap fixture construction in failure-safe cleanup for items.

            var etag = await UpsertAsync(entry, guarantee);
            items.Add(new RangeItem(entry, hash, etag));

src/Orleans.Reminders.TestKit/ReminderTableTestRunner.cs:998

• The outer Task.WhenAll can fail after some grains have completed, and the inner Task.WhenAll can fail after some reminders for a grain have been written; either path skips the removal loop below. Successful rows are therefore left in the shared table and can pollute later tests. Add failure-safe cleanup for every completed upsert.

        var results = await Task.WhenAll(grains.Select(async grainId =>
        {
            var etags = await Task.WhenAll(Enumerable.Range(0, perGrain).Select(index =>
                ReminderTable.UpsertRow(NewEntry(grainId, $"parallel-{index}", BaseTime.AddSeconds(index), TimeSpan.FromMinutes(1)))));
            return (GrainId: grainId, ETags: etags);

src/Orleans.Reminders.TestKit/ReminderTableTestRunner.cs:1095

• The runner starts a table created by the CreateIsolatedTableAsync extension point but never stops or disposes it. A provider which enables cross-table isolation and allocates a connection per table will leak that resource for every conformance invocation. Put the isolated table in cleanup and stop/dispose it according to the provider's lifetime interfaces.

        using var cancellation = new CancellationTokenSource(TimeSpan.FromMinutes(1));
        await isolated!.StartAsync(cancellation.Token);

src/Orleans.Reminders.TestKit/ReminderTableTestRunner.cs:1242

• Provider calls made through this helper can throw before the runner constructs a ReminderConformanceException, so a failed operation loses the provider, guarantee, identity, expected/observed values, and ETags promised by the runner's failure contract. The same direct-call pattern exists for reads and removals. Catch operation exceptions and convert them to a report at the operation boundary while preserving the original exception as the inner exception.

    {
        var etag = await ReminderTable.UpsertRow(entry);
        if (string.IsNullOrEmpty(etag))

test/Orleans.Runtime.Internal.Tests/RemindersTest/ReminderTableTestsBase.cs:65

• Portable leaves SupportsStopAsync enabled, but the inherited stop/restart guarantee is not valid for the Azure Table and Firestore implementations: both StopAsync permanently cancel their one-shot initialization task, and a subsequent StartAsync cannot complete it again, so the ReadRow after restart observes a canceled task. This makes the newly shared provider suites fail (and their cleanup can fail too). Either make those providers' lifecycle restartable or declare this capability disabled for the affected suites instead of using Portable unconditionally.

src/Orleans.Reminders.TestKit/IdealizedReminderTable.cs:406

• BeforeOperationAsync records failures before the operation-specific arguments are available, but it has no suppliedETag parameter. Therefore an unavailable or injected UpsertRow/RemoveRow is logged with SuppliedETag == null even when the caller supplied an ETag, which makes the operation trace incomplete and misleading for failure diagnosis. Thread the supplied ETag through this path and into the failure record.

        ReminderTableOperationKind kind,
        GrainId? grainId = null,
        string? reminderName = null,
        uint? begin = null,
        uint? end = null)

src/Orleans.Reminders.TestKit/IdealizedReminderTable.cs:95

• This XML documentation says CreateCapabilities returns a strict capability set, but ReminderTableProviderProfiles.Oracle explicitly disables SupportsConditionalUpsert. Consumers following this documentation will expect the oracle to exercise every optional guarantee, while the returned manifest silently skips conditional-upsert conformance. Please describe the returned oracle capability profile accurately (or return Strict if conditional upsert is intended to be part of the oracle contract).

    /// Gets the capability set which this reference implementation satisfies.
    /// </summary>
    /// <returns>A strict capability set naming this instance.</returns>
    public ReminderTableCapabilities CreateCapabilities()
        => ReminderTableProviderProfiles.Oracle(Name);

src/Orleans.Reminders.TestKit/ReminderTableTestFixture.cs:107

• The cluster is not assigned to _cluster until DeployAsync completes. If deployment throws after allocating cluster resources, DisposeAsync has no cluster to stop or dispose, leaving the in-process cluster leaked. Dispose the local cluster in a catch/finally around deployment (while preserving the deployment exception).

        var cluster = builder.Build();
        await cluster.DeployAsync().ConfigureAwait(false);
        _cluster = cluster;

src/Orleans.Reminders.TestKit/ReminderTableTestRunner.cs:1639

• If CreateRangeFixtureAsync fails after one or two upserts, it throws before returning a RangeFixture, so each caller's finally block is never entered and the already-created reminders remain in the provider. A transient provider failure can therefore contaminate subsequent conformance tests. Clean up partial items in a finally inside this method (or return an owning fixture before performing the writes).

            var etag = await UpsertAsync(entry, guarantee);

src/Orleans.Reminders.TestKit/ReminderTableTestRunner.cs:1770

• CompleteUpsertBatchAsync only adds entries to created after Task.WhenAll succeeds and after each ETag has been validated. If a batch partially succeeds or one task throws/returns an empty ETag, successful writes from the rest of that batch are not tracked, so the enclosing finally cannot remove them. Track all successful tasks/ETags even when the batch has a failure, or clear the table as a final fallback.

            var etags = await Task.WhenAll(pending.Select(item => item.Upsert));
            for (var index = 0; index < pending.Count; index++)
            {
                var entry = pending[index].Entry;
                var etag = etags[index];

src/Orleans.Reminders.TestKit/ReminderTableTestRunner.cs:563

• This guarantee is gated only by SupportsConditionalUpsert, but the setup first writes the row and then requires the second write to produce a different ETag before it can test a stale value. The Firestore profile enables conditional upsert while explicitly disabling ETag rotation, so a valid same-ETag replacement is reported as a conformance failure. Gate this guarantee on ETag rotation as well, or define a separate capability for obtaining a distinguishable stale token.

        var currentETag = await ReminderTableConvergence.ReadUntilAsync(
            () => UpsertAsync(current, Guarantee),
            etag => !string.Equals(etag, staleETag, StringComparison.Ordinal),
            Capabilities,

src/Orleans.Reminders.TestKit/ReminderTableTestRunner.cs:753

• This assertion only tests hash-zero exclusion if the fixture happens to contain a row with hash 0; CreateRangeFixtureAsync merely asks for three distinct generated hashes, so excluded is normally empty and the (0, uint.MaxValue] boundary is never exercised. Make the fixture deterministically include a hash-zero row (or otherwise inject one) before claiming this boundary guarantee.

            var expected = fixtureState.All.Where(item => item.Hash != 0).ToList();
            var excluded = fixtureState.All.Where(item => item.Hash == 0).ToList();