[API Proposal]: Sharing pipeline values between incremental generators #85239

Background and Motivation

Most requests to "run generator B after generator A" have one shape: A knows what work needs to be done, B knows how to do it.

• A generator discovers types that need JSON serialization and wants the System.Text.Json generator to produce the serializers. It cannot generate them itself, and it cannot make the STJ generator see a [JsonSerializable] context inside its own output: STJ discovers contexts with ForAttributeWithMetadataName over the input compilation.
• A generator declares request types and wants the ASP.NET validations generator to produce IValidatableInfo for them. That generator collects types from [ValidatableType] and from the minimal API endpoints it can see, and neither source reaches generated code. ASP.NET ships an analyzer for this case; its message reads "Source generators cannot inspect each other's output. Declare the type in a regular .cs file instead."

In both cases the producer needs the compilation to know what to ask for, because it reacts to attributes on user types. That rules out RegisterPreCompilationSourceOutput (#83089), which is restricted to non-compilation inputs by design. The only working arrangement today is a project boundary, and it is not available when the generated code has to live next to the user's types.

Ordering proposals (#57239, #81395) were not declined for lack of demand. Each of them introduces an additional Compilation, and binding a compilation is the most expensive thing the driver does. The feedback on #81395 is explicit: "Making new compilations is precisely what we must not do with new SG apis."

This proposal stays inside that constraint. One generator publishes a stream of values of a contract type; another consumes that stream as an ordinary IncrementalValuesProvider<T>, in the standard phase, against the same compilation every other generator sees. No new phase, no new compilation. Ordering between generators follows from data dependencies, the same way it already does between nodes inside one generator. Generators that do not use the feature are not affected.

RegisterPreCompilationSourceOutput and this proposal are complementary: pre-compilation moves source across generators before the compilation is built, this moves data across generators after it is built.

Proposed API

namespace Microsoft.CodeAnalysis
{
    public readonly partial struct IncrementalGeneratorInitializationContext
    {
+       // Producer side. Values from `provider` are published as T.
+       public void RegisterExternalOutput<T>(IncrementalValuesProvider<T> provider)
+           where T : IEquatable<T>;
+
+       // Consumer side. Everything published as T by any generator in the run.
+       public IncrementalValuesProvider<T> ExternalInputsProvider<T>()
+           where T : IEquatable<T>;
    }
}

The contract type is the address. The generator that owns a contract defines T and ships it as a separate contracts package, a plain netstandard2.0 library that a producer references from its generator project and bundles next to its own analyzer assembly. The driver matches producers to consumers by the namespace-qualified name of T.

T is a plain data type: records of primitives, strings and collections of those.

Semantics:

• Once every generator has been initialized and before the standard phase runs, the driver builds a graph: for every contract type, each producer has an edge to each consumer. The graph must be acyclic.
• An external input resolves to an empty stream when no generator in the run produces the contract, or when the input takes part in a cycle. Only the cycle is reported, as a driver diagnostic naming the generators involved. A missing producer is silent, because removing a package must not break the build of a project that consumes its contract.
• Generators run in topological order. The driver already runs generators sequentially, so for generators without dependencies this changes the order and nothing else. External inputs are evaluated in the standard phase, after post-initialization and pre-compilation outputs have been added to the compilation, so a producer may compute its values from the compilation and from syntax providers.
• A contract with several producers resolves to the concatenation of their streams, ordered by producer assembly name and then generator type name.
• The external input node participates in incremental caching like any other node. If a producer's stream is unchanged between runs, nothing downstream of the consumer's input node is re-executed.

Driver changes: a graph over the registered contract types, topological execution order, and a new node kind for external inputs whose state table is filled from the producer's node table. GeneratorDriverRunResult exposes external inputs in TrackedSteps. AddGenerators and ReplaceGenerators rebuild the graph. CommonCompiler and the workspaces need no change: every generator observes the same compilation, and the final compilation is assembled once from all outputs, as today.

Usage Examples

Requesting serializers from the STJ generator

Contract, shipped as System.Text.Json.SourceGeneration.Contracts:

namespace System.Text.Json.SourceGeneration
{
    // TypeName is a fully qualified metadata name. ContextName is the metadata name of the
    // JsonSerializerContext that should own the generated serializer. If no such context is
    // declared in user code, the STJ generator creates it, so the producer can reference it
    // from its own output.
    public sealed record SerializableTypeRequest(string TypeName, string ContextName);
}

Producer, an endpoint generator that knows which types cross the wire:

public void Initialize(IncrementalGeneratorInitializationContext context)
{
    var endpoints = context.SyntaxProvider.ForAttributeWithMetadataName(
        "FluentEndpoints.EndpointAttribute",
        static (node, _) => node is ClassDeclarationSyntax,
        static (ctx, _) => EndpointModel.Create(ctx));

    var wireTypes = endpoints
        .SelectMany(static (e, _) => e.WireTypes)
        .Select(static (t, _) => new SerializableTypeRequest(t.MetadataName, "FluentEndpoints.Generated.JsonContext"));

    context.RegisterExternalOutput(wireTypes);
    context.RegisterSourceOutput(endpoints, EmitEndpoint);
}

Consumer, inside the STJ generator:

var requested = context
    .ExternalInputsProvider<SerializableTypeRequest>()
    .Combine(context.CompilationProvider)
    .Select(static (pair, ct) => ResolveRequest(pair.Left, pair.Right, ct));

var fromAttributes = /* existing [JsonSerializable] pipeline */;

context.RegisterSourceOutput(fromAttributes.Collect().Combine(requested.Collect()), EmitContexts);

EndpointModel and WireTypes are equatable records. Editing the body of a handler leaves the published stream unchanged, and the STJ generator does not re-run its emit step.

Requesting validation info from the ASP.NET validations generator

That generator already merges two streams of types, one from [ValidatableType] and one from endpoint parameters, with Concat before it walks the type graph and emits. An external input is a third stream in the same place.

Contract, shipped as Microsoft.Extensions.Validation.Contracts:

namespace Microsoft.Extensions.Validation
{
    // TypeName is a fully qualified metadata name. The generator resolves it and walks
    // its members the same way it does for a type marked with [ValidatableType].
    public sealed record ValidatableTypeRequest(string TypeName);
}

Producer, the same endpoint generator, publishing the request types its handlers bind:

var requestTypes = endpoints
    .SelectMany(static (e, _) => e.RequestTypes)
    .Select(static (t, _) => new ValidatableTypeRequest(t.MetadataName));

context.RegisterExternalOutput(requestTypes);

Consumer, inside the validations generator:

var validatableTypesFromInputs = context
    .ExternalInputsProvider<ValidatableTypeRequest>()
    .Combine(context.CompilationProvider)
    .Select(static (pair, ct) => ResolveValidatableType(pair.Left.TypeName, pair.Right, ct))
    .Where(static type => !type.IsDefault);

var allValidatableTypesProviders = validatableTypesFromEndpoints
    .Concat(validatableTypesWithAttribute)
    .Concat(validatableTypesFromInputs);

Everything after the Concat, including Distinct and the emit step, is unchanged. The request types are ordinary user-declared classes; only the code that names them as validatable lives in generated source. That is why the attribute route is closed and the data route is open.

Alternative Designs

Explicit ordering (#57239). Every downstream generator would see a new Compilation on every run, and ordering across NuGet packages has no natural owner.

Two-phase generators (#81395). Declarations from every generator are collected into an enriched compilation that every implementation phase observes. It solves the "see other generators' types" case in general, at the cost of one additional compilation for every project with generators, which is why it was sent back for rework.

RegisterPreCompilationSourceOutput. Already approved, and the right tool when the producer does not need the compilation. It cannot express a producer that reacts to attributes on user code.

Library-level composition. Works when the upstream logic is small enough to ship as a library and the downstream author is willing to call it. Does not work for STJ or the validations generator.

Risks

External inputs carry data, not source. A consumer that has to bind against code the producer emitted is not served by this proposal, and that is left out on purpose.

Producer and consumer do not share a runtime type. The analyzer assembly loader loads each generator's dependencies into that generator's own AssemblyLoadContext, so a contract assembly referenced from two packages is loaded twice, and a cast between the two copies fails. The driver has to bridge that boundary, and the restriction on T above exists so that it can: values would be serialized into the consumer's T, with the format and the compatibility rules between contract versions left to the implementation. The consequence is that a producer that drifted from the consumer's contract is caught by the driver at run time, not by the compiler. Loading a designated contract assembly once into a context both generators resolve against would give real type identity, but it is a loader change with its own versioning problems and should not gate the first version.

The feature has value only once the owner of a downstream generator adds a consumer branch. The downstream owner is the one who knows what a request should look like, so this is the right place for the work, but it means nothing ships until STJ or the validations generator adopt it.

No breaking changes: the two methods are additive, and a run without producers or consumers behaves exactly as today. The performance cost for generators that do not use the feature is the graph construction over an empty set of contracts.