Skip to content

Benchmark introspection against multiple large schemas#4260

Merged
andimarek merged 1 commit intomasterfrom
introspection-benchmark-multiple-schemas
Feb 23, 2026
Merged

Benchmark introspection against multiple large schemas#4260
andimarek merged 1 commit intomasterfrom
introspection-benchmark-multiple-schemas

Conversation

@andimarek
Copy link
Member

@andimarek andimarek commented Feb 23, 2026 •
edited
Loading

Summary

  • Enhanced IntrospectionBenchmark to use JMH @Param to run against 5 schemas of increasing size instead of only large-schema-4.graphqls
  • Schemas tested: large-schema-2 (medium), large-schema-3 (large), large-schema-4 (very large), large-schema-5 (split, ~11 MB), large-schema-federated-1 (largest federated)
  • Added @OutputTimeUnit(TimeUnit.MILLISECONDS) for average time readability

Baseline results (2 warmup iterations, 3 measurement iterations, 2 forks)

Benchmark                                                                     (schemaFile)   Mode  Cnt    Score    Error  Units
IntrospectionBenchmark.benchMarkIntrospectionThroughput            large-schema-2.graphqls  thrpt    6   35.257 ±  0.502  ops/s
IntrospectionBenchmark.benchMarkIntrospectionThroughput            large-schema-3.graphqls  thrpt    6   11.138 ±  3.687  ops/s
IntrospectionBenchmark.benchMarkIntrospectionThroughput            large-schema-4.graphqls  thrpt    6    3.067 ±  0.268  ops/s
IntrospectionBenchmark.benchMarkIntrospectionThroughput            large-schema-5.graphqls  thrpt    6    2.546 ±  0.145  ops/s
IntrospectionBenchmark.benchMarkIntrospectionThroughput  large-schema-federated-1.graphqls  thrpt    6    1.716 ±  0.312  ops/s
IntrospectionBenchmark.benchMarkIntrospectionAvgTime               large-schema-2.graphqls   avgt    6   22.628 ±  1.697  ms/op
IntrospectionBenchmark.benchMarkIntrospectionAvgTime               large-schema-3.graphqls   avgt    6   79.248 ±  2.164  ms/op
IntrospectionBenchmark.benchMarkIntrospectionAvgTime               large-schema-4.graphqls   avgt    6  331.683 ± 27.475  ms/op
IntrospectionBenchmark.benchMarkIntrospectionAvgTime               large-schema-5.graphqls   avgt    6  388.928 ± 66.078  ms/op
IntrospectionBenchmark.benchMarkIntrospectionAvgTime     large-schema-federated-1.graphqls   avgt    6  614.070 ± 20.609  ms/op
Schema Avg Time (ms/op) Throughput (ops/s)
large-schema-2 22.6 35.3
large-schema-3 79.2 11.1
large-schema-4 331.7 3.1
large-schema-5 388.9 2.5
large-schema-federated-1 614.1 1.7

Test plan

  • ./gradlew jmhClasses compiles successfully
  • Full benchmark run with 2 warmups, 3 iterations, 2 forks produces 10 rows (5 schemas × 2 modes)

🤖 Generated with Claude Code

@andimarek @claude
Benchmark introspection against multiple large schemas
9f509f3 
Use JMH @param to run IntrospectionBenchmark against 5 schemas of
increasing size (medium to very large) instead of only large-schema-4.
This helps characterize introspection performance scaling.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@github-actions
Copy link
Contributor

github-actions bot commented Feb 23, 2026 •
edited
Loading

Test Results

  335 files  ±0    335 suites  ±0   5m 3s ⏱️ -3s
5 378 tests ±0  5 370 ✅ ±0  8 💤 ±0  0 ❌ ±0 
5 467 runs  ±0  5 459 ✅ ±0  8 💤 ±0  0 ❌ ±0 

Results for commit 9f509f3. ± Comparison against base commit f7a2377.

This pull request removes 196 and adds 172 tests. Note that renamed tests count towards both. 
	?

	, expected: combo-\"\\\b\f\n\r\t, #4]
                __schema { types { fields { args { type { name fields { name }}}}}}
                __schema { types { fields { type { name fields { name }}}}}
                __schema { types { inputFields { type { inputFields { name }}}}}
                __schema { types { interfaces { fields { type { interfaces { name } } } } } }
                __schema { types { name} }
                __type(name : "t") { name }
                a1: __schema { types { name} }
                a1: __type(name : "t") { name }
…

graphql.AssertTest ‑ assertFalse with different number of error args but false does not throw assertions [toRun: <graphql.AssertTest$__spock_feature_0_21prov0_closure23@717d7587 delegate=inaccessible owner=inaccessible thisObject=inaccessible resolveStrategy=inaccessible directive=inaccessible parameterTypes=inaccessible maximumNumberOfParameters=inaccessible bcw=inaccessible thisType=inaccessible>, expectedMessage: error arg1, #0]
graphql.AssertTest ‑ assertFalse with different number of error args but false does not throw assertions [toRun: <graphql.AssertTest$__spock_feature_0_21prov0_closure24@2489e84a delegate=inaccessible owner=inaccessible thisObject=inaccessible resolveStrategy=inaccessible directive=inaccessible parameterTypes=inaccessible maximumNumberOfParameters=inaccessible bcw=inaccessible thisType=inaccessible>, expectedMessage: error arg1 arg2, #1]
graphql.AssertTest ‑ assertFalse with different number of error args but false does not throw assertions [toRun: <graphql.AssertTest$__spock_feature_0_21prov0_closure25@2fd954f delegate=inaccessible owner=inaccessible thisObject=inaccessible resolveStrategy=inaccessible directive=inaccessible parameterTypes=inaccessible maximumNumberOfParameters=inaccessible bcw=inaccessible thisType=inaccessible>, expectedMessage: error arg1 arg2 arg3, #2]
graphql.AssertTest ‑ assertFalse with different number of error args throws assertions [toRun: <graphql.AssertTest$__spock_feature_0_20prov0_closure20@7c51782d delegate=inaccessible owner=inaccessible thisObject=inaccessible resolveStrategy=inaccessible directive=inaccessible parameterTypes=inaccessible maximumNumberOfParameters=inaccessible bcw=inaccessible thisType=inaccessible>, expectedMessage: error arg1, #0]
graphql.AssertTest ‑ assertFalse with different number of error args throws assertions [toRun: <graphql.AssertTest$__spock_feature_0_20prov0_closure21@4fe533ff delegate=inaccessible owner=inaccessible thisObject=inaccessible resolveStrategy=inaccessible directive=inaccessible parameterTypes=inaccessible maximumNumberOfParameters=inaccessible bcw=inaccessible thisType=inaccessible>, expectedMessage: error arg1 arg2, #1]
graphql.AssertTest ‑ assertFalse with different number of error args throws assertions [toRun: <graphql.AssertTest$__spock_feature_0_20prov0_closure22@5d5b9ecb delegate=inaccessible owner=inaccessible thisObject=inaccessible resolveStrategy=inaccessible directive=inaccessible parameterTypes=inaccessible maximumNumberOfParameters=inaccessible bcw=inaccessible thisType=inaccessible>, expectedMessage: error arg1 arg2 arg3, #2]
graphql.AssertTest ‑ assertNotNull with different number of  error args throws assertions [toRun: <graphql.AssertTest$__spock_feature_0_5prov0_closure3@6468a7b6 delegate=inaccessible owner=inaccessible thisObject=inaccessible resolveStrategy=inaccessible directive=inaccessible parameterTypes=inaccessible maximumNumberOfParameters=inaccessible bcw=inaccessible thisType=inaccessible>, expectedMessage: error arg1, #0]
graphql.AssertTest ‑ assertNotNull with different number of  error args throws assertions [toRun: <graphql.AssertTest$__spock_feature_0_5prov0_closure4@2121d1f9 delegate=inaccessible owner=inaccessible thisObject=inaccessible resolveStrategy=inaccessible directive=inaccessible parameterTypes=inaccessible maximumNumberOfParameters=inaccessible bcw=inaccessible thisType=inaccessible>, expectedMessage: error arg1 arg2, #1]
graphql.AssertTest ‑ assertNotNull with different number of  error args throws assertions [toRun: <graphql.AssertTest$__spock_feature_0_5prov0_closure5@46e64760 delegate=inaccessible owner=inaccessible thisObject=inaccessible resolveStrategy=inaccessible directive=inaccessible parameterTypes=inaccessible maximumNumberOfParameters=inaccessible bcw=inaccessible thisType=inaccessible>, expectedMessage: error arg1 arg2 arg3, #2]
graphql.AssertTest ‑ assertNotNull with different number of error args with non null does not throw assertions [toRun: <graphql.AssertTest$__spock_feature_0_6prov0_closure6@60c73e58 delegate=inaccessible owner=inaccessible thisObject=inaccessible resolveStrategy=inaccessible directive=inaccessible parameterTypes=inaccessible maximumNumberOfParameters=inaccessible bcw=inaccessible thisType=inaccessible>, expectedMessage: error arg1, #0]
…

♻️ This comment has been updated with latest results.

@andimarek andimarek added the not release related changes which are not released (for example unit tests or docs) label Feb 23, 2026
@andimarek andimarek merged commit 30e0cb4 into master Feb 23, 2026
9 checks passed
@andimarek andimarek mentioned this pull request Feb 23, 2026
Open
2 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

not release related changes which are not released (for example unit tests or docs)

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@andimarek