Repository navigation
feat(query): push LIMIT down across shard groups for raw selects - #27690
Open
alrieckert wants to merge 1 commit into
Open
alrieckert wants to merge 1 commit into
alrieckert wants to merge 1 commit into
Conversation
SELECT ... LIMIT N on a single, ungrouped measurement with no aggregate call currently scans every shard in the query's time range before LIMIT is ever applied: Shards.CreateIterator unconditionally calls CreateIterator on every shard, and the per-series LimitIterator that eventually trims the result never stops pulling from its input early - it only returns once the fully-merged, all-shards input itself is exhausted. So LIMIT reduces rows returned, not work done, which is expensive once a measurement spans many shards/shard groups with no tag filter to narrow the series set. Add a narrow, provably-safe fast path: when a fetch is for a single, non-regex Measurement, has no GROUP BY (tags or time), no SLIMIT/SOFFSET, and is the sole source feeding the statement's LIMIT/OFFSET (IteratorOptions.GlobalLimitEligible, set only by buildAuxIterator), LocalShardMapping.CreateIterator lazily opens shard groups one at a time - in the order matching ORDER BY's direction, reversed for DESC - and stops opening further groups once enough rows have already been produced to satisfy Limit+Offset (query.NewLazyGroupChainIterator). All point filtering still happens in the existing, unchanged top-level LimitIterator; the new iterator is a pure counting pass-through. Every other query shape (GROUP BY, regex, multi-measurement, SLIMIT/SOFFSET, aggregates) is completely unaffected and takes the original, unchanged path.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
SELECT ... LIMIT N on a single, ungrouped measurement with no aggregate call currently scans every shard in the query's time range before LIMIT is ever applied: Shards.CreateIterator unconditionally calls CreateIterator on every shard, and the per-series LimitIterator that eventually trims the result never stops pulling from its input early - it only returns once the fully-merged, all-shards input itself is exhausted. So LIMIT reduces rows returned, not work done, which is expensive once a measurement spans many shards/shard groups with no tag filter to narrow the series set.
Add a narrow, provably-safe fast path: when a fetch is for a single, non-regex Measurement, has no GROUP BY (tags or time), no SLIMIT/SOFFSET, and is the sole source feeding the statement's LIMIT/OFFSET (IteratorOptions.GlobalLimitEligible, set only by buildAuxIterator), LocalShardMapping.CreateIterator lazily opens shard groups one at a time - in the order matching ORDER BY's direction, reversed for DESC - and stops opening further groups once enough rows have already been produced to satisfy Limit+Offset (query.NewLazyGroupChainIterator). All point filtering still happens in the existing, unchanged top-level LimitIterator; the new iterator is a pure counting pass-through. Every other query shape (GROUP BY, regex, multi-measurement, SLIMIT/SOFFSET, aggregates) is completely unaffected and takes the original, unchanged path.