[v5.0] Put binary AST buffers into cache#6257
Merged
lukastaegert merged 32 commits intorollup-5from Feb 20, 2026
Merged
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Thank you for your contribution! ❤️You can try out this pull request locally by installing Rollup via npm install rollup/rollup#binary-cacheNotice: Ensure you have installed the latest nightly Rust toolchain. If you haven't installed it yet, please see https://www.rust-lang.org/tools/install to learn how to download Rustup and install Rust. or load it into the REPL: |
Performance report
|
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## rollup-5 #6257 +/- ##
============================================
- Coverage 98.79% 98.37% -0.43%
============================================
Files 272 273 +1
Lines 10702 12288 +1586
Branches 2846 2926 +80
============================================
+ Hits 10573 12088 +1515
- Misses 88 150 +62
- Partials 41 50 +9 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
4c2510c to
ed9d5d5
Compare
acbb408 to
565bd0a
Compare
Contributor
There was a problem hiding this comment.
Pull request overview
This PR introduces binary AST buffer caching and lazy AST generation to improve performance. Instead of storing full ESTree ASTs in the cache, Rollup now stores binary buffers and generates AST nodes lazily using getters that replace themselves on first access. This reduces memory usage and processing time when only some AST nodes are needed.
Changes:
- Replaced full AST caching with binary buffer caching
- Implemented lazy AST generation with self-replacing getters
- Added AST serialization/deserialization helpers
- Updated documentation to discourage returning ASTs from hooks
Reviewed changes
Copilot reviewed 150 out of 157 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| test/testHelpers.js | Added test for AST serialization/deserialization roundtrip |
| test/function/samples/serialize-ast/* | New test cases for AST serialization error handling |
| test/function/samples/plugin-error-should-transform/_config.js | Updated to use astBuffer instead of ast in cache |
| test/function/samples/extend-more-hooks-to-include-import-attributes/_config.js | Updated to use astBuffer instead of ast in cache |
| test/function/samples/enforce-plugin-order/_config.js | Updated to use astBuffer instead of ast in cache |
| test/form/samples/custom-ast/* | Removed test for custom AST support |
| test/cli/samples/unfulfilled-hook-actions/rollup.config.js | Updated to use astBuffer instead of ast in cache |
| src/utils/transform.ts | Modified to serialize AST to buffer format |
| src/utils/serialize-ast-strings.ts | Generated file mapping strings to indices for serialization |
| src/utils/parseAstType.d.ts | Added type definitions for new serialization functions |
| src/utils/parseAst.ts | Updated to use lazy deserialization |
| src/utils/logs.ts | Added error logging for AST serialization failures |
| src/utils/getAstBuffer.ts | Added buffer creation and string encoding functions |
| src/utils/convert-ast-strings.ts | Removed + and - operators from constant list |
| src/utils/astToBuffer.ts | Generated file implementing AST to binary serialization |
| src/rollup/types.d.ts | Updated types to use astBuffer and renamed interfaces |
| src/ast/* | Removed generic type parameters from NodeBase classes |
| src/ModuleLoader.ts | Updated to deserialize AST buffers when needed |
| src/Module.ts | Changed to use binary buffers and lazy AST generation |
| src/Graph.ts | Removed AST LRU cache, updated cache type |
| scripts/* | Updated code generation scripts for serialization |
| docs/plugin-development/index.md | Added documentation about lazy ASTs and serialization |
| docs/javascript-api/index.md | Added documentation for AST serialization helpers |
Comments suppressed due to low confidence (1)
test/function/samples/serialize-ast/undefined-for-optional/_config.js:1
- The test expects
exports.xto be an empty array[], but the AST defines it as{ type: 'Literal', value: 42, raw: '42' }. The assertion will fail because the code saysexport const x = [];but the AST specifies the init value as the literal42instead of an empty array.
565bd0a to
26e364f
Compare
ed9d5d5 to
b92aeb4
Compare
9 tasks
b92aeb4 to
acdfc3d
Compare
3b8a718 to
9738148
Compare
acdfc3d to
05992df
Compare
* Convert scripts to TypeScript * Auto-generate types * Remove estree dependency * Use ! instead of declare for more concise syntax * Replace explicit enums with generated enums * Use Super instead of SuperElement to better align with type names * Remove small and little-used interfaces * Replace LeftHandSideExpression with Expression This is what the proposed EStree update suggests * Get rid of StatementBase in favor of NodeBase * Also generate and use union types for builtins * Generate parent types * Improve types * Remove many node identifiers
* Convert scripts to TypeScript * Auto-generate types * Remove estree dependency * Use ! instead of declare for more concise syntax * Replace explicit enums with generated enums * Use Super instead of SuperElement to better align with type names * Remove small and little-used interfaces * Replace LeftHandSideExpression with Expression This is what the proposed EStree update suggests * Get rid of StatementBase in favor of NodeBase * Also generate and use union types for builtins * Generate parent types * Improve types * Remove many node identifiers
TODO: - optionalFallback - enlarge buffer as needed
TODO: Grow buffer as needed
TODO: Provide a way to debug invalid ASTs and fix tests
This will have negative performance implications if we need all properties of the AST, e.g. for an exhaustive tree-walker. Thus, we should provide builtin performant ways for doing this.
ccc3f9d to
ce48fae
Compare
Member
Author
|
I will improve coverage in #6271 further |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR contains:
Are tests included?
Breaking Changes?
List any relevant issue numbers:
Description
This finally removes the need to generate full ESTree ASTs at all in many situations and directly stores the binary buffers in the cache.
All generated ASTs except the ones returned by this.parse and the parseAst(Async) helpers will now be generated lazily, which means that all non-primitive properties of AST nodes are getters that are replaced with the actual properties on first access. This should severely reduce processing time if only some of the nodes are needed.
This will likely have quite some performance impact, especially when a standard ESTree-Walker is used. There might be a follow-up PR to provide an efficient interface to walk an AST without materializing nodes that are not needed.
Additionally, it exposes new helpers for manually converting an AST to binary and back.
Exports from
rollup/parseAstparseAst/parseAstAsync— return eager (fully materialized) ASTsparseLazyAst/parseLazyAstAsync— new helpers returning lazy ASTs (same signatures asparseAst/parseAstAsync)deserializeAst— new helper that deserializes a binary buffer to an eager AST (symmetric counterpart todeserializeLazyAst)deserializeLazyAst— new helper that deserializes a binary buffer to a lazy ASTserializeAst— create binary buffer from a JSON ASTThe lazy and non-lazy deserializers have interchangeable signatures.
BREAKING CHANGES