Skip to content
Permalink

Comparing changes

Choose two branches to see what’s changed or to start a new pull request. If you need to, you can also or learn more about diff comparisons.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also . Learn more about diff comparisons here.
base repository: feldera/feldera
Failed to load repositories. Confirm that selected base ref is valid, then try again.
Loading
base: main
Choose a base ref
...
head repository: feldera/feldera
Failed to load repositories. Confirm that selected head ref is valid, then try again.
Loading
compare: jemalloc_tuning
Choose a head ref
Checking mergeability… Don’t worry, you can still create the pull request.
  • 3 commits
  • 5 files changed
  • 1 contributor

Commits on Mar 3, 2026

  1. [dbsp] ListMerger and input_map benchmarks. 

    Two benchmarks to help with performance tuning:
- The ListMerger is the algorithm we use to merge batches in background worker threads.
  We benchmark it by merging N randomly generated indexed Z-set on-disk or in-memory.
- The input map benchmark simulates inges into a table with a primary key using a
  circuit with a single input_map operator.

Signed-off-by: Leonid Ryzhyk <ryzhyk@gmail.com>
    @ryzhyk
    ryzhyk committed Mar 3, 2026
    Configuration menu
    Copy the full SHA
    ed6d2ce View commit details
    Browse the repository at this point in the history

  2. [dbsp] Use binary heaps in ListMerger. 

    This commit applies the same optimization to ListMerge that we previously
implemented for CursorList: replace linear scan for min values at each step of
the cursor with a binary heap that keeps the cursors partially sorted and only
does O(log(n) x m) work at each step, where n is the number of cursors and m is
the number of cursors that point to the current min key or value.

Benchmark results on ListMerger benchmarks. The `key range: 100` column
represents the workload with many keys and few values per key; the key range:
100000000 workload is lots of unique keys in each batch a small number of values
per key.

The key in these benchmarks is u64; the value is Tup10<u64,...,u64>.

This implementation doesn't yet contain some of the lower-level optimizations we
implemented for CursorList: replacing array indexig with `get_unchecked` and storing
raw pointers to keys and values instead of reading them on each access.

BEFORE

Memory-backed batches
┌─────────────┬────────────────────┬──────────────────────────┐
│ # Batches   │ key range: 100     │ key range: 100000000     │
├─────────────┼────────────────────┼──────────────────────────┤
│           1 │                7.2 │                      3.0 │
│           8 │                5.0 │                      2.6 │
│          32 │                3.2 │                      2.0 │
│          64 │                2.2 │                      1.6 │
└─────────────┴────────────────────┴──────────────────────────┘
File-backed batches
┌─────────────┬────────────────────┬──────────────────────────┐
│ # Batches   │ key range: 100     │ key range: 100000000     │
├─────────────┼────────────────────┼──────────────────────────┤
│           1 │                5.3 │                      2.2 │
│           8 │                4.3 │                      1.9 │
│          32 │                3.1 │                      1.7 │
│          64 │                2.4 │                      1.4 │
└─────────────┴────────────────────┴──────────────────────────┘

AFTER

Memory-backed batches
┌─────────────┬────────────────────┬──────────────────────────┐
│ # Batches   │ key range: 100     │ key range: 100000000     │
├─────────────┼────────────────────┼──────────────────────────┤
│           1 │                7.4 │                      3.1 │
│           8 │                5.6 │                      2.6 │
│          32 │                4.9 │                      2.3 │
│          64 │                4.4 │                      2.1 │
└─────────────┴────────────────────┴──────────────────────────┘
File-backed batches
┌─────────────┬────────────────────┬──────────────────────────┐
│ # Batches   │ key range: 100     │ key range: 100000000     │
├─────────────┼────────────────────┼──────────────────────────┤
│           1 │                5.3 │                      2.2 │
│           8 │                4.4 │                      1.9 │
│          32 │                4.0 │                      1.8 │
│          64 │                3.8 │                      1.7 │
└─────────────┴────────────────────┴──────────────────────────┘

Signed-off-by: Leonid Ryzhyk <ryzhyk@gmail.com>
    @ryzhyk
    ryzhyk committed Mar 3, 2026
    Configuration menu
    Copy the full SHA
    eaa6548 View commit details
    Browse the repository at this point in the history

  3. Tune jemalloc for speed. 

    Signed-off-by: Leonid Ryzhyk <ryzhyk@gmail.com>
    @ryzhyk
    ryzhyk committed Mar 3, 2026
    Configuration menu
    Copy the full SHA
    ee8d052 View commit details
    Browse the repository at this point in the history
Loading