Repository navigation
fix(influxdb3_wal): skip empty WalPeriods during replay to avoid panic - #27565
Open
TheManishCode wants to merge 1 commit into
Open
TheManishCode wants to merge 1 commit into
TheManishCode wants to merge 1 commit into
Conversation
influxdata#26237) Old WAL files written before an empty WriteBatch was skipped can have their min/max timestamps left at the fold sentinels (i64::MAX/i64::MIN). WalPeriod::new asserts min_time <= max_time, so replaying one of these pre-existing files on startup panicked and crashed the server. Guard replay_wal_period so it detects the sentinel case and skips registering a WalPeriod for that file (there's no data in it to track anyway), while still advancing the wal file sequence number and letting the rest of the replay path run as before. The live flush path already avoids this because it writes a no-op before building an empty period, so this only affects replay of old files.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #26237
Problem
Old WAL files written before an empty
WriteBatchwas skipped can have their min/max timestamps left at the fold sentinels (i64::MAX/i64::MIN).WalPeriod::new()assertsmin_time <= max_time, so replaying one of these pre-existing files on startup panics and crashes the server, with no way to get past it short of manually editing/deleting object-store keys.Root cause
WalObjectStore::replay()callsFlushBuffer::replay_wal_period, which built aWalPeriod::new(...)unconditionally from each replayed file's min/max timestamps. An empty batch leaves those fields at the sentinel values, tripping the assert. The live flush path (flush_buffer_into_contents_and_responses) doesn't hit this because it inserts a no-op before building an empty period via a plain struct literal — so this only affects replay of old, pre-existing files.Fix
replay_wal_periodnow takes the raw(wal_file_number, min_timestamp_ns, max_timestamp_ns)instead of a pre-builtWalPeriod, so the guard lives in the one shared function every caller routes through. It still always advanceswal_buffer.wal_file_sequence_number, but skips constructing/registering aWalPeriodwhen the sentinel case is detected, logging awarn!instead. All other replay side effects (file_notifier.notify, snapshot cleanup) are unchanged.Tests
Added
test_replay_wal_period_skips_empty_sentinel_valuesininfluxdb3_wal/src/object_store/tests.rs, asserting no panic on the sentinel case, thatsnapshot_tracker.num_wal_periods()stays at 0 for it, and that the wal file sequence number still advances; also checks a normal period is still tracked as before.