Allowlist rdb_test_ DDLWorker UUID-collision upgrade-check restart noise by groeneai · Pull Request #107126 · ClickHouse/ClickHouse

groeneai · 2026-06-11T01:47:19Z

Changelog category (leave one):

CI Fix or Improvement (changelog entry is not required)

Changelog entry (a user-readable short description of the changes that goes into CHANGELOG.md):

...

Description

Allowlists a benign, self-healing error from the --replicated-database test wrapper that trips the upgrade-check post-restart <Error> scrub. Chronic CI noise on Upgrade check (amd_release), not a code regression.

The failing line (repeated every ~5s as the DDLWorker retries):

<Error> DDLWorker(rdb_test_mur4ipro_2): Error on initialization of rdb_test_mur4ipro_2: Code: 57. DB::Exception: Mapping for table with UUID=69047a76-6be0-4eef-8696-576514388289 already exists. It happened due to UUID collision, most likely because some not random UUIDs were manually specified in CREATE queries. (TABLE_ALREADY_EXISTS), Stack trace ...

clickhouse-test --replicated-database names each test DB rdb_test_<rnd>_<shard> with ENGINE=Replicated(...). On the upgrade restart its DDLWorker runs DatabaseReplicatedDDLWorker::initializeReplication -> recoverLostReplica, re-creating tables from the ZooKeeper metadata snapshot. If a stale local table still owns a table's UUID (a leftover _tmp_replace_* from CREATE OR REPLACE, or a table not yet finally dropped), DatabaseCatalog::addUUIDMapping reports the collision as a non-fatal TABLE_ALREADY_EXISTS (code 57), not a LOGICAL_ERROR. initializeMainThread catches it, logs this line, waits 5s and retries; recovery self-heals (after max_retries_before_automatic_recovery retries it forces a digest reset). The server stays up - every other upgrade-check sub-test passes, including "Server successfully started"; only the <Error> scrub trips.

The two existing UUID-mapping masks do not cover this case: one hardcodes a single UUID (useless, UUIDs are random), the other matches only _repl_01111_ databases. The added regex requires ALL of Error on initialization of (logged at exactly one site, the DDLWorker recovery retry), the rdb_test_ test-DB prefix, the UUID mapping message, AND the TABLE_ALREADY_EXISTS code together. So a real LOGICAL_ERROR UUID-mapping crash, the same collision on a non-test database, a different init failure on an rdb_test_ DB, and unrelated TABLE_ALREADY_EXISTS errors all still surface. This mirrors the narrowing approach of the existing multi-substring filters in the same scrub.

Provenance: Upgrade check (amd_release) on PR #106107, commit 942a72f, sub-test "Error message in clickhouse-server.log".
Report: https://s3.amazonaws.com/clickhouse-test-reports/json.html?PR=106107&sha=942a72f5e14987e3d86f7011d1b688ed4a11e9ea&name_0=PR&name_1=Upgrade%20check%20%28amd_release%29
Chronic across many unrelated PRs in the last 30 days (e.g. #101757, #103952, #101033, #106230).

groeneai · 2026-06-11T01:47:45Z

Pre-PR validation gate

#	Question	Answer
a	Deterministic repro?	Yes. The exact CI log line is captured from the failing run (PR #106107, `Upgrade check (amd_release)`, sub-test "Error message in clickhouse-server.log"). Fed through the real allow-list pipeline (`rg -Fav` first stage + the `grep -av` secondary pipes from `upgrade_runner.sh`): the benign line is suppressed deterministically while 4 counter-examples surface every run.
b	Root cause explained?	`clickhouse-test --replicated-database` names each test DB `rdb_test_<rnd>_<shard>` (`ENGINE=Replicated`). On the upgrade restart its DDLWorker runs `initializeReplication` -> `recoverLostReplica`, re-creating tables from the ZK metadata snapshot. A stale local table still owning a UUID (`_tmp_replace_*` leftover, or not-yet-finally-dropped table) makes `addUUIDMapping` return a non-fatal `TABLE_ALREADY_EXISTS` (code 57, not `LOGICAL_ERROR`). `initializeMainThread` catches it, logs the `<Error>` line, waits 5s, retries; recovery self-heals. Server stays up — only the post-restart `<Error>` scrub trips.
c	Fix matches root cause?	Yes. The error is benign, self-healing log noise; the correct, precedented fix for benign upgrade-restart `<Error>` noise is a targeted entry in `upgrade_runner.sh`'s allow-list (same class as the existing `_repl_01111_`, Kafka2-broker, and `metric_log` `DEADLOCK_AVOIDED` masks). No engine behavior is changed.
d	Test intent preserved / new tests added?	Yes. The mask is maximally narrow: it requires ALL of `Error on initialization of` (logged at exactly one site, the DDLWorker recovery retry), the `rdb_test_` test-DB prefix, the UUID-mapping message, AND `TABLE_ALREADY_EXISTS` together. A real `LOGICAL_ERROR` UUID-mapping crash, the same collision on a non-test DB, a different init failure on an `rdb_test_` DB, and unrelated `TABLE_ALREADY_EXISTS` all still surface — so the scrub keeps its ability to catch real regressions.
e	Both directions demonstrated?	Yes. Without the mask the benign `rdb_test_*` line lands in `upgrade_error_messages.txt` and FAILs the check (the actual CI failure). With the mask it is removed; on a 6-line corpus (2 benign retries + 4 counter-examples) exactly the 4 counter-examples remain after the pipeline, 0 benign lines.
f	Fix is general, not a narrow patch?	N/A — this is a CI log-scrubber allow-list entry, not an engine code fix. The benign condition has one log site (`DatabaseReplicatedWorker.cpp` `tryLogCurrentException("Error on initialization of ...")`); the regex anchors on it. The underlying non-fatal `TABLE_ALREADY_EXISTS` reporting is intended engine behavior (a sibling crash on the load path is handled separately in PR #107110).

Session id: cron:clickhouse-worker-slot-24:20260611-012600

groeneai · 2026-06-11T01:48:26Z

cc @alexey-milovidov — could you review this? Same class as #106616 and #106516: a targeted upgrade_runner.sh allow-list entry for the benign, self-healing DDLWorker(rdb_test_*) ... Mapping for table with UUID=... already exists (TABLE_ALREADY_EXISTS) line the --replicated-database wrapper logs during the upgrade restart. The regex requires the rdb_test_ prefix + Error on initialization of + the UUID message + the code together, so real LOGICAL_ERROR crashes and non-test collisions still surface.

clickhouse-gh · 2026-06-11T02:11:48Z

Workflow [PR], commit [b97afb4]

Summary: ✅

AI Review

Summary

This PR adds a new upgrade_runner.sh allow-list entry for a DatabaseReplicated UUID-collision message emitted during the post-upgrade restart. I do not think the current change is safe as written: it suppresses the only remaining signal that a replicated test database may still be stuck in recoverLostReplica after upgrade, while the job does not add any replacement assertion that those databases actually finish recovery.

Findings

❌ Blockers

[tests/docker_scripts/upgrade_runner.sh:530] DatabaseReplicatedDDLWorker::initializeMainThread logs this exact Error on initialization of ... line for every failed recovery attempt and just sleeps/retries. The upgrade job only proves that the server starts and answers a trivial query; it never exercises the leftover rdb_test_* databases again, and clickhouse-test --upgrade-check intentionally leaves those databases behind via --fake-drop. After this new grep -av, a real post-upgrade recoverLostReplica regression that leaves an rdb_test_* database stuck in the same UUID-collision loop will pass as long as the server stays up.
Suggested fix: keep this error visible, or add a post-restart assertion that every rdb_test_* database finishes recovery and can be enumerated/queried before suppressing the line.

Final Verdict

Status: ⚠️ Request changes

Add a replacement post-upgrade check that proves the rdb_test_* replicated databases recover successfully, or do not mask this error.

groeneai · 2026-06-11T23:57:48Z

CI summary, covered HEAD `9a74820c`

CI fully finished (run=0, fail=0, aggregator complete, >20h buffer). No test failures. The only non-success is CH Inc sync (CIDB staging-cluster log sync), a known non-blocking internal-cluster condition, not a test failure. This is a test-harness-only allow-list change (upgrade_runner.sh), so a clean run is expected. Ready for review.

…noise The upgrade-check post-restart `<Error>` scrub catches a benign, self-healing error from the `--replicated-database` test wrapper: ``` <Error> DDLWorker(rdb_test_mur4ipro_2): Error on initialization of rdb_test_mur4ipro_2: Code: 57. DB::Exception: Mapping for table with UUID=69047a76-6be0-4eef-8696-576514388289 already exists. It happened due to UUID collision, most likely because some not random UUIDs were manually specified in CREATE queries. (TABLE_ALREADY_EXISTS), Stack trace ... ``` `clickhouse-test --replicated-database` creates each test's database as `ENGINE=Replicated(...)`, named `rdb_test_<rnd>_<shard>`. On the upgrade restart that database's DDLWorker runs `DatabaseReplicatedDDLWorker::initializeReplication` -> `recoverLostReplica`, which re-creates tables from the ZooKeeper metadata snapshot. If a stale local table still owns a table's UUID (a leftover `_tmp_replace_*` from `CREATE OR REPLACE`, or a table not yet finally dropped), `DatabaseCatalog::addUUIDMapping` reports the collision as a non-fatal `TABLE_ALREADY_EXISTS` (code 57), not a `LOGICAL_ERROR`. The DDLWorker main loop (`initializeMainThread`) catches it, logs this `<Error>` line, waits 5s and retries; recovery self-heals (after `max_retries_before_automatic_recovery` retries it forces a digest reset). The server stays up, so every other upgrade-check sub-test passes (including "Server successfully started"); only the post-restart `<Error>` scrub trips. This is chronic CI noise across many unrelated PRs, not a code regression. The two existing UUID-mapping masks do not cover this: one hardcodes a single UUID (useless, UUIDs are random), the other matches only `_repl_01111_` databases. The new regex requires ALL of `Error on initialization of` (logged at exactly one site, the DDLWorker recovery retry), the `rdb_test_` test-DB prefix, the UUID mapping message, AND the `TABLE_ALREADY_EXISTS` code together. A real `LOGICAL_ERROR` UUID-mapping crash, the same collision on a non-test database, a different init failure on an `rdb_test_` DB, and unrelated `TABLE_ALREADY_EXISTS` errors all still surface. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

clickhouse-gh · 2026-07-20T07:04:37Z

           -e "Exception during get topic partitions from Kafka: Local: Broker transport failure" \
    /test_output/clickhouse-server.upgrade.log \
    | grep -av -e "_repl_01111_.*Mapping for table with UUID" \
+    | grep -av -e "Error on initialization of rdb_test_.*Mapping for table with UUID=.*already exists.*TABLE_ALREADY_EXISTS" \


DatabaseReplicatedDDLWorker::initializeMainThread logs this exact Error on initialization of ... line for every failed recovery attempt and then just waits 5s and retries. In this job we only prove that the server process starts and answers a trivial query; we never touch the leftover rdb_test_* databases again, and clickhouse-test --upgrade-check runs with --fake-drop, so those replicated databases are intentionally left behind for the post-upgrade phase.

That means this new grep -av no longer only suppresses transient noise: it also hides a real recoverLostReplica regression if an rdb_test_* database gets stuck in the same UUID-collision retry loop after upgrade. As long as the server answers SELECT 1, the upgrade check will now pass with no remaining signal that replicated-database recovery is broken. We need an extra post-restart assertion that the rdb_test_* databases actually finish recovery before masking this line.

Verified against source. Your two facts check out:

Error on initialization of {db} is emitted at one site, src/Databases/DatabaseReplicatedWorker.cpp:131, in the while (!stop_flag) catch that waits 5s and retries, so grep -av drops every occurrence, not just transient ones.

The rdb_test_* DBs are intentionally left by --fake-drop (tests/clickhouse-test:6736) and hit initializeReplication -> recoverLostReplica on restart.

The problem with a post-restart assertion is timing: upgrade_runner.sh stops the server at line 322 and only scrubs clickhouse-server.upgrade.log at line 454, so there is no live server left to query system.databases. The only remaining signal is log-string correlation (require Finishing replica initialization, line 264, for every errored rdb_test_ DB). That is racy: recovery retries on a 5s cadence and stop_server can land between an error and the next successful attempt on a fully benign run, which would fail the upgrade check for every PR. A new cross-PR flake in shared CI is worse than one masked benign line.

Residual risk acknowledged: after max_retries_before_automatic_recovery (default 10, DatabaseReplicatedSettings.cpp:23) the loop force-resets the digest and re-recovers from the ZK snapshot; a regression that still fails past that point would keep this line masked. I am weighing that against the flake cost above, and against the 4-token regex still surfacing a LOGICAL_ERROR UUID crash, the same collision on a non-rdb_test_ DB, a different init failure on an rdb_test_ DB, or an unrelated TABLE_ALREADY_EXISTS. This matches the merged precedent for benign scrub noise (#106616, #106516).

Keeping the line-level allowlist. If a maintainer prefers a stricter guard, a count threshold (fail if the collision repeats more than K times) is the least-fragile option, but the flake-vs-signal call is theirs.

alexey-milovidov added the can be tested Allows running workflows for external contributors label Jun 11, 2026

clickhouse-gh Bot added the pr-ci label Jun 11, 2026

groeneai force-pushed the groeneai/upgrade-check-allowlist-rdb-uuid-collision branch from 9a74820 to 7fc5706 Compare July 1, 2026 11:29

clickhouse-gh Bot added the manual approve Manual approve required to run CI label Jul 1, 2026

groeneai force-pushed the groeneai/upgrade-check-allowlist-rdb-uuid-collision branch from 7fc5706 to b97afb4 Compare July 2, 2026 06:03

groeneai mentioned this pull request Jul 4, 2026

Fix LOGICAL_ERROR in HashJoin::buildAdditionalFilter with join_use_nulls and a non-equi JOIN condition #107957

Open

This was referenced Jul 17, 2026

Add ZXC compression codec #110620

Merged

Speed up groupUniqArray for numeric types #110917

Merged

alexey-milovidov added can be tested Allows running workflows for external contributors and removed can be tested Allows running workflows for external contributors labels Jul 20, 2026

clickhouse-gh Bot reviewed Jul 20, 2026

View reviewed changes

alexey-milovidov mentioned this pull request Jul 23, 2026

Parse unquoted JSON numbers for DateTime64 and DateTime as Unix timestamps #108091

Open

1 task

groeneai mentioned this pull request Jul 23, 2026

Fsync projection parts to survive power loss #111392

Open

alexey-milovidov mentioned this pull request Jul 23, 2026

Parallel full sorting merge join (parallel_full_sorting_merge) #109005

Open

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Allowlist rdb_test_ DDLWorker UUID-collision upgrade-check restart noise#107126

Allowlist rdb_test_ DDLWorker UUID-collision upgrade-check restart noise#107126
groeneai wants to merge 1 commit into
ClickHouse:masterfrom
groeneai:groeneai/upgrade-check-allowlist-rdb-uuid-collision

groeneai commented Jun 11, 2026

Uh oh!

groeneai commented Jun 11, 2026

Uh oh!

groeneai commented Jun 11, 2026

Uh oh!

clickhouse-gh Bot commented Jun 11, 2026 •

edited

Loading

Uh oh!

groeneai commented Jun 11, 2026

Uh oh!

clickhouse-gh Bot Jul 20, 2026

Uh oh!

groeneai Jul 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

groeneai commented Jun 11, 2026

Changelog category (leave one):

Changelog entry (a user-readable short description of the changes that goes into CHANGELOG.md):

Description

Uh oh!

groeneai commented Jun 11, 2026

Pre-PR validation gate

Uh oh!

groeneai commented Jun 11, 2026

Uh oh!

clickhouse-gh Bot commented Jun 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

AI Review

Summary

Findings

Final Verdict

Uh oh!

groeneai commented Jun 11, 2026

CI summary, covered HEAD 9a74820c

Uh oh!

clickhouse-gh Bot Jul 20, 2026

Choose a reason for hiding this comment

Uh oh!

groeneai Jul 20, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

clickhouse-gh Bot commented Jun 11, 2026 •

edited

Loading

CI summary, covered HEAD `9a74820c`