Allowlist `system.metric_log` flush DEADLOCK_AVOIDED in upgrade-check log scrub by groeneai · Pull Request #106616 · ClickHouse/ClickHouse

groeneai · 2026-06-05T23:51:07Z

Changelog category (leave one):

CI Fix or Improvement (changelog entry is not required)

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

...

Documentation entry for user-facing changes

Documentation is unchanged.

Modify your CI run

(default settings, no labels)

The upgrade-check post-restart <Error> log scrub catches a transient lock-timeout emitted by SystemLog<MetricLogElement>::flushImpl when the background MetricLog flush loop races with shutdown work on system.metric_log during the upgrade restart sequence. The signature, from CIDB:

<Error> void DB::SystemLog<DB::MetricLogElement>::flushImpl(...) [LogElement = DB::MetricLogElement]: Failed to flush system log system.metric_log with N entries up to offset M: Code: 473. DB::Exception: WRITE locking attempt on "system.metric_log" has timed out! (60000ms) Possible deadlock avoided. Client should retry. Owner query ids: . (DEADLOCK_AVOIDED), Stack trace ...

Another worker (DROP/RENAME/DETACH on system.metric_log, or a parallel mutation/merge) holds the table-level write lock during the upgrade-test restart, the flush blocks for the 60s timeout, and then aborts with code 473 (DEADLOCK_AVOIDED). Code 473 is the engine's own "I gave up to avoid a deadlock, not a crash" signal — losing a few MetricLogElement samples during shutdown is harmless; the upgrade-check stage is not asserting on metric continuity.

CIDB confirms this is chronic CI noise, not a code regression:

4 hits across 4 distinct PRs in the last 30 days, 0 master, 100% on Upgrade check (amd_release)
Reported by @ alexey-milovidov on Allow cache disk for datalake table engines. #102017 (cited PRs Fix used_privileges leaking across unrelated queries in system.query_log #106425, Avoid pure-virtual abort during MergeTree destruction race (STID 3631-4165) #104561, Fix Logical error on OPTIMIZE PARTITION for Time-typed partition key #106202)

The added regex requires BOTH the SystemLog flush wrapper for metric_log AND the DEADLOCK_AVOIDED error code together, mirroring the narrowing approach of the existing rdk:FAIL.*Connect to.*failed: Connection refused and StorageKafka2.*Broker transport failure filters. Real metric_log regressions (other flush errors) and unrelated DEADLOCK_AVOIDED hits on other system_log tables (query_log, text_log, etc.) still surface.

Related: #102017
Related: #106563
Related: #101576

Version info

Merged into: 26.6.1.461

…utdown noise The upgrade-check post-restart `<Error>` scrub catches a transient lock-timeout emitted by `SystemLog<MetricLogElement>::flushImpl` when the background `MetricLog` flush loop races with shutdown work on `system.metric_log` during the upgrade restart sequence: ``` <Error> void DB::SystemLog<DB::MetricLogElement>::flushImpl(...) [LogElement = DB::MetricLogElement]: Failed to flush system log system.metric_log with N entries up to offset M: Code: 473. DB::Exception: WRITE locking attempt on "system.metric_log" has timed out! (60000ms) Possible deadlock avoided. Client should retry. Owner query ids: . (DEADLOCK_AVOIDED), Stack trace ... ``` Another worker (DROP/RENAME/DETACH on `system.metric_log`, or a parallel mutation/merge) holds the table-level write lock, the flush blocks for the 60s timeout, and then aborts with code 473 (`DEADLOCK_AVOIDED`). The 473 code is the engine's own "gave up to avoid a deadlock, not a crash" signal — losing a few `MetricLogElement` samples during shutdown is harmless; the upgrade-check stage is not asserting on metric continuity. CIDB confirms this is chronic CI noise, not a code regression: 4 hits across 4 distinct PRs in the last 30 days, 0 master, 100% on `Upgrade check (amd_release)`. Reported by @ alexey-milovidov on PR ClickHouse#102017 (cited ClickHouse#106425, ClickHouse#104561, ClickHouse#106202). The added regex requires BOTH the `SystemLog` flush wrapper for `metric_log` AND the `DEADLOCK_AVOIDED` error code together, mirroring the narrowing approach of the existing `rdk:FAIL.*Connect to.*failed: Connection refused` and `StorageKafka2.*Broker transport failure` filters. Real `metric_log` regressions (other flush errors) and unrelated `DEADLOCK_AVOIDED` hits on other system_log tables (`query_log`, `text_log`, etc.) still surface. Related: ClickHouse#102017 Related: ClickHouse#106563 Related: ClickHouse#101576 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

groeneai · 2026-06-05T23:52:08Z

Pre-PR validation gate

#	Question	Answer
a	Deterministic repro?	N/A (CI scrub allowlist, not a code bug). The behaviour change is in a `grep -av` filter; correctness verified by a minimal sed/grep harness against four hand-crafted log lines (target line + 3 controls).
b	Root cause explained?	`SystemLog<MetricLogElement>::flushImpl` runs a background flush loop. During the upgrade-test restart sequence, another worker (`DROP`/`RENAME`/`DETACH` on `system.metric_log`, or a parallel mutation/merge) holds the table-level write lock. The flush attempts to acquire it, blocks for 60s, and then aborts with code 473 (`DEADLOCK_AVOIDED`). 473 is the engine's own "gave up to avoid a deadlock, not a crash" signal — the discarded `MetricLogElement` samples are harmless; the upgrade-check stage is not asserting on metric continuity.
c	Fix matches root cause?	Yes. The chronic noise is benign by design (the engine's own DEADLOCK_AVOIDED protection during shutdown). Like the existing `rdk:FAIL.Connect to.failed: Connection refused` and `StorageKafka2.*Broker transport failure` filters, this is the right scope: filter the specific `SystemLog`+`metric_log`+`DEADLOCK_AVOIDED` signature in the upgrade-check log scrub. Not a code change.
d	Test intent preserved / new tests added?	Yes. The upgrade-check log scrub still catches every other class of error. Local sed/grep harness confirmed with 3 control lines: an unrelated `<Error>` is kept; a `metric_log` flush failure with a DIFFERENT code is kept (real flush bug not masked); a DEADLOCK_AVOIDED on a DIFFERENT system_log table (`query_log`) is kept (other log tables not masked).
e	Both directions demonstrated?	Yes. Without the fix, the exact CI signature line would survive the scrub and trip "Error message in clickhouse-server.log". With the fix (sed/grep harness), the target line is filtered and the 3 controls pass through.
f	Fix is general, not a narrow patch?	The narrow regex is the GENERAL fix for this class of CI scrub work: require both the wrapper context AND the specific error code together. Widening (e.g. blanket `DEADLOCK_AVOIDED` filter, or blanket `metric_log` filter) would mask real DEADLOCK_AVOIDED bugs in other tables, or real `metric_log` regressions with other root causes. The pattern matches the precedent set by PR #106563 and the older `rdk:FAIL.Connect to.failed: Connection refused` filter.

Session id: cron:clickhouse-ci-task-worker:20260605-234100

groeneai · 2026-06-05T23:53:14Z

cc @ alexey-milovidov — could you review this? It mirrors the narrowing approach of your merged #106563 (StorageKafka2 broker noise) for the SystemLog<MetricLogElement> flush DEADLOCK_AVOIDED chronic flake you flagged on PR #102017.

clickhouse-gh · 2026-06-06T04:41:31Z

Workflow [PR], commit [03805a3]

Summary: ✅

AI Review

Summary

This PR narrows the upgrade-check post-restart log scrub by allowlisting only SystemLog flush failures for system.metric_log that also carry DEADLOCK_AVOIDED. The regex is scoped to the documented signature and still lets unrelated metric_log flush failures, unrelated DEADLOCK_AVOIDED errors, and other SystemLog tables surface.

Final Verdict

Status: ✅ Approve

groeneai · 2026-06-06T07:05:55Z

CI Ledger — covered SHA: `03805a3`

job	classification	disposition
(none)	(no failures)	All checks pass or skipped (31 SUCCESS / 135 SKIPPED). 0 test failures across full PR history. AI Review: Approve. Finish Workflow + Mergeable Check + Upgrade check (amd_release) all SUCCESS.

Gate satisfied: no queued/in_progress checks, Finish Workflow + Mergeable Check complete, ≥20-min CIDB ingestion buffer met (last activity 04:41Z, ~2.5h ago).

waiting_ci → on_review.

groeneai mentioned this pull request Jun 5, 2026

Allow cache disk for datalake table engines. #102017

Merged

1 task

alexey-milovidov added the can be tested Allows running workflows for external contributors label Jun 6, 2026

clickhouse-gh Bot added the pr-ci label Jun 6, 2026

alexey-milovidov approved these changes Jun 6, 2026

View reviewed changes

alexey-milovidov self-assigned this Jun 6, 2026

alexey-milovidov added this pull request to the merge queue Jun 6, 2026

Merged via the queue into ClickHouse:master with commit 003baa5 Jun 6, 2026
166 checks passed

robot-ch-test-poll1 added the pr-synced-to-cloud The PR is synced to the cloud repo label Jun 6, 2026

This was referenced Jun 7, 2026

Fix used_privileges leaking across unrelated queries in system.query_log #106425

Merged

Allowlist rdb_test_ DDLWorker UUID-collision upgrade-check restart noise #107126

Open

groeneai mentioned this pull request Jul 19, 2026

Speed up groupUniqArray for numeric types #110917

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Allowlist `system.metric_log` flush DEADLOCK_AVOIDED in upgrade-check log scrub#106616

Allowlist `system.metric_log` flush DEADLOCK_AVOIDED in upgrade-check log scrub#106616
alexey-milovidov merged 1 commit into
ClickHouse:masterfrom
groeneai:groeneai/upgrade-check-allowlist-metric-log-deadlock

groeneai commented Jun 5, 2026 •

edited by robot-clickhouse

Loading

Uh oh!

groeneai commented Jun 5, 2026

Uh oh!

groeneai commented Jun 5, 2026

Uh oh!

clickhouse-gh Bot commented Jun 6, 2026 •

edited

Loading

Uh oh!

groeneai commented Jun 6, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

groeneai commented Jun 5, 2026 • edited by robot-clickhouse Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changelog category (leave one):

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Documentation entry for user-facing changes

Modify your CI run

Version info

Uh oh!

groeneai commented Jun 5, 2026

Pre-PR validation gate

Uh oh!

groeneai commented Jun 5, 2026

Uh oh!

clickhouse-gh Bot commented Jun 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

AI Review

Summary

Final Verdict

Uh oh!

groeneai commented Jun 6, 2026

CI Ledger — covered SHA: 03805a3

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

groeneai commented Jun 5, 2026 •

edited by robot-clickhouse

Loading

clickhouse-gh Bot commented Jun 6, 2026 •

edited

Loading

CI Ledger — covered SHA: `03805a3`