Web UI prototype for framing formats by alexey-milovidov · Pull Request #110210 · ClickHouse/ClickHouse

alexey-milovidov · 2026-07-13T04:17:42Z

Prototype changes to the Web UI (programs/server/play.html) to test and showcase the framing formats feature. This is a draft for experimentation and review of the client-side experience, not intended to merge as-is.

It builds on the framing-formats server work in #110127 (this branch is based on that PR, so the diff includes those commits until it merges into master).

What the prototype does:

Streams every query with the EventStream framing format over HTTP (framing_output_format=EventStream, send_logs_level=trace), decoding data / progress / log / profile_events / exception packets as they arrive - so progress and logs work regardless of the output format (including FORMAT Pretty, binary formats via base64, and images).
Shows realtime CPU, memory, and disk usage like clickhouse-client, aggregated per host (total and max/host), switching to peak RAM when the query finishes. The meter state is owned by each tab: the CPU counters in profile_events packets are per-packet increments, so a query running in a background tab keeps accumulating them, and reopening the tab continues the meter from its live values (rather than restarting near zero).
Merges elapsed time, realtime metrics, and rows/bytes read stats into a single progress area with the progress-bar gradient rendered behind all of them; the text is tinted by a mask clipped from the same gradient so it stays readable over the fill.
Displays server logs in real time, colored like the client, with a Logs button available even on exception; the log view stays fast for 100k+ lines while keeping native browser search, bounding retention to the most recent 100k lines (older lines are dropped and counted in a marker line). The full response text kept for the tab/history snapshot is also capped as it is collected (at ~100 KB, the size above which the snapshot is dropped), so a single large framed data/log burst is not retained in full only to be discarded later. A failed run whose reply outgrew that cap is persisted as a compact snapshot that keeps the exception carrier in the form its framing kind replays (the terminal event: exception block, the {"packet":"exception",...} line, or the plain {"exception":...} line, captured at stream-read time since the capped text stops growing before the terminal exception arrives), so reopening or reloading a failed tab still shows the error reason.
Renders inline images for image output formats, and base64-decodes binary payloads.
Uses a framing-compatible default format for queries without an explicit FORMAT clause: the framed request asks for JSONCompactStringsEachRowWithNamesAndTypes (the framing rejects the in-band-progress JSONStringsEachRowWithProgress) and the client reassembles the compact rows into the table renderer's shapes; on the server side (Framing formats: multiplex data, totals, extremes, progress, logs, and profile events in the HTTP response stream #110127) the compact family emits totals and extremes under framing, so WITH TOTALS and extremes-based column coloring keep working on the default path.
Propagates a framed exception packet as a query failure even when the HTTP status was already 200 (so Run all stops at the failed statement) - and, when a query chooses its own JSONEachPacket* framing that fails before the 200 OK header (coming back as a non-200 application/x-ndjson packet stream ending with a {"packet":"exception",...} line), shows those packets verbatim and records framing_kind = 'ndjson_packets' for replay rather than rendering the whole stream as one opaque error string; retries framing-incompatible explicit formats (e.g. FORMAT JSONEachRowWithProgress, FORMAT Template) once without framing for read-only queries (read-only-ness is resolved with a CTE-aware lexer walk - WITH y AS (SELECT 1) INSERT INTO t SELECT * FROM y is a write - shared with Run all's grouping, so such a statement is also a barrier there and never runs in parallel with the reads that follow it; the port with regression coverage lives in src/Parsers/tests/gtest_play_query_is_read_only.cpp); a retried JSON*EachRowWithProgress format that itself reports a failure in-band - a trailing {"exception":...} object while the HTTP status stays 200 (http_write_exception_in_output_format) - is detected as a failure too, keyed off the output format rather than only a user-chosen JSONEachPacket* framing, so Run all stops after such a retried query fails; and does not add its own framing to a query that sets its own framing_output_format to a real framing choice (the response is then dispatched by content type, so the requested packets are shown verbatim); a query that sets framing_output_format = 'None' is refused client-side instead, since this page's rendering depends on framing (values the page cannot know upfront - = DEFAULT, a reset to the session/server default, and query-parameter placeholders like = {fmt:String} - are classified conservatively the same way and refused, rather than sent with a request shape the response might not match); likewise a standalone SET framing_output_format = ... is refused, because it would change the setting for the whole session (with a session_id) while the page keeps adding its own framing per request - a query-level SETTINGS framing_output_format = ... clause is the supported way to choose framing for one query. A query that carries its own framing_output_format is also refused for download (the setting would override the download's chosen default_format, so the file would be the framing packet stream), and if such a query fails, its history snapshot - the raw JSONEachPacket* packet stream - is replayed as raw text on tab-switch/reload rather than as one opaque error string.
Records the framing kind (event_stream / ndjson_packets / none) with each result snapshot and keys the history/tab replay off it, instead of guessing from the payload's first bytes - so a raw result whose text happens to start with event: (e.g. SELECT 'event: data' FORMAT RawBLOB) or {"packet": is not reparsed as a framing stream after a tab switch or reload. A snapshot recorded as ndjson_packets is replayed as raw text regardless of whether the run succeeded and of its underlying output format, matching the live path - so a successful user-framed JSONEachPacket* result whose format has its own restore path (a table or a JSONCompactColumns chart) is not reparsed as that format's JSON.
Detects both the query's FORMAT clause and its framing_output_format setting with the WASM lexer rather than a raw text match, so a mention inside a string literal or a comment - e.g. SELECT 'FORMAT JSONCompactColumns' - does not make the page silently opt out of its own framing. The detection is positional, not keyword-adjacent: a settings context is recognized by its name = value list grammar (a column merely named settings does not open one), and a FORMAT clause candidate must follow a token that ends an expression (so in WITH 1 AS format SELECT format JSONCompactColumns SETTINGS max_threads = 1 both format words are identifiers, not a clause). The download reuses the same FORMAT-clause detection (now returning the clause span) to strip only a real trailing FORMAT clause from the download query, leaving text or ordinary SQL like SELECT 'FORMAT TSV' AS s untouched. Regression coverage for both lives in src/Parsers/tests/gtest_play_detect_explicit_format.cpp and gtest_play_detect_framing_setting.cpp (ports of the token walking onto the real DB::Lexer).
On the server side (Framing formats: multiplex data, totals, extremes, progress, logs, and profile events in the HTTP response stream #110127), framed pulling SELECT queries now also end with the documented final progress packet carrying result_rows / result_bytes / memory_usage, matching the native protocol and the no-result path.

Changelog category (leave one):

Not for changelog (changelog entry is not required)

Changelog entry (a user-readable short description of the changes that goes into CHANGELOG.md):

Prototype Web UI changes to test framing formats (draft).

Related: #110127

…, logs, and profile events in the HTTP response stream A framing format multiplexes different response parts of the query in a single stream - chunks of data, totals and extremes, progress packets, profile events (metrics), and server logs - everything that the native protocol supports. Framing formats are independent of output formats: they encapsulate bytes produced by any output format. The framing format is selected by the new query setting `framing_output_format` and currently applies to the HTTP protocol. Implemented framing formats: - `None` - transparently routes everything applicable to the output format, so everything works as it is by default. - `EventStream` - frames packets as HTTP server-sent events (`text/event-stream`). - `JSONEachPacketBase64` - every packet is a JSON object; the formatted data is base64-encoded. - `JSONEachPacketString` - every packet is a JSON object; the formatted data is put into a string. The framing format works as a multiplexor: the output format writes into the framing format's payload buffer, and `IOutputFormat` notifies it on packet boundaries, which wraps everything accumulated since the previous boundary into a packet. The concatenation of the payloads of all `data`, `totals`, and `extremes` packets is exactly what the output format would have produced without framing. Auxiliary packets (progress, logs, profile events, exceptions) are represented as JSON. Server logs and profile events reuse `InternalTextLogsQueue` and `ProfileEvents::getProfileEvents` - the same mechanisms as the native protocol - newly attached for HTTP queries. Exceptions are always written as the last packet of the stream, so the client can parse the response as a stream of packets. Processing of multiple queries at once is out of scope of the first implementation, but the design allows extending every packet with the information about the query index. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

When the last payload line has no trailing `\n` (for example `FORMAT JSON`), `find_first_symbols<'\n'>` returns `end`, and `pos = line_end + 1` formed a pointer past the one-past-the-end position - undefined behavior, even though the loop exited immediately afterwards. Stop when `line_end == end` instead. Addresses a review comment on #110127 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…treaming `IFramingFormat` wrote each packet to `out` and called `out.next()`, which only flushes the outermost buffer. When the HTTP output is wrapped in compression (`wrapWriteBufferWithCompressionMethod` / a `WriteBufferWithOwnMemoryDecorator`), `next` leaves the compressed bytes in the nested buffer, so `data`, `progress`, and `log` packets could sit there until the end of the query instead of being delivered interactively. Add a `flushOut` helper that also flushes the nested buffer - mirroring `IOutputFormat::flushImpl` - and use it at every packet boundary and on finalize. Addresses a review comment on #110127 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

`IOutputFormat` notifies the framing format of a `Totals` / `Extremes` packet right after `consumeTotals` / `consumeExtremes`. That assumes the bytes are already in the payload buffer, which holds for row formats but not for `Template` (`areTotalsAndExtremesUsedInFinalize` is true): it stores totals and extremes and emits them later from `finalizeImpl`, where the framing format can no longer tell them apart from the main data and would mislabel them as `data` packets. Reject such formats in `setFraming` with a clear exception instead of producing wrong output. Addresses a review comment on #110127 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

… rejection The `GROUP BY ... WITH TOTALS` queries in `04512_framing_formats` emitted a nondeterministic number of `data` packets: a `data` packet follows every output block boundary, and the number of blocks depends on the number of threads, two-level aggregation, the output block size, and external sorting - all of which the CI settings randomizer varies. This made the test fail in the `arm_asan_ubsan` targeted run and in the `amd_debug` / `amd_asan_ubsan` / `amd_tsan` / `amd_msan` flaky checks. Pin the settings that determine the block boundaries for those queries so the result is a single block. Also add a case asserting that a framing format is rejected for `Template`, which defers totals and extremes to finalize. CI report: #110127 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

… and HTTP compression Cover the two remaining test suggestions from the review: - `EventStream` with a payload whose last line has no trailing newline (`FORMAT Values` produces `(0),(1),(2)`), exercising the end-of-buffer fix. - Framing over an `enable_http_compression` response, so a regression in the nested-buffer flush that leaves packets stuck in the compression buffer is caught. Related to #110127 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

The `EventStream` and `JSONEachPacketString` framing formats embed the raw output bytes into the response as UTF-8 text: `EventStream` writes them into `data:` fields of a `text/event-stream` response, and `JSONEachPacketString` passes them through `writeJSONString`, which does not sanitize invalid UTF-8. A binary output format such as `Native` or `RowBinary` would therefore produce invalid SSE / JSON instead of a byte-preserving stream. Reject such combinations at framing selection time (mirroring the existing rejection of formats that defer totals and extremes) and point the user to `JSONEachPacketBase64`, which encodes arbitrary bytes safely. The output format is classified as text via its content type: text formats declare a charset (e.g. `text/tab-separated-values; charset=UTF-8`, `application/json; charset=UTF-8`), while binary formats use types such as `application/octet-stream` without a charset. Addresses the AI review blocker on #110127 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

The text framing formats (`EventStream`, `JSONEachPacketString`) embed the output payload as UTF-8 text, so an output format that can emit arbitrary bytes corrupts the stream. Binary formats (`Native`, `RowBinary`) were already rejected via their `application/octet-stream` content type, but raw passthrough formats slipped through: `RawBLOB` writes the column bytes verbatim while inheriting the default `text/plain; charset=UTF-8` content type, and the `TSVRaw` / `TabSeparatedRaw` / `LineAsString` / `Raw` family advertise a textual content type while calling `serializeTextRaw` (no escaping). Both can produce non-UTF-8 output, reproducing the invalid SSE / JSON stream the earlier review flagged. Content type alone is not a reliable signal for text-safety, so make it an explicit output-format capability: add `may_produce_raw_bytes` to the format registration (`markOutputFormatMayProduceRawBytes`) and mark the raw formats with it. `outputFormatProducesText` now rejects any format marked this way in addition to the content-type check. `JSONEachPacketBase64` is unaffected, since it encodes arbitrary bytes safely. Extended `04512_framing_formats` to cover rejecting `RawBLOB` (always raw), `TSVRaw` and `LineAsString` (text-labeled raw) for text framings, and to confirm `JSONEachPacketBase64` still carries `RawBLOB` output. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

`FramingFormatEventStream::writeDataFields` emitted one SSE `data:` field per `\n`-terminated line of the payload. The client reconstructs `event.data` by joining the values of consecutive `data:` fields with `\n` and then stripping a single trailing `\n` (per the server-sent events specification). For a payload that ends with `\n` - the common case for line-based formats such as `JSONEachRow`, `TSV`, or `CSV` - the stripped trailing `\n` was lost, so a payload like `"row\n"` was reconstructed as `"row"`. That broke the central framing contract that concatenating the payloads of the `data`, `totals`, and `extremes` packets reproduces the output of the format without framing. Emit an extra empty `data:` field when the payload ends with `\n`, so the trailing newline survives the reconstruction. Payloads without a trailing newline (for example `FORMAT JSON`) are unchanged. Extended `04512_framing_formats` with a regression that rebuilds `event.data` from a newline-terminated `EventStream` payload and compares it byte-for-byte with the unframed output, and updated the existing `EventStream` reference and the documentation example to reflect the trailing empty `data:` field. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Pulls in the merged fix #110162 (revert of EXPLAIN ANALYZE pipeline timing, #110162), which fixes the 'clock' logical error that failed the Stress test (amd_msan) on this PR: https://s3.amazonaws.com/clickhouse-test-reports/json.html?PR=110127&sha=e774374cff8ac2e6fb0e606fe60ed73930998360&name_0=PR&name_1=Stress%20test%20%28amd_msan%29

The `EventStream` (and other) framing formats sent fewer server log packets than the native protocol: `clickhouse-client` receives log entries from the start (parsing, planning, analysis - `executeQuery`, `Planner`, `SelectExecutor`, "Reading approx N rows") and the end (`executeQuery: Read N rows`, `MemoryTracker: Query peak memory usage`) of the query, while the framing format captured only the logs emitted during pipeline execution. Two causes: - The logs and profile-events queues were attached to the thread only after the query had been interpreted, so the interpretation-phase logs were lost. Attach them before interpretation (in `attachQueuesForFramingIfApplicable`), as the native protocol does, and wire them into the framing format once it is created. - The framing format was finalized during pipeline execution, before the query-finish logging ran, so the trailing logs were lost. Defer the framing finalization (`IOutputFormat::deferFramingFinalize`) until after `onFinish`, while the HTTP response stream is still open, and log the peak memory usage explicitly before the drain (mirroring `TCPHandler`). The exception path in `HTTPHandler` finalizes the deferred framing format explicitly. With this change the framing format streams the same set of server logs as the native protocol. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

The `EventStream` framing format previously rejected output formats that are not guaranteed to produce valid UTF-8 text (binary formats such as `Native` or `RowBinary`, and raw passthrough formats such as `RawBLOB` or `TSVRaw`), because server-sent events is a text protocol. Instead of rejecting them, `EventStream` now base64-encodes the `data`, `totals`, and `extremes` payloads (each into a single `data:` field) so that arbitrary bytes survive the text transport, and signals this by adding a `payload=base64` parameter to the `Content-Type` (`text/event-stream; charset=UTF-8; payload=base64`). The client base64-decodes those payloads; their concatenation is byte-for-byte what the output format would have produced without framing. Text output formats are still embedded as plain text, and the auxiliary JSON packets (progress, logs, profile events, exception) are never encoded. This lets progress, server logs, and profile events be streamed for any output format over `EventStream`. `JSONEachPacketString` still requires a text output format (it puts the bytes into a JSON string and cannot encode arbitrary bytes), and `Template` is still rejected for all framings because it defers totals and extremes to finalize. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Rework the query progress display in the Web UI to consume the `EventStream` framing format over HTTP, so it works regardless of the output format (including `FORMAT Pretty`, binary formats via base64, and images). - Stream every query with `framing_output_format=EventStream` and `send_logs_level=trace`, decoding data/progress/log/profile_events/ exception packets as they arrive. - Show realtime CPU, memory, and disk usage like `clickhouse-client`, aggregated per host (total and max/host), switching to peak RAM when the query finishes. - Merge elapsed time and realtime metrics into a single element rendered on top of the progress bar; keep the text readable with a background-clip gradient that darkens the glyphs only over the colored fill (per-theme peak color and fraction). - Display server logs in real time, colored like the client, with a Logs button available even on exception; keep the view fast for 100k+ lines via batched appends and content-visibility while preserving native browser search. - Render inline images for image output formats, and base64-decode binary payloads signalled by the `payload=base64` content type. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Iterate on the query progress display: - Merge the elapsed time, realtime resource metrics, and rows/bytes read stats into a single progress area so the progress-bar gradient renders behind all of them; the read stats stay right-aligned and wrap when long, the elapsed/metrics text keeps its natural width. - Draw the colored bar only one text line tall, so when the read stats wrap and grow the area the fill stays a single-line strip. - Tint every glyph (left metrics and right stats alike) with a text mask clipped from the same full-width `--progress` gradient as the bar, so the two correspond exactly and the text stays readable over the fill. Light theme uses a plain gray to black; dark theme uses a steep gray to white (first quarter) then a jump to black. - Use a 10pt font for the progress area. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

mintlify · 2026-07-13T04:17:45Z

Preview deployment for your docs. Learn more about Mintlify Previews.

Project	Status	Preview	Updated (UTC)
ClickHouse-docs	🟢 Ready	View Preview	Jul 13, 2026, 11:14 AM

clickhouse-gh · 2026-07-13T04:18:28Z

Workflow [PR], commit [92361a1]

Summary: ⏳

job_name	test_name	status	info	comment
Docs check		FAIL
	Build docusaurus	FAIL	cidb

AI Review

Summary

This draft adds the server-side framing support plus the play.html prototype that consumes framed HTTP streams, including progress, logs, profile events, and client-side fallback/replay logic. I found one correctness issue in the new framing finalization path: a late failure after the final counters are stashed can still emit the success-only final progress packet before the terminal exception, which violates the stream contract the PR documents and can mislead framed clients.

Findings

⚠️ Majors

src/Processors/Formats/Framing/IFramingFormat.cpp:122 always writes final_progress before looking at exception_message. finishExecutedQuery now calls writeFinalProgress before io.onFinish(), so a late io.onFinish() failure (for example, while writing query-log state) or a later output_format->finalize() failure can recover through the same framing instance with both has_final_progress and exception_message set. The resulting failed stream carries a success-style final progress packet and then the exception, even though the new docs/tests in this PR define the terminal packet of a failed stream as exception. Suppress or clear final_progress once an exception is recorded, and add a regression that throws after flushQueryProgress / inside io.onFinish().

Final Verdict

⚠️ Changes requested.

…and framed exceptions Address the review findings on the framing formats (#110127): - `EventStream` now serializes a multi-row `profile_events` batch as a single JSON array in one `data:` field. An SSE client reconstructs `event.data` by joining consecutive `data:` fields with a newline, so emitting one field per row produced `{...}\n{...}`, which is not valid JSON and did not match the documented `profile_events` contract (an array of profile events as JSON). This mirrors the `profile_events` array of the `JSONEachPacket` framings. - Raw-byte detection for text framings is now settings-aware. `CustomSeparated` with `format_custom_escaping_rule = 'Raw'` writes the column bytes verbatim (like `TSVRaw`), so it may produce non-UTF-8 output even though it advertises a textual content type. Since this depends on a setting rather than the format name, it cannot be marked statically with `markOutputFormatMayProduceRawBytes`; a settings-aware checker (`registerOutputFormatMayProduceRawBytesChecker`) is added alongside the static flag. `EventStream` now base64-encodes such output, and `JSONEachPacketString` rejects it, pointing to `JSONEachPacketBase64`. - When the failure is the framing/output-format compatibility check itself (for example `JSONEachPacketString` with `FORMAT RowBinary`, or a framing with a format that defers totals/extremes such as `Template`), the error is now delivered as a framed `exception` packet instead of a plain HTTP error body. The exception packet is always JSON regardless of the output format, so the exception-recovery path creates the framing with a `for_exception` flag that skips the data-payload compatibility check and the deferred-totals check. - The exception-recovery formatter now wires in the logs and profile-events queues attached before the query is interpreted, so `log` / `profile_events` packets accumulated during parsing and planning are drained on `finalize` instead of being dropped when the query fails before producing output. - Update the `framing_output_format` setting description: `EventStream` no longer requires a text output format, it base64-encodes non-UTF-8 output. Extend `04512_framing_formats` and `04513_framing_formats_logs_profile_events` with regressions for each of the above. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

- Replace the Result/Logs button pair with a single "Logs" toggle at the front of the progress line (just before the elapsed/metrics text). It switches the result element(s) between the query result and the streamed server logs, shows a dotted underline, uses the monospace font, and gets a yellow background while the logs view is active. - Show the toggle only when the query produced at least one log line, so it stays hidden when a query has not started or produced no logs. - Drop the now-meaningless leading `NN.N%, ` from the read stats once the query finishes (the percentage only matters while progressing to 100%). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…xception path, document JSONEachPacketString UTF-8 contract Addresses the AI review of `3a52d17a`. `HTTPHandler` buffered exception path (`http_wait_end_of_query=1`): on an exception the buffered output is discarded and the framing format is recreated from `framing_name` alone, so the `log` and `profile_events` queues that `executeQuery` attaches before parsing (and wires into the framing) were dropped - `http_wait_end_of_query=1&framing_output_format=JSONEachPacketString&send_logs_level=trace` lost the parsing/planning `log` packets that the streaming path and the docs promise. The queues are now carried into the buffered exception writer: they are captured by value (the original framing object goes out of scope before the writer runs) and re-applied to the recreated framing via new `IFramingFormat` accessors. Added a `wait_end_of_query` regression to `04513_framing_formats_logs_profile_events`. `JSONEachPacketString` UTF-8 contract: documented that it puts the payload bytes into a JSON string without validating or re-encoding them, so text output formats such as `JSONEachRow`, `TSV` or `CSV` may emit invalid UTF-8 for `String` / `FixedString` values holding arbitrary bytes (just as ClickHouse's own `JSONEachRow` does with the default `output_format_json_validate_utf8 = 0`), in which case the NDJSON stream is not guaranteed to be valid UTF-8; pointed to `JSONEachPacketBase64` for byte-exact transport. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…b-ui

…m snapshot Address review (#110210): the history/tab replay path only distinguished the default (table) output from `rawText`; it never recorded whether the saved `EventStream` response used `payload=base64` (a binary or raw output format such as `Native` or `RawBLOB`). A restored snapshot of such a result was therefore replayed as literal base64 text, whereas the live `readEventStream` path decodes the payload to bytes and renders it as an image or raw text. Persist an `is_base64` bit with the result snapshot (single-query, multi-query, and the `saveHistory` / flat-entry serialization) and have `renderEventStreamText` decode the base64 `data`/`totals`/`extremes` payloads and hand them to `renderBinaryPayload`, exactly like the live path. A base64 snapshot is rendered as an image or raw text and is never finished as a table. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Keep the Web UI prototype current with master and clear the reported conflict with the base branch. The only conflict was a single hunk in `programs/server/play.html`, in the non-OK response branch of `postImpl`: master added query-error highlighting (`highlightQueryError`, guarded by `editorInteractionGen === runEditorInteraction`) while this branch added the `is_base64` field to the returned result and reworked the image `else if` condition to exclude framed (`text/event-stream`) responses. Resolved by keeping both: master's error-highlighting block, followed by this branch's `is_base64` return and framing-aware image branch. The `queryStart` argument that master threads into `postImpl` is already present (auto-merged), so the highlight call is in scope. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…fective query settings The `log` / `profile_events` queues for a framing format were attached once, before the query was interpreted, from the pre-parse context. The framing decision (`createFramingFormatIfApplicable`), however, uses the effective settings after the query's own `SETTINGS` clause has been applied inside `executeQueryImpl`. The two could disagree: - A query that enables framing only in its own `SETTINGS` clause (`SELECT ... SETTINGS framing_output_format='JSONEachPacketString'`) still built a framed response, but the queues stayed empty, so no `log` / `profile_events` packets ever appeared. - The inverse override (framing or `send_logs_level` / `send_profile_events` enabled in the session / URL but turned off by the query's `SETTINGS` clause) still allocated the unbounded queues and captured packets that nobody drains. Replace `attachQueuesForFramingIfApplicable` with an idempotent `syncFramingQueuesWithSettings` that brings the attached queues into agreement with the current effective settings (attaching what is wanted, dropping what is not; the thread group keeps only a weak reference, so resetting the owning `shared_ptr` detaches the queue). It is called before the query is interpreted (so parse / plan-phase logs are still captured for framing requested from the session or URL) and again after `executeQueryImpl` has applied the query's `SETTINGS` clause, on both the success and the exception paths. A framing format enabled only by a query-level `SETTINGS` clause is not known before parsing, so its queues capture from query execution onwards - the parse / plan phase logs are captured only when framing is requested from the session or URL. Add regressions to `04513_framing_formats_logs_profile_events` covering query-level `SETTINGS` for `framing_output_format`, `send_logs_level`, and `send_profile_events`, including the inverse override. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…o read-only queries Two fixes to the framing `EventStream` consumer in `play.html`, addressing the AI review of the PR: - `profile_events`: `EventStream` serializes each `profile_events` packet as a single JSON array of event objects in one `data:` field, but the client pushed every parsed line as one element, producing `[[...]]`. `feedProfileEvents` then saw the whole batch as a single element with no `thread_id` / `name`, so the realtime CPU / RAM / disk meters stopped updating for framed queries. Flatten the parsed batch into individual events (a bare object is tolerated too, for older one-per-line servers). - No-framing retry: the Web UI enables framing for every query, and a non-pulling pipeline (`INSERT`, DDL) never creates a framing stream, so such a statement can return a plain-HTTP error after it has already run. The previous code retried the whole request on any non-event-stream error, which could duplicate the side effect. Gate the retry to read-only queries (`queryIsReadOnly`, the same read-only kinds treated as parallelizable), so a side-effecting statement is never blindly resubmitted. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…group_by` in the totals boundary case The `WITH TOTALS` boundary-draining case grouped by `toString(intDiv(number, 2))`, an injective function of the key. When the CI settings randomizer sets `optimize_injective_functions_in_group_by=0`, the `GROUP BY` key stays the `String` expression and the totals row carries the empty default instead of the recomputed '0', so the result differed from the reference. Pin the setting to 0 for this request and update the reference accordingly. https://s3.amazonaws.com/clickhouse-test-reports/json.html?PR=110127&sha=4c63eb30f0fd7caa853b35e02b5cbaa97b314488&name_0=PR&name_1=Stateless%20tests%20%28arm_asan_ubsan%2C%20targeted%29 #110127

…b-ui

…ransport-failure snapshots Two fixes for the framed Web UI streaming paths: 1. The resource-meter state is now one aggregate per run instead of per stream: `Run all` executes a parallelizable group concurrently, so several framed statements of one tab can be inside `readEventStream` at once, and resetting `tab.resources` per stream discarded increments a sibling statement had already accumulated. `clearPanel` drops the previous run's state; the run's first stream creates the aggregate and every stream accumulates into it. The per-host peak gauge keeps the maximum, so a sibling statement's smaller peak cannot erase a larger one already observed. 2. The transport-error catch in `postImpl` no longer clears `framing_kind`: a framed stream can already have replayed partial output into `reply`, and forcing an empty kind sent the restore through `renderFailedSnapshot`, which is not framing-aware. The catch now preserves the kind and synthesizes the failure carrier in the form the kind replays (an `event: exception` block or a `{"packet":"exception",...}` / `{"exception":...}` line built from `display_error`), appended to the capped snapshot after a frame terminator, so a restored tab replays the framed prefix with the error beneath it - matching what was shown live. `renderEventStreamText` skips an unparseable partial block instead of throwing out of the whole restore. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…sion The totals boundary case keyed on `toString(intDiv(number, 2))`. The old analyzer eliminates injective functions from `GROUP BY` unconditionally (`optimizeGroupBy` in `TreeOptimizer.cpp`), so pinning `optimize_injective_functions_in_group_by` (which gates only the new analyzer's pass) still produced a totals key of '0' instead of the empty `String` default in the old-analyzer job. A conditional with string literals does not work either: `optimize_if_transform_strings_to_enum` rewrites it to an `Enum` whose default again changes the totals row. Use `substring`, which is not injective and not subject to either rewrite, so both analyzers keep the `String` key. Failed job: https://s3.amazonaws.com/clickhouse-test-reports/json.html?PR=110127&sha=8ef2ed09c7a818a6bb964bc210767fb09dd4a210&name_0=PR&name_1=Stateless%20tests%20%28amd_llvm_coverage%2C%20old%20analyzer%2C%20s3%20storage%2C%20DBReplicated%2C%20WasmEdge%2C%20parallel%2C%203%2F3%29 PR: #110127 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…oundaries `writeProgressIfNeededUnlocked` flushed a pending framed progress update at every packet boundary without consulting `interactive_delay`, so a stream of many small blocks emitted a `progress` packet at every `data` boundary, violating the documented contract. Apply the same throttle as the concurrent `onProgress` path and advance the shared timestamp; a still-throttled update is not lost because the final counters are delivered by the framing's final `progress` packet. Add a regression test that drives many small blocks with a one-hour `interactive_delay` and asserts the only `progress` packet is the final one. PR: #110127 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…dary failures The recovery block in `update_format_on_exception_if_needed` caught only `DB::Exception`, but the code inside can throw non-DB exceptions (for example `std::bad_function_call` via the `execute_query_calling_empty_set_result_func_on_exception` failpoint, or standard/Poco exceptions from the `set_result_details` callback), which then replaced the original query exception. Add a `catch (...)` that logs and ignores the secondary failure the same way. Add a regression test that injects the failpoint and asserts the framed response still carries the original `UNKNOWN_TABLE` error as an `exception` packet. PR: #110127 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

alexey-milovidov · 2026-07-23T06:01:57Z

@pamarcos, but we still don't have JavaScript (Playwright) tests for Web UI, it's the current compromise choice, so ok for now. I'm a little bit afraid to have 500k LOC brittle tests for 10k LOC code.

alexey-milovidov · 2026-07-23T06:03:23Z

@pamarcos, what I would appreciate is taking a look at the new features - realtime logs and realtime metric views.

…rd an analyzer comment

…b-ui

…apped snapshot The raw explicit-format branch detected `XML`, single-document `JSON*`, and `*WithProgress` in-band failures by reparsing `reply`, but `reply` is capped by `appendCappedSnapshot` at ~100 KB - a query emitting more than that before its terminal exception trailer was reported as successful, and `Run all` could continue past the failed statement. Keep a bounded (256 KB) tail of the raw stream separately from the snapshot cap (`appendBoundedTail`, re-aligned to a line boundary so the line-anchored probes cannot match a line cut open mid-value) and probe it instead (`parseInbandExceptionFromTail`): the `XML` and `*WithProgress` probes work on a suffix as-is, and when the trimmed tail no longer parses as a whole JSON document, fall back to matching the terminal trailer these formats write on failure - which a successful document can never end with.

…ion; gate named Tuple element names in JSON keys Two fixes for the framing formats: 1. A runtime failure after the real framed output format was already created (`HTTPHandler`'s streaming exception recovery: `framing->setException` + `current_output_format.finalize()`) still ran the format's prefix/suffix/ `finalizeImpl`, so `FORMAT JSON` emitted its empty document skeleton as a `data` packet before the terminal `exception` packet. `finalizeUnlocked` now takes the exception-only path whenever the framing has a terminal exception recorded (`IFramingFormat::hasException`): a query that fails before producing output delivers no `data` packet, and a mid-stream failure leaves the payload truncated without the format's suffix. 2. The JSON raw-bytes gate only validated top-level header names and `meta.type`, but `SerializationTuple::serializeTextJSON` synthesizes JSON object keys from named `Tuple` element names verbatim (with `output_format_json_named_tuples_as_objects = 1`, the default), and `DataTypeTuple::textCanContainOnlyValidUTF8` ignores the element names, so a non-UTF-8 element name skipped the whole-output validating buffer even with `output_format_json_validate_utf8 = 1`. New `JSONUtils::tupleElementNamesMayProduceRawBytesInJSON` walks the header types recursively (`IDataType::forEachChild`) and is wired into the checkers of `JSONEachRow` (and aliases), the non-`Strings` `JSONCompactEachRow` variants (including the plain one), `JSONColumns`, `JSONCompactColumns`, `JSONObjectEachRow`, `GeoJSON` (which forces named tuples in `properties` to object form), and `CustomSeparated` with the `JSON` escaping rule. The `Strings` variants write a `Tuple` value in its plain text form (no element names) and stay accepted; the full-document `JSON*` formats were already covered because their `meta.type` check validates `type->getName()`, which embeds the element names.

…b-ui

… failure snapshots A truncated framed stream (`EventStream` residual, or an NDJSON packet stream cut mid-line, in both the 200 OK and non-200 branches) rendered the explicit truncation error live and captured a synthesized `exception_carrier`, but the carrier was only used by `compactFailureSnapshot` for oversized (>100 KB) snapshots. A sub-cap failure was saved as `reply` verbatim, and the `event_stream` / `ndjson_packets` restore branches replay the saved stream text alone - so a reload or tab switch showed only the partial output and lost the error. Append the synthesized carrier to `reply` at all three truncation sites (as its own well-formed block/line, the same way the transport catch does), so the restored snapshot replays the same error the live run showed.

clickhouse-gh · 2026-07-24T00:25:27Z

LLVM Coverage Report

Metric	Baseline	Current	Δ
Lines	86.20%	86.20%	+0.00%
Functions	92.00%	92.00%	+0.00%
Branches	78.20%	78.30%	+0.10%

Changed lines: Changed C/C++ lines covered: 1295/1322 (97.96%) · Uncovered code

Full report · Diff report

…n restore The AI review found that failed single-query `event_stream` snapshots returned from the restore path before the table-failure finalization the live non-framed mid-stream failure applies (`renderTotals`, `applyColumnColors`, `transposeIfNeeded`, `_changeTableLayout`, and the no-rows clear), so a failed framed table reopened with a half-rendered table - or, header-only, an empty table above the error. The live framed failure path and the multi-query restore helper had the same gap. Extract the finalize-or-clear invariant into one shared `QueryResultElement.finalizeFailedTable(measureNow)` and route all four sites through it: the live framed `event_stream` failure and the live non-framed `stream_exception` branch in `postImpl` (deferring the measuring passes for a background tab), the single-query restore in `materializeResult`, and the multi-query restore in `renderSnapshotIntoElement`. The no-rows case hides the empty table with a targeted clear that - unlike `clear` - preserves the already-rendered exception and the streamed server logs, which a framed stream renders before the failure is finalized. Verified with a node harness driving the real extracted method (15 cases: finalize order, background deferral, empty-table hide preserving error/logs, call-site shapes); test 04548 checks the shared method and its call sites on the served page.

The Upgrade check fails with "New settings are not reflected in settings changes history": the entry for `framing_output_format` was recorded under 26.7, but the 26.7 release was cut without this branch, and master is now on 26.8 (#111450) - the upgrade from the released 26.7 therefore sees the new setting as unexplained. Same class of fix as #111671 for `join_runtime_filter_min_probe_rows`. https://s3.amazonaws.com/clickhouse-test-reports/json.html?PR=110210&sha=206f1045ea330aa4587214870bb3e8e3208f61dd&name_0=PR&name_1=Upgrade%20check%20%28amd_release%29 Related: #110210 Related: #110127

The merge of master auto-merged its new 26.8 `addSettingsChanges` bucket next to the one added for `framing_output_format` in the previous commit. `addSettingsChanges` throws a logical error (`Detected duplicate version`) on server startup for duplicate versions, so fold both into one bucket.

…b-ui

clickhouse-gh · 2026-07-24T04:59:19Z

+    /// The final progress is written after the logs and profile events above, so a successful
+    /// stream ends with it (see `setFinalProgress`). On a failure the `exception` packet below
+    /// stays terminal.
+    if (has_final_progress)


Emitting the deferred final progress packet unconditionally here breaks the framing contract on late failures. finishExecutedQuery runs the callback that calls writeFinalProgress before io.onFinish(), and the outer catch then reuses the same framing instance to serialize the exception. If io.onFinish() (for example, a query-log write) or the later output_format->finalize() path throws after has_final_progress is set, this finalize sends a success-style final progress packet and then the exception. The new docs/tests in this PR say a failed stream should end with the exception instead of a final-progress tail, so clients can use that packet as the success terminator. Please clear/suppress final_progress once setException is called, or guard this branch on exception_message.empty(), and add a regression that throws after flushQueryProgress / inside io.onFinish().

alexey-milovidov and others added 15 commits July 11, 2026 22:43

Merge branch 'master' into framing-formats

3b113db

clickhouse-gh Bot added the pr-not-for-changelog This PR should not be mentioned in the changelog label Jul 13, 2026

alexey-milovidov and others added 2 commits July 13, 2026 04:25

clickhouse-gh Bot reviewed Jul 13, 2026

View reviewed changes

Comment thread src/Interpreters/executeQuery.cpp

Comment thread src/Interpreters/executeQuery.cpp

Comment thread programs/server/play.html Outdated

alexey-milovidov and others added 4 commits July 13, 2026 08:13

Merge remote-tracking branch 'origin/framing-formats' into framing-we…

48c14bf

…b-ui

clickhouse-gh Bot reviewed Jul 13, 2026

View reviewed changes

Comment thread programs/server/play.html Outdated

Comment thread src/Interpreters/executeQuery.cpp Outdated

Comment thread src/Interpreters/executeQuery.cpp Outdated

clickhouse-gh Bot reviewed Jul 13, 2026

View reviewed changes

Comment thread src/Interpreters/executeQuery.cpp Outdated

Comment thread programs/server/play.html Outdated

mintlify Bot deployed to staging - docs July 13, 2026 11:14 View deployment

alexey-milovidov and others added 2 commits July 13, 2026 12:11

clickhouse-gh Bot reviewed Jul 22, 2026

View reviewed changes

Comment thread programs/server/play.html Outdated

Comment thread programs/server/play.html Outdated

alexey-milovidov and others added 2 commits July 22, 2026 23:09

Merge remote-tracking branch 'origin/framing-formats' into framing-we…

c7a2f6e

…b-ui

mintlify Bot deployed to staging - docs July 22, 2026 23:47 View deployment

clickhouse-gh Bot reviewed Jul 22, 2026

View reviewed changes

Comment thread programs/server/play.html

alexey-milovidov and others added 3 commits July 23, 2026 00:52

alexey-milovidov added 7 commits July 23, 2026 08:51

Fix style check: rename test 04629 (no "fail" in test names) and rewo…

eb15aa0

…rd an analyzer comment

Merge remote-tracking branch 'origin/framing-formats' into framing-we…

0286acb

…b-ui

Merge remote-tracking branch 'origin/framing-formats' into framing-we…

d694e06

…b-ui

Merge remote-tracking branch 'origin/master' into framing-web-ui

206f104

mintlify Bot deployed to staging - docs July 23, 2026 19:27 View deployment

clickhouse-gh Bot reviewed Jul 23, 2026

View reviewed changes

Comment thread programs/server/play.html

alexey-milovidov added 6 commits July 24, 2026 03:20

Merge remote-tracking branch 'origin/master' into framing-formats

7ac05f9

Merge remote-tracking branch 'origin/framing-formats' into framing-we…

75d2f11

…b-ui

Merge remote-tracking branch 'origin/master' into framing-web-ui

92361a1

mintlify Bot deployed to staging - docs July 24, 2026 04:51 View deployment

clickhouse-gh Bot reviewed Jul 24, 2026

View reviewed changes

mintlify Bot deployed to staging - docs July 24, 2026 05:13 View deployment

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Web UI prototype for framing formats#110210

Web UI prototype for framing formats#110210
alexey-milovidov wants to merge 137 commits into
masterfrom
framing-web-ui

alexey-milovidov commented Jul 13, 2026 •

edited

Loading

Uh oh!

mintlify Bot commented Jul 13, 2026 •

edited

Loading

Uh oh!

clickhouse-gh Bot commented Jul 13, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

alexey-milovidov commented Jul 23, 2026 •

edited

Loading

Uh oh!

alexey-milovidov commented Jul 23, 2026

Uh oh!

Uh oh!

clickhouse-gh Bot commented Jul 24, 2026

Uh oh!

clickhouse-gh Bot Jul 24, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

alexey-milovidov commented Jul 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changelog category (leave one):

Changelog entry (a user-readable short description of the changes that goes into CHANGELOG.md):

Uh oh!

mintlify Bot commented Jul 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

clickhouse-gh Bot commented Jul 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

AI Review

Summary

Findings

Final Verdict

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

alexey-milovidov commented Jul 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

alexey-milovidov commented Jul 23, 2026

Uh oh!

Uh oh!

clickhouse-gh Bot commented Jul 24, 2026

LLVM Coverage Report

Uh oh!

clickhouse-gh Bot Jul 24, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

alexey-milovidov commented Jul 13, 2026 •

edited

Loading

mintlify Bot commented Jul 13, 2026 •

edited

Loading

clickhouse-gh Bot commented Jul 13, 2026 •

edited

Loading

alexey-milovidov commented Jul 23, 2026 •

edited

Loading