[adapters] Report recent connector errors in the API by ryzhyk · Pull Request #5775 · feldera/feldera

ryzhyk · 2026-03-06T23:54:17Z

Partially fixes #1108.

Today, connector errors are only reported in the log where they can be
difficult to find or get lost completely if the log is noisy.

This commit exposes connector errors via the REST API:

We store up to a fixed number (100) most recent errors of each kind
(transport/encoder/parser) and for each tag (we reuse the same tags that were
recently introduced for log throttling).
Old errors are evicted in FIFO order
The error list is returned via
/tables/table_name/connectors/connector_name/stats endpoint, but not via
global /stats to avoid bloating the output of that endpoint.

Describe Manual Test Plan

Checklist

Unit tests added/updated
Integration tests added/updated
Documentation updated
Changelog updated

Breaking Changes?

Mark if you think the answer is yes for any of these components:

OpenAPI / REST HTTP API / feldera-types / manager (What is a breaking change?)
Feldera SQL (Syntax, Semantics)
feldera-sqllib (incl. dependencies fxp, etc.) (What is a breaking change?)
Python SDK (What is a breaking change?)
fda (CLI arguments)
Adapters (including configuration)
Storage Format / Checkpoints
Others (specify)

Describe Incompatible Changes

crates/feldera-types/src/adapter_stats.rs

mihaibudiu · 2026-03-06T23:59:51Z

crates/adapters/src/controller/stats.rs

    }

-    pub fn parse_error(&self, endpoint_id: EndpointId) {
+    pub fn parse_error(


I would name these functions with "report_parse_error", to make it clear that they are not constructors.

crates/adapters/src/controller/stats.rs

mihaibudiu · 2026-03-07T00:01:56Z

crates/adapters/src/controller/stats.rs

+/// Stores up to MAX_CONNECTOR_ERRORS most recent errors for each tag.
+/// When the number of errors for a tag exceeds MAX_CONNECTOR_ERRORS, the oldest error is removed.
+#[derive(Debug)]
+pub struct ConnectorErrorList {


Should the constant MAX_CONNECTOR_ERRORS be part of this class?
Alternatively, it should be a constructor parameter if not.
Also, this class could have "Recent" in its name.

mihaibudiu · 2026-03-07T00:02:44Z

crates/adapters/src/controller/stats.rs

+    /// If `full` is false, generates a trimmed version of the endpoint status,
+    /// not including the list of errors. This status can be returned via the
+    /// `/stats` endpoint.
+    pub fn to_api_type(&self, full: bool) -> ExternalOutputEndpointStatus {


I would rename "full" to "include_errors"

We may add more optional fields in the future. I think full is adequate.

ryzhyk · 2026-03-07T00:04:58Z

TODOs:

tests
checkpointing error list
webconsole support

Not in scope: system table for errors.

mihaibudiu · 2026-03-07T00:07:36Z

System tables would tricky if the pipeline needs to essentially block. If it can't run any more steps it won't be able to write to any output connectors.

Karakatiza666 · 2026-03-07T10:31:51Z

crates/feldera-types/src/adapter_stats.rs

    /// The first fatal error that occurred at the endpoint.
    pub fatal_error: Option<String>,
+    #[serde(default, skip_serializing_if = "Option::is_none")]
+    /// Recent encoding errors on this endpoint.


Comments above decorators make the whole field more readable

Karakatiza666 · 2026-03-07T10:34:00Z

crates/feldera-types/src/adapter_stats.rs

+    /// Optional tag for the error.
+    ///
+    /// The tag is used to group errors by their type.
+    pub tag: Option<String>,


An optional tag is not pretty, this forces clients to come up with own strategy to treat tag=null records. Would it be meaningful to have an "Other" or "Generic" tag assigned by the pipeline-manager, making this field required in the API?

It's easy to substitute "other" if a tag is missing; it's harder to go in the other direction. In any case, I don't think we have much use for the tag, you may want to just ignore it in the webconsole

Karakatiza666 · 2026-03-07T10:35:49Z

crates/feldera-types/src/adapter_stats.rs

    /// The first fatal error that occurred at the endpoint.
    pub fatal_error: Option<String>,
+    #[serde(default)]
+    pub parse_errors: Option<Vec<ConnectorError>>,


Since you introduce error lists in Input and Output Status, do you deprecate metrics.num_*_errors in both?

No, because these error lists only contain recent errors, the error count should account for all errors that have ever occurred in this pipeline.

Karakatiza666 · 2026-03-07T10:37:19Z

crates/feldera-types/src/adapter_stats.rs

+    pub tag: Option<String>,
+
+    /// Error message.
+    pub error: String,


message is more fitting and consistent with other errors?

gz

lgtm maybe we should store 1000 since it should be cheap and use http pagination headers to retrieve them https://docs.github.com/en/rest/using-the-rest-api/using-pagination-in-the-rest-api?apiVersion=2022-11-28

(might be a future PR if too hard)

gz · 2026-03-07T20:16:53Z

crates/adapters/src/controller/stats.rs

-    fn parse_error(&self) {
-        self.metrics.num_parse_errors.fetch_add(1, Ordering::AcqRel);
+    fn parse_error(&self, tag: Option<&'static str>, error: &ParseError) {
+        let last_error = self.metrics.num_parse_errors.fetch_add(1, Ordering::AcqRel);


it's odd that you need both parse_errors and num_parse_errors

num errors accounts for all errors generated by the connector. The error list only lists recent errors. @mihaibudiu suggested renaming it to recent_parse_errors.

But note that we include error index to help track gaps in the list of reported errors.

gz · 2026-03-07T20:18:03Z

crates/adapters/src/controller/stats.rs

+/// When the number of errors for a tag exceeds MAX_CONNECTOR_ERRORS, the oldest error is removed.
+#[derive(Debug)]
+pub struct ConnectorErrorList {
+    errors: BTreeMap<Option<&'static str>, VecDeque<ConnectorError>>,


would it be better if this is ConcurrentHashMap<string, Mutex<VecDequeue>>

I don't think it matters. These are per-connector, so there's not gonna be a lot of contention. And in any case if the connector is producing so many errors of different types, lock contention is probably not your biggest problem.

mythical-fred · 2026-03-07T20:30:10Z

crates/feldera-types/src/adapter_stats.rs

+    /// Optional tag for the error.
+    ///
+    /// The tag is used to group errors by their type.
+    pub tag: Option<String>,


Mihai asked for skip_serializing_if here and ryzhyk agreed ("yes!"), but it wasn't added. tag: None will serialize as "tag": null in the JSON response. Should be:

#[serde(default, skip_serializing_if = "Option::is_none")] pub tag: Option<String>,

crates/adapters/src/controller/stats.rs

crates/adapters/src/controller.rs

mythical-fred

Good fixes — both the tag bug and the comment on semantics are addressed. One remaining item: still lacks . Approving, but worth a quick fix before merge.

mythical-fred · 2026-03-08T15:14:21Z

crates/feldera-types/src/adapter_stats.rs

+    ///
+    /// The tag is used to group errors by their type.
+    pub tag: Option<String>,
+


Missing #[serde(skip_serializing_if = "Option::is_none")]. Mihai flagged this earlier and you agreed — every tagless error will still serialize as "tag": null in the API response. One-liner fix.

mythical-fred · 2026-03-08T15:14:34Z

crates/feldera-types/src/adapter_stats.rs

+    ///
+    /// The tag is used to group errors by their type.
+    pub tag: Option<String>,
+


Missing #[serde(skip_serializing_if = "Option::is_none")] here. Mihai flagged this earlier and you agreed — every tagless error will still serialize as "tag": null in the API response. One-liner fix.

We had duplicate code converting connector status to the REST API format. Signed-off-by: Leonid Ryzhyk <ryzhyk@gmail.com>

mythical-fred

LGTM on the rebase — same fixes as before. The tag/skip_serializing_if comment above still stands.

Partially fixes #1108. Today, connector errors are only reported in the log where they can be difficult to find or get lost completely if the log is noisy. This commit exposes connector errors via the REST API: - We store up to a fixed number (100) most recent errors of each kind (transport/encoder/parser) and for each tag (we reuse the same tags that were recently introduced for log throttling). - Old errors are evicted in FIFO order - The error list is returned via /tables/table_name/connectors/connector_name/stats endpoint, but not via global /stats to avoid bloating the output of that endpoint. Signed-off-by: Leonid Ryzhyk <ryzhyk@gmail.com>

- Add Python bindings for querying input/output connector statuses. - Add a test that validates all connector fields and specifically tests input/out metric reporting. Signed-off-by: Leonid Ryzhyk <ryzhyk@gmail.com>

Per-table/view status endpoints now return strongly typed results of typr `InputConnectorStatus`/`OutpuConnectorStatus`. Signed-off-by: Leonid Ryzhyk <ryzhyk@gmail.com>

Signed-off-by: feldera-bot <feldera-bot@feldera.com>

ryzhyk requested a review from Karakatiza666 March 6, 2026 23:54

ryzhyk added the connectors Issues related to the adapters/connectors crate label Mar 6, 2026

mihaibudiu reviewed Mar 7, 2026

View reviewed changes

Karakatiza666 reviewed Mar 7, 2026

View reviewed changes

ryzhyk force-pushed the issue-1108 branch from 8ca85c1 to 1119d8c Compare March 7, 2026 19:34

ryzhyk marked this pull request as ready for review March 7, 2026 19:34

ryzhyk requested review from Karakatiza666 and gz March 7, 2026 20:10

ryzhyk force-pushed the issue-1108 branch from 29e6e29 to 3f3601e Compare March 7, 2026 20:19

gz approved these changes Mar 7, 2026

View reviewed changes

mythical-fred approved these changes Mar 7, 2026

View reviewed changes

mythical-fred reviewed Mar 7, 2026

View reviewed changes

crates/adapters/src/controller.rs Outdated Show resolved Hide resolved

ryzhyk force-pushed the issue-1108 branch 4 times, most recently from 435ed07 to 49c6acb Compare March 8, 2026 05:15

mythical-fred approved these changes Mar 8, 2026

View reviewed changes

mythical-fred reviewed Mar 8, 2026

View reviewed changes

[adapters] Use to_api_type consistently.

33941a7

We had duplicate code converting connector status to the REST API format. Signed-off-by: Leonid Ryzhyk <ryzhyk@gmail.com>

ryzhyk force-pushed the issue-1108 branch from 49c6acb to a163460 Compare March 8, 2026 16:56

mythical-fred approved these changes Mar 8, 2026

View reviewed changes

ryzhyk temporarily deployed to ci March 8, 2026 17:15 — with GitHub Actions Inactive

ryzhyk added 3 commits March 9, 2026 09:30

[py] Connector status API.

b868c5c

- Add Python bindings for querying input/output connector statuses. - Add a test that validates all connector fields and specifically tests input/out metric reporting. Signed-off-by: Leonid Ryzhyk <ryzhyk@gmail.com>

[api] Strongly typed connector status endpoints.

8cd6257

Per-table/view status endpoints now return strongly typed results of typr `InputConnectorStatus`/`OutpuConnectorStatus`. Signed-off-by: Leonid Ryzhyk <ryzhyk@gmail.com>

ryzhyk force-pushed the issue-1108 branch from a163460 to 8cd6257 Compare March 9, 2026 16:32

ryzhyk enabled auto-merge March 9, 2026 16:32

[ci] apply automatic fixes

e5ef467

Signed-off-by: feldera-bot <feldera-bot@feldera.com>

ryzhyk added this pull request to the merge queue Mar 9, 2026

Merged via the queue into main with commit c8a71c4 Mar 9, 2026
1 check passed

ryzhyk deleted the issue-1108 branch March 9, 2026 18:14

Conversation

ryzhyk commented Mar 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Describe Manual Test Plan

Checklist

Breaking Changes?

Describe Incompatible Changes

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ryzhyk commented Mar 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mihaibudiu commented Mar 7, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gz left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ryzhyk Mar 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

mythical-fred left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mythical-fred left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

ryzhyk commented Mar 6, 2026 •

edited

Loading

ryzhyk commented Mar 7, 2026 •

edited

Loading

gz left a comment •

edited

Loading

ryzhyk Mar 7, 2026 •

edited

Loading