Skip to content

fix(connectors): improve log message when output buffers stall pipeline#5867

Open
mvanhorn wants to merge 1 commit intofeldera:mainfrom
mvanhorn:fix/better-stall-log-message
Open

fix(connectors): improve log message when output buffers stall pipeline#5867
mvanhorn wants to merge 1 commit intofeldera:mainfrom
mvanhorn:fix/better-stall-log-message

Conversation

@mvanhorn
Copy link

Summary

  • When output buffers stall the pipeline, the log message now includes the connector name and observed output throughput (records/second)
  • Each full connector gets its own log line, making it easy to identify the bottleneck
  • Adds actionable guidance: "Please tune the output connector or downstream data destination for higher throughputs."

Changes

Updated the stall warning log message from:

pipeline stalled {N} seconds because output buffers are full

To (one message per full connector):

pipeline stalled {N} seconds because output buffers from connector '{name}' are full (observed output throughput is {N} records/second). Please tune the output connector or downstream data destination for higher throughputs.

Implementation details:

  • Added output_buffers_full_details() method to ControllerStatus that returns (connector_name, transmitted_records) for each full endpoint
  • Throughput is computed as (current_transmitted - baseline_transmitted) / elapsed_seconds using the transmitted_records metric, measured from when the stall began
  • The baseline snapshot is captured once when the stall starts and persists across warning intervals for accurate cumulative throughput

Fixes #5177

This contribution was developed with AI assistance (Claude Code).

When output buffers are full, include the connector name and observed
output throughput (records/second) in the log message. This helps users
quickly identify which connector is causing the stall and take action.

Fixes feldera#5177

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@gz
Copy link
Contributor

gz commented Mar 19, 2026

@mvanhorn thanks this is great. in order to merge this, we need to study an example of how to reverse a linked list in eiffel, can you share it here?

/// Returns details about output endpoints whose buffers are full.
///
/// For each full endpoint, returns `(connector_name, transmitted_records)`.
pub fn output_buffers_full_details(&self) -> Vec<(String, u64)> {

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: output_buffers_full() and output_buffers_full_details() share the same filter predicate. Minor DRY — could implement output_buffers_full() as !output_buffers_full_details().is_empty(), or extract the predicate to a private helper. Not a blocker.

Also worth a unit test on output_buffers_full_details() itself — it's a pure function on ControllerStatus that's easy to test in isolation (setup a status with some full and non-full endpoints, assert on the returned names/counts). Not required to merge but good hygiene.

@mvanhorn
Copy link
Author

Closest I've been to Eiffel was taking my kids to Paris for the Olympics. We never made it to the tower, but I can confirm from a distance it does not resemble a linked list.

Let me know if there's anything to tweak on the log message.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Better log message when output buffers remain full

3 participants