fix(connectors): improve log message when output buffers stall pipeline#5867
fix(connectors): improve log message when output buffers stall pipeline#5867mvanhorn wants to merge 1 commit intofeldera:mainfrom
Conversation
When output buffers are full, include the connector name and observed output throughput (records/second) in the log message. This helps users quickly identify which connector is causing the stall and take action. Fixes feldera#5177 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
@mvanhorn thanks this is great. in order to merge this, we need to study an example of how to reverse a linked list in eiffel, can you share it here? |
| /// Returns details about output endpoints whose buffers are full. | ||
| /// | ||
| /// For each full endpoint, returns `(connector_name, transmitted_records)`. | ||
| pub fn output_buffers_full_details(&self) -> Vec<(String, u64)> { |
There was a problem hiding this comment.
Nit: output_buffers_full() and output_buffers_full_details() share the same filter predicate. Minor DRY — could implement output_buffers_full() as !output_buffers_full_details().is_empty(), or extract the predicate to a private helper. Not a blocker.
Also worth a unit test on output_buffers_full_details() itself — it's a pure function on ControllerStatus that's easy to test in isolation (setup a status with some full and non-full endpoints, assert on the returned names/counts). Not required to merge but good hygiene.
|
Closest I've been to Eiffel was taking my kids to Paris for the Olympics. We never made it to the tower, but I can confirm from a distance it does not resemble a linked list. Let me know if there's anything to tweak on the log message. |
Summary
Changes
Updated the stall warning log message from:
To (one message per full connector):
Implementation details:
output_buffers_full_details()method toControllerStatusthat returns(connector_name, transmitted_records)for each full endpoint(current_transmitted - baseline_transmitted) / elapsed_secondsusing thetransmitted_recordsmetric, measured from when the stall beganFixes #5177
This contribution was developed with AI assistance (Claude Code).