Bound HTTP requests in built-in poll callbacks - #1779
Open
robjarawan wants to merge 14 commits into
Open
robjarawan wants to merge 14 commits into
robjarawan wants to merge 14 commits into
Conversation
The paho callback thread appended to rx_msg without holding the mutex, racing with the main thread's buffer rotation. This could cause lost messages or IndexError when both threads accessed the same buffer. - Acquire rx_msg_mutex in __sub_on_message before appending - Use `with` statement for mutex in _rotateInputBuffers - Remove time.sleep(0.1) that held the lock for 100ms per rotation - Fix getNewMessage checking len(rx_msg) (dict, always 5) instead of len(rx_msg[iToApp]) (the actual buffer) — could IndexError on empty
Fixes #5. Subscriptions.read() (the JSON-on-disk path) converts pre-3.02 bindings that carry 'prefix'+'sub' into the post-3.02 shape with a single 'topic' key. Subscriptions([...]) built in memory -- the pattern used by sr_insects' moth_api_consumer.py, flow_api_consumer.py, and the flow_maint_test.sh harness -- did not. The raw prefix+sub dicts then hit `topic = b['topic']` in moth/amqp.py:423 and moth/mqtt.py:233 and raised KeyError: 'topic', which the connect retry loop caught and spun on until CI killed the job at 20 minutes. A second KeyError of the same class, 'bindings_to_remove', fired once the topic issue was patched -- moth/amqp.py:432 iterates subscription['bindings_to_remove'] directly, but in-memory subs never populate that key (it's only filled by finalize() when diffing). Changes to sarracenia/config/subscription.py: - Extract the conversion into two module-level helpers: _broker_proto(broker) -> 'amqp' | 'mqtt' normalize_subscription(s) -> mutates in place, idempotent - Subscriptions.__init__ normalizes each passed-in subscription so Subscriptions([...]) produces the same shape as Subscriptions.read(). - Subscriptions.add() normalizes new_subscription before merging so late additions of pre-3.02 bindings work too. - read() now delegates to normalize_subscription instead of repeating the conversion inline (single source of truth). - normalize_subscription also defaults bindings_to_remove = [] so the downstream moth iterators can't KeyError on in-memory subs. Tests: Adds tests/sarracenia/subscription_test.py with 14 cases covering: - AMQP dot-joining of prefix+sub into topic - MQTT slash-joining with $share/<queue_name>/ shared-subscription prefix - Idempotence (running normalize twice produces the same result) - Leaving a binding that already has 'topic' untouched - Stripping leftover sub/prefix when 'topic' is also present - Publisher-side subs without a 'queue' key - Scalar (non-list) prefix and sub values - Subscriptions([...]) normalizes on construction (the regression case) - Subscriptions() / Subscriptions(None) / Subscriptions([]) all empty - Subscriptions.add() normalizes - Subscriptions.add() merges a second pre-3.02 binding into an existing subscription, with topic computed for both - bindings_to_remove defaulted to [] by both normalize and __init__ Verification: - 14/14 new unit tests pass. - 28/28 sarracenia/config + credentials + subscription tests pass. - 15/15 sarracenia/moth tests pass (excluding amq1 which needs proton). - sr_insects/static_flow/moth_api_consumer.py runs cleanly against amqps://anonymous@hpfx.collab.science.gc.ca and prints "got 5 messages" with no errors in the log. - sr_insects/static_flow/flow_maint_test.sh EXIT=0, all 5 tests pass: test 1 success (queues baseline) test 2 success (moth_api_consumer.py consumes 5) test 3 success (flow_api_consumer.py consumes 5) test 4 success (post-cleanup queue count) test 5 success (erase test) Previously this script hung at test 3 for the full 20-min CI budget.
…tion Fix KeyError in Moth when Subscriptions built in memory (issue #5)
Brings the fork's development branch up to upstream as of 25e9335 (Merge pull request MetPX#1691 from MetPX/issue1690). Preserves fork-only work: - e5df257 fix: normalize Subscriptions built in memory, default bindings_to_remove (fork issue #5) - kept, no upstream conflict. Drops superseded fork-only work: - 8c7d7a7 Fix MQTT rx_msg thread-safety: lock callback append, remove sleep - resolved by taking upstream's mqtt.py entirely. The mutex-based fix has been superseded by my own queue.Queue redesign (b7acf4b "Simplify MQTT message reception with queue.Queue") that landed upstream via local development.
Restore the working directory after Flow work
andreleblanc11
self-requested a review
September 14, 2026 12:18
andreleblanc11
requested changes
Sep 14, 2026
andreleblanc11
left a comment
Member
There was a problem hiding this comment.
These are not the same changes as in robjarawan#173...
They include some? of the changes in #1776.. which has not been merged in the main line code yet and still has pending issues to be addressed. It also adds separate functionality changes in the subscription.py configuration class.
Please adjust your branch so that it adds the same functionality changes as in robjarawan#173.
If your PR adds more unit tests from your source repositories' source branch, that's likely not a problem. Added unit tests are welcome.
But adding functionality changes that are already addressed in the other PRs and that are not part of the same scope of your original changes can't be merged via this PR.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Built-in poll callbacks can wait indefinitely when an HTTP provider or network path stops responding. This was found while configuring an EUMETSAT poll and confirmed across the AirNow, Copernicus Marine, EUMETSAT, NASA CMR, NEXRAD, NOAA hydrometric, OData, and USGS callbacks.
Change
Pass the existing SR3
timeoutsetting to each directrequests.get()andurllib.request.urlopen()call in those callbacks. The change stays in the built-in polls and does not add a new option, retry policy, batching behavior, or shared abstraction.Closes #1778.
Validation
Nine regression tests cover all 14 changed request paths, including EUMETSAT browse/details, OData pagination, NASA catalogue/checksum, both Copernicus STAC requests, both USGS modes, and all NOAA hydrometric requests.
The tests fail against the base development commit
1c738c8aebecause the requests receive no timeout, and pass at439b13dc8.pytest -q tests/sarracenia/flowcb/poll tests/sarracenia/config_test.py tests/sarracenia/flow/poll_test.py: 25 passedpytest -q tests/sarracenia/flowcb --ignore=tests/sarracenia/flowcb/wistree_test.py: 132 passed, 11 skippedgit diff --check: passedpython3 -m compileall -q sarracenia/flowcb/poll tests/sarracenia/flowcb/poll: passedA complete local
pytest testscollection also requires the optionalGTStoWIS2andprotonmodules, which are not installed in this checkout.