Skip to content

Bound HTTP requests in built-in poll callbacks - #1779

Open
robjarawan wants to merge 14 commits into
MetPX:developmentfrom
robjarawan:fix/issue-172-http-poll-timeouts
Open

robjarawan wants to merge 14 commits into
MetPX:developmentfrom
robjarawan:fix/issue-172-http-poll-timeouts

Conversation

@robjarawan

Copy link
Copy Markdown
Contributor

What

Built-in poll callbacks can wait indefinitely when an HTTP provider or network path stops responding. This was found while configuring an EUMETSAT poll and confirmed across the AirNow, Copernicus Marine, EUMETSAT, NASA CMR, NEXRAD, NOAA hydrometric, OData, and USGS callbacks.

Change

Pass the existing SR3 timeout setting to each direct requests.get() and urllib.request.urlopen() call in those callbacks. The change stays in the built-in polls and does not add a new option, retry policy, batching behavior, or shared abstraction.

Closes #1778.

Validation

Nine regression tests cover all 14 changed request paths, including EUMETSAT browse/details, OData pagination, NASA catalogue/checksum, both Copernicus STAC requests, both USGS modes, and all NOAA hydrometric requests.

The tests fail against the base development commit 1c738c8ae because the requests receive no timeout, and pass at 439b13dc8.

  • pytest -q tests/sarracenia/flowcb/poll tests/sarracenia/config_test.py tests/sarracenia/flow/poll_test.py: 25 passed
  • pytest -q tests/sarracenia/flowcb --ignore=tests/sarracenia/flowcb/wistree_test.py: 132 passed, 11 skipped
  • git diff --check: passed
  • python3 -m compileall -q sarracenia/flowcb/poll tests/sarracenia/flowcb/poll: passed

A complete local pytest tests collection also requires the optional GTStoWIS2 and proton modules, which are not installed in this checkout.

robjarawan and others added 13 commits March 29, 2026 22:12
The paho callback thread appended to rx_msg without holding the mutex,
racing with the main thread's buffer rotation. This could cause lost
messages or IndexError when both threads accessed the same buffer.

- Acquire rx_msg_mutex in __sub_on_message before appending
- Use `with` statement for mutex in _rotateInputBuffers
- Remove time.sleep(0.1) that held the lock for 100ms per rotation
- Fix getNewMessage checking len(rx_msg) (dict, always 5) instead of
  len(rx_msg[iToApp]) (the actual buffer) — could IndexError on empty
Fixes #5.

Subscriptions.read() (the JSON-on-disk path) converts pre-3.02 bindings
that carry 'prefix'+'sub' into the post-3.02 shape with a single 'topic'
key. Subscriptions([...]) built in memory -- the pattern used by
sr_insects' moth_api_consumer.py, flow_api_consumer.py, and the
flow_maint_test.sh harness -- did not. The raw prefix+sub dicts then
hit `topic = b['topic']` in moth/amqp.py:423 and moth/mqtt.py:233 and
raised KeyError: 'topic', which the connect retry loop caught and
spun on until CI killed the job at 20 minutes.

A second KeyError of the same class, 'bindings_to_remove', fired once
the topic issue was patched -- moth/amqp.py:432 iterates
subscription['bindings_to_remove'] directly, but in-memory subs never
populate that key (it's only filled by finalize() when diffing).

Changes to sarracenia/config/subscription.py:

- Extract the conversion into two module-level helpers:
    _broker_proto(broker)      -> 'amqp' | 'mqtt'
    normalize_subscription(s)  -> mutates in place, idempotent
- Subscriptions.__init__ normalizes each passed-in subscription so
  Subscriptions([...]) produces the same shape as Subscriptions.read().
- Subscriptions.add() normalizes new_subscription before merging so
  late additions of pre-3.02 bindings work too.
- read() now delegates to normalize_subscription instead of repeating
  the conversion inline (single source of truth).
- normalize_subscription also defaults bindings_to_remove = [] so the
  downstream moth iterators can't KeyError on in-memory subs.

Tests:

Adds tests/sarracenia/subscription_test.py with 14 cases covering:
- AMQP dot-joining of prefix+sub into topic
- MQTT slash-joining with $share/<queue_name>/ shared-subscription prefix
- Idempotence (running normalize twice produces the same result)
- Leaving a binding that already has 'topic' untouched
- Stripping leftover sub/prefix when 'topic' is also present
- Publisher-side subs without a 'queue' key
- Scalar (non-list) prefix and sub values
- Subscriptions([...]) normalizes on construction (the regression case)
- Subscriptions() / Subscriptions(None) / Subscriptions([]) all empty
- Subscriptions.add() normalizes
- Subscriptions.add() merges a second pre-3.02 binding into an
  existing subscription, with topic computed for both
- bindings_to_remove defaulted to [] by both normalize and __init__

Verification:

- 14/14 new unit tests pass.
- 28/28 sarracenia/config + credentials + subscription tests pass.
- 15/15 sarracenia/moth tests pass (excluding amq1 which needs proton).
- sr_insects/static_flow/moth_api_consumer.py runs cleanly against
  amqps://anonymous@hpfx.collab.science.gc.ca and prints
  "got 5 messages" with no errors in the log.
- sr_insects/static_flow/flow_maint_test.sh EXIT=0, all 5 tests pass:
    test 1 success (queues baseline)
    test 2 success (moth_api_consumer.py consumes 5)
    test 3 success (flow_api_consumer.py consumes 5)
    test 4 success (post-cleanup queue count)
    test 5 success (erase test)
  Previously this script hung at test 3 for the full 20-min CI budget.
…tion

Fix KeyError in Moth when Subscriptions built in memory (issue #5)
Brings the fork's development branch up to upstream as of 25e9335
(Merge pull request MetPX#1691 from MetPX/issue1690).

Preserves fork-only work:
- e5df257 fix: normalize Subscriptions built in memory, default
  bindings_to_remove (fork issue #5) - kept, no upstream conflict.

Drops superseded fork-only work:
- 8c7d7a7 Fix MQTT rx_msg thread-safety: lock callback append,
  remove sleep - resolved by taking upstream's mqtt.py entirely.
  The mutex-based fix has been superseded by my own queue.Queue
  redesign (b7acf4b "Simplify MQTT message reception with
  queue.Queue") that landed upstream via local development.
Restore the working directory after Flow work
@robjarawan robjarawan self-assigned this Sep 10, 2026

@andreleblanc11 andreleblanc11 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These are not the same changes as in robjarawan#173...

They include some? of the changes in #1776.. which has not been merged in the main line code yet and still has pending issues to be addressed. It also adds separate functionality changes in the subscription.py configuration class.

Please adjust your branch so that it adds the same functionality changes as in robjarawan#173.

If your PR adds more unit tests from your source repositories' source branch, that's likely not a problem. Added unit tests are welcome.

But adding functionality changes that are already addressed in the other PRs and that are not part of the same scope of your original changes can't be merged via this PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Built-in HTTP poll callbacks do not bound requests

2 participants