Skip to content

fix(backtest): honor explicit local source for engine routing and benchmark#552

Merged
warren618 merged 4 commits into
HKUDS:mainfrom
xor-xe:fix/550-local-source-engine-benchmark
Jul 17, 2026
Merged

fix(backtest): honor explicit local source for engine routing and benchmark#552
warren618 merged 4 commits into
HKUDS:mainfrom
xor-xe:fix/550-local-source-engine-benchmark

Conversation

@xor-xe

@xor-xe xor-xe commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Route data sources without a dedicated engine branch (e.g. local) by detected instrument market, so a local AAPL.US dataset gets GlobalEquityEngine US-equity rules instead of the CryptoEngine default.
  • Fetch explicit benchmarks through the configured source's loader first, keeping yfinance as the fallback, so source: local backtests stay offline and reproducible.

Why

Engine routing in _create_market_engine fell through to the crypto default for any source without its own branch, and resolve_benchmark always created a YfinanceLoader, ignoring the configured source. Closes #550.

Changes

  • agent/backtest/runner.py: market-based routing (US/HK equity → GlobalEquityEngine) in the source fallback branch.
  • agent/backtest/benchmark.py: resolve_benchmark/_fetch_benchmark accept the source's loader and try it before yfinance.
  • agent/backtest/engines/base.py: pass the run's loader through; source: auto keeps the yfinance default.
  • agent/tests/test_local_source_routing.py: routing + benchmark-forwarding regression tests.
  • CHANGELOG.md: entry under Unreleased → Fixed.

Test Plan

  • Existing tests pass (pytest --ignore=agent/tests/e2e_backtest --tb=short -q) — 5391 passed, 10 skipped
  • New tests added (agent/tests/test_local_source_routing.py)
  • Tested manually: verified _create_market_engine("local", ..., ["AAPL.US"]) returns GlobalEquityEngine and benchmark resolution uses the local loader without constructing a yfinance loader

Checklist

  • No changes to protected areas (src/agent/, src/session/, src/providers/) without prior discussion
  • No hardcoded values (API keys, file paths, magic numbers)
  • Code follows CONTRIBUTING.md guidelines
  • Documentation updated (if user-facing change)

…chmark

Sources without a dedicated routing branch (e.g. local) sent US/HK
equity datasets to the crypto engine, and explicit benchmarks were
always fetched via yfinance even when all required price data was
available locally.

Route unbranched sources by detected instrument market, and fetch the
explicit benchmark through the configured source's loader first,
keeping yfinance as the fallback. source: auto keeps the yfinance
default.

Fixes HKUDS#550

Signed-off-by: xor_xe <bedauri.it@gmail.com>
@warren618

Copy link
Copy Markdown
Collaborator

Thanks for fixing the engine-routing half of #550. One blocking issue remains: when an explicit source=local loader returns no benchmark data or raises, _fetch_benchmark() silently falls back to yfinance. That contradicts the PR’s offline/reproducible contract and introduces an unexpected network request.

Please make explicit source=local fail closed for benchmark loading—return no external benchmark rather than falling back to yfinance—and replace test_falls_back_to_yfinance_when_source_loader_has_no_data with a regression proving YfinanceLoader is never constructed for explicit local runs. source=auto may retain the existing yfinance fallback.

xor-xe added 2 commits July 15, 2026 19:07
…e-engine-benchmark

Signed-off-by: xor_xe <bedauri.it@gmail.com>

# Conflicts:
#	CHANGELOG.md
…lback

An explicit source=local run whose loader returned no benchmark data
(or raised) silently fell back to yfinance, breaking the offline
contract with an unexpected network request. Local now returns no
external benchmark instead; other sources keep the yfinance fallback.

Signed-off-by: xor_xe <bedauri.it@gmail.com>
@xor-xe

xor-xe commented Jul 15, 2026

Copy link
Copy Markdown
Contributor Author

Done, in later commit local now fails closed for benchmark loading: no yfinance fallback when the local loader yields no data or raises (or when no loader is passed at all), so the run returns no external benchmark instead of making a network request. The replacement regression test monkeypatches YfinanceLoader to assert it is never constructed for explicit local runs, parametrized over the empty-loader, raising-loader, and missing-loader cases. source: auto and other explicit sources keep the existing yfinance fallback. Also merged main to resolve the CHANGELOG conflict.

@warren618

Copy link
Copy Markdown
Collaborator

Holding this — the fix doesn't fully close the gap it's meant to close yet. Ran a focused test directly against this PR's code: when the existing (unrelated to this diff) empty-result fallback in fetch_data_map() silently swaps the loader for a network source (e.g. the yahoo fallback, which triggers whenever the primary local loader returns nothing for a requested symbol — an easy case to hit, not contrived), config['source'] never gets updated to reflect that. resolve_benchmark() then calls .fetch() on whatever loader it's handed before checking allow_fallback — so a source=local run can still make a real network call for the benchmark. That's exactly the behavior the CHANGELOG/docstring claim this PR prevents, and the same class of problem flagged in the original review thread.

The new regression tests only assert YfinanceLoader is never constructed, which doesn't cover this path since the leak goes through a different loader class entirely.

Could you add a check in resolve_benchmark/_fetch_benchmark that respects allow_fallback/source before calling .fetch() on the handed-in loader, not just before constructing a fresh YfinanceLoader? Happy to merge once that's closed — the engine-routing half of this fix is solid.

fetch_data_map()'s empty-result fallback can silently replace the local
loader with a network source (e.g. yahoo) while config['source'] still
reads local, so resolve_benchmark() fetched the benchmark through a
network loader before the allow_fallback check applied. Verify the
handed-in loader actually belongs to the local source before calling
fetch() on it, and fail closed otherwise.

Signed-off-by: xor_xe <bedauri.it@gmail.com>
@xor-xe

xor-xe commented Jul 16, 2026

Copy link
Copy Markdown
Contributor Author

Done in 488fc2f. resolve_benchmark() now verifies the handed-in loader actually belongs to the configured source (via the registry name attribute) before calling .fetch() on it so a loader silently swapped by fetch_data_map()'s empty-result fallback is dropped and the run fails closed, instead of the check only applying to a fresh YfinanceLoader.

The fail-closed regression is now parametrized over four cases for explicit source=local: empty local loader, raising local loader, a swapped-in network loader (whose fetch() asserts if ever invoked), and no loader at all, with YfinanceLoader construction also asserting. That covers both leak paths: fetching through a foreign loader and constructing a fresh network loader.

@warren618
warren618 merged commit c81c0c0 into HKUDS:main Jul 17, 2026
@warren618

Copy link
Copy Markdown
Collaborator

Thanks so much, @xor-xe — this is now merged into main as part of a reviewed batch integration, with your original commits and authorship preserved. Really appreciate the contribution!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] Explicit local data source is not honored for US equity engine routing and benchmark loading

2 participants