fix(backtest): honor explicit local source for engine routing and benchmark#552
Conversation
…chmark Sources without a dedicated routing branch (e.g. local) sent US/HK equity datasets to the crypto engine, and explicit benchmarks were always fetched via yfinance even when all required price data was available locally. Route unbranched sources by detected instrument market, and fetch the explicit benchmark through the configured source's loader first, keeping yfinance as the fallback. source: auto keeps the yfinance default. Fixes HKUDS#550 Signed-off-by: xor_xe <bedauri.it@gmail.com>
|
Thanks for fixing the engine-routing half of #550. One blocking issue remains: when an explicit Please make explicit |
…e-engine-benchmark Signed-off-by: xor_xe <bedauri.it@gmail.com> # Conflicts: # CHANGELOG.md
…lback An explicit source=local run whose loader returned no benchmark data (or raised) silently fell back to yfinance, breaking the offline contract with an unexpected network request. Local now returns no external benchmark instead; other sources keep the yfinance fallback. Signed-off-by: xor_xe <bedauri.it@gmail.com>
|
Done, in later commit local now fails closed for benchmark loading: no yfinance fallback when the local loader yields no data or raises (or when no loader is passed at all), so the run returns no external benchmark instead of making a network request. The replacement regression test monkeypatches YfinanceLoader to assert it is never constructed for explicit local runs, parametrized over the empty-loader, raising-loader, and missing-loader cases. source: auto and other explicit sources keep the existing yfinance fallback. Also merged main to resolve the CHANGELOG conflict. |
|
Holding this — the fix doesn't fully close the gap it's meant to close yet. Ran a focused test directly against this PR's code: when the existing (unrelated to this diff) empty-result fallback in The new regression tests only assert Could you add a check in |
fetch_data_map()'s empty-result fallback can silently replace the local loader with a network source (e.g. yahoo) while config['source'] still reads local, so resolve_benchmark() fetched the benchmark through a network loader before the allow_fallback check applied. Verify the handed-in loader actually belongs to the local source before calling fetch() on it, and fail closed otherwise. Signed-off-by: xor_xe <bedauri.it@gmail.com>
|
Done in 488fc2f. resolve_benchmark() now verifies the handed-in loader actually belongs to the configured source (via the registry name attribute) before calling .fetch() on it so a loader silently swapped by fetch_data_map()'s empty-result fallback is dropped and the run fails closed, instead of the check only applying to a fresh YfinanceLoader. The fail-closed regression is now parametrized over four cases for explicit source=local: empty local loader, raising local loader, a swapped-in network loader (whose fetch() asserts if ever invoked), and no loader at all, with YfinanceLoader construction also asserting. That covers both leak paths: fetching through a foreign loader and constructing a fresh network loader. |
|
Thanks so much, @xor-xe — this is now merged into |
Summary
local) by detected instrument market, so a localAAPL.USdataset getsGlobalEquityEngineUS-equity rules instead of theCryptoEnginedefault.source: localbacktests stay offline and reproducible.Why
Engine routing in
_create_market_enginefell through to the crypto default for any source without its own branch, andresolve_benchmarkalways created aYfinanceLoader, ignoring the configured source. Closes #550.Changes
agent/backtest/runner.py: market-based routing (US/HK equity →GlobalEquityEngine) in the source fallback branch.agent/backtest/benchmark.py:resolve_benchmark/_fetch_benchmarkaccept the source's loader and try it before yfinance.agent/backtest/engines/base.py: pass the run's loader through;source: autokeeps the yfinance default.agent/tests/test_local_source_routing.py: routing + benchmark-forwarding regression tests.CHANGELOG.md: entry under Unreleased → Fixed.Test Plan
pytest --ignore=agent/tests/e2e_backtest --tb=short -q) — 5391 passed, 10 skippedagent/tests/test_local_source_routing.py)_create_market_engine("local", ..., ["AAPL.US"])returnsGlobalEquityEngineand benchmark resolution uses the local loader without constructing a yfinance loaderChecklist
src/agent/,src/session/,src/providers/) without prior discussion