fix(retrieval): run the reranker on CPU under CoreML - #33

Closes #41.

Problem

On an onnx-jina-coreml backend the reranker fails at inference:

Non-zero status code returned while running 11619352191107991357_CoreML_11619352191107991357_3 node.
Name:'CoreMLExecutionProvider_11619352191107991357_CoreML_11619352191107991357_3_3'
Status Message: Error executing model: Unable to compute the prediction using a neural network
model. It can be an invalid input data or broken/unsupported model (error code: -1).

vera doctor --probe surfaces it as a probe-tiny-inference failure, with probe-reranker-session still green. Reproduced deterministically on three consecutive runs.

The user-visible effect is worse than the probe suggests. Searches do not fail. They log

reranker unavailable, returning unreranked results

and return fusion order, so the CoreML backend silently loses cross-encoder ranking while the command still exits 0.

Root cause

build_session in local_reranker.rs registered CoreMLExecutionProvider whenever the backend was CoreML.

#25 established that no prebuilt reranker export runs on the CoreML GPU and moved every backend onto the quantized INT8 export. The premise recorded alongside that change is that the CoreML EP can execute none of those ops, so ORT partitions zero nodes and falls back to CPU on its own, and vera doctor's probe-reranker-coreml-cpu warning was added on that basis.

That premise does not hold here. The failing node name carries a CoreMLExecutionProvider_ prefix and a subgraph index, so ORT did assign a fused subgraph to CoreML. Partitioning and CoreML model compilation both succeed, which is why probe-reranker-session passes; only the prediction fails.

The existing CPU retry cannot catch this. It wraps session construction only, and construction succeeds, so inference runs with no fallback behind it.

This is not an ORT version drift: ORT_VERSION is 1.24.4 at v0.12.13, at #25's merge commit, and at v1.0.0.

Fix

Select the CPU provider for the reranker up front, through a small mapping function placed next to RERANKER_ONNX_FILE where the reranker-asset reasoning already lives. This makes the behaviour that probe-reranker-coreml-cpu and docs/troubleshooting.md already describe actually true.

Embedding still registers CoreML. Only the reranker moves to CPU, which is what the quantized INT8 export is already chosen for.

The mapping is a pure function, so the regression is unit testable without ONNX Runtime or model assets. The pre-existing test_local_reranker could not have caught this: it pins OnnxExecutionProvider::Cpu and early-returns when ORT is unavailable.

docs/troubleshooting.md is corrected to stop asserting that ORT falls back on its own.

Verification

Apple M4, macOS 26.6.1, ONNX Runtime 1.24.4, backend onnx-jina-coreml, index of this repository (375 files, 4790 chunks). Same index and query for both runs.

cargo fmt --check clean. cargo clippy -p vera-core --lib introduces no new warnings; the 5 reported are pre-existing (pip_package_for_ep, CUDA_RUNTIME_LIBRARY_PREFIXES, parse_cuda_major_from_runtime_library_entry, an unused import, an unused variable). New test and the local_models suite pass.

Note

Unrelated to this change and present on both binaries against the same index: vector search failed, falling back to BM25-only results with storage error: vector search failed. Not addressed here.

^{Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.}

Summary by CodeRabbit

• Bug Fixes

  • Improved reranker reliability on Apple Silicon by explicitly using CPU execution when CoreML is selected.
  • Added diagnostics for cases where CoreML accepts part of the workload but fails during inference.

• Documentation

  • Updated troubleshooting guidance to explain CoreML reranker behavior and the resulting diagnostic warning.