Nightly evals: summary job reads green when no eval reports exist #29598

What happened?

In scripts/aggregate_evals.js (lines 242-249 on main), when findReports returns nothing, the script logs "No reports found." and calls process.exit(0). The comment just above it says "practically if current has no reports, something is wrong."

The Run Evals step in .github/workflows/evals-nightly.yml (lines 67-68) is continue-on-error: true. So if every matrix leg fails inside that step before writing a report (an expired API key, say) and the other steps succeed, the nightly workflow finishes green, and the summary just says "No reports found."

Reproduced locally: node scripts/aggregate_evals.js <empty dir> prints "No reports found." and exits 0.

What did you expect to happen?

A nightly run that produced no eval reports should be visible as a problem, either as a failed job or at least as an error annotation, rather than reading green.

The only in-repo caller is the nightly workflow (evals-nightly.yml:121). I haven't checked whether a filtered workflow_dispatch run (suite_name / test_name_pattern) can end with no report file. If it can, a hard failure would be wrong for those runs. Some options, for you to choose from:

• emit a ::error:: or ::warning:: annotation and keep exit 0;
• exit non-zero only on the scheduled trigger (github.event_name == 'schedule');
• fail when zero reports exist across the whole matrix.

Client information

N/A. This is about the repository's CI script, not the CLI client.

Anything else we need to know?

Found while reading the repo's CI. Happy to help with a fix if one of these directions suits you.