Describe the bug
When using the --filter-providers option, I got this error:
ProviderReferenceValidationError: Test #1 ("Test Skill reads the config file first") references provider "gemini" which does not exist. Available providers: openrouter:deepseek/deepseek-r1, openrouter:free
To Reproduce
I have a test with following providers (that are matching provider configuration):
- description: Test Skill reads the config file first
providers:
- gemini
- openrouter:*
When running: npx promptfoo eval --filter-providers openrouter:*
i got this execption:
ProviderReferenceValidationError: Test #1 ("Test Skill reads the config file first") references provider "gemini" which does not exist. Available providers: openrouter:deepseek/deepseek-r1, openrouter:free
Expected behavior
The test should run on openrouter:* providers only, as indicated by the documentation:
CLI --filter-providers: If you use --filter-providers to filter providers at the CLI level, validation only sees the filtered providers. Tests referencing providers excluded by --filter-providers will fail validation
- Promptfoo version: 0.122.2
Describe the bug
When using the
--filter-providersoption, I got this error:ProviderReferenceValidationError: Test #1 ("Test Skill reads the config file first") references provider "gemini" which does not exist. Available providers: openrouter:deepseek/deepseek-r1, openrouter:freeTo Reproduce
I have a test with following providers (that are matching provider configuration):
When running:
npx promptfoo eval --filter-providers openrouter:*i got this execption:
ProviderReferenceValidationError: Test #1 ("Test Skill reads the config file first") references provider "gemini" which does not exist. Available providers: openrouter:deepseek/deepseek-r1, openrouter:freeExpected behavior
The test should run on
openrouter:*providers only, as indicated by the documentation: