Skip to content

--filter-providers makes an exception #10932

Description

@fletort

Describe the bug
When using the --filter-providers option, I got this error:
ProviderReferenceValidationError: Test #1 ("Test Skill reads the config file first") references provider "gemini" which does not exist. Available providers: openrouter:deepseek/deepseek-r1, openrouter:free

To Reproduce

I have a test with following providers (that are matching provider configuration):

  - description: Test Skill reads the config file first
     providers:
       - gemini
       - openrouter:*

When running: npx promptfoo eval --filter-providers openrouter:*
i got this execption:
ProviderReferenceValidationError: Test #1 ("Test Skill reads the config file first") references provider "gemini" which does not exist. Available providers: openrouter:deepseek/deepseek-r1, openrouter:free

Expected behavior
The test should run on openrouter:* providers only, as indicated by the documentation:

CLI --filter-providers: If you use --filter-providers to filter providers at the CLI level, validation only sees the filtered providers. Tests referencing providers excluded by --filter-providers will fail validation

  • Promptfoo version: 0.122.2

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions