diff --git a/docs/ai-coder/ai-bridge.md b/docs/ai-coder/ai-bridge.md deleted file mode 100644 index a993cee71319c..0000000000000 --- a/docs/ai-coder/ai-bridge.md +++ /dev/null @@ -1,300 +0,0 @@ -# AI Bridge - -![AI bridge diagram](../images/aibridge/aibridge_diagram.png) - -Bridge is a smart proxy for AI. It acts as a man-in-the-middle between your users' coding agents / IDEs -and providers like OpenAI and Anthropic. By intercepting all the AI traffic between these clients and -the upstream APIs, Bridge can record user prompts, token usage, and tool invocations. - -Bridge solves 3 key problems: - -1. **Centralized authn/z management**: no more issuing & managing API tokens for OpenAI/Anthropic usage. - Users use their Coder session or API tokens to authenticate with `coderd` (Coder control plane), and - `coderd` securely communicates with the upstream APIs on their behalf. Use a single key for all users. -2. **Auditing and attribution**: all interactions with AI services, whether autonomous or human-initiated, - will be audited and attributed back to a user. -3. **Centralized MCP administration**: define a set of approved MCP servers and tools which your users may - use, and prevent users from using their own. - -## When to use AI Bridge - -As the library of LLMs and their associated tools grow, administrators are pressured to provide auditing, measure adoption, provide tools through MCP, and track token spend. Disparate SAAS platforms provide _some_ of these for _some_ tools, but there is no centralized, secure solution for these challenges. - -If you are an administrator or devops leader looking to: - -- Measure AI tooling adoption across teams or projects -- Provide an LLM audit trail to security administrators -- Manage token spend in a central dashboard -- Investigate opportunities for AI automation -- Uncover the high-leverage use cases from experienced engineers - -We advise trying Bridge as self-hosted proxy to monitor LLM usage agnostically across AI powered IDEs like Cursor and headless agents like Claude Code. - -## Setup - -Bridge runs inside the Coder control plane, requiring no separate compute to deploy or scale. Once enabled, `coderd` hosts the bridge in-memory and brokers traffic to your configured AI providers on behalf of authenticated users. - -**Required**: - -1. A **premium** licensed Coder deployment -1. Feature must be [enabled](#activation) using the server flag -1. One or more [provider](#providers) API keys must be configured - -### Activation - -You will need to enable AI Bridge explicitly: - -```sh -CODER_AIBRIDGE_ENABLED=true coder server -# or -coder server --aibridge-enabled=true -``` - -### Providers - -Bridge currently supports OpenAI and Anthropic APIs. - -**API Key**: - -The single key used to authenticate all requests from Bridge to OpenAI/Anthropic APIs. - -- `CODER_AIBRIDGE_OPENAI_KEY` or `--aibridge-openai-key` -- `CODER_AIBRIDGE_ANTHROPIC_KEY` or `--aibridge-anthropic-key` - -**Base URL**: - -The API to which Bridge will relay requests. - -- `CODER_AIBRIDGE_OPENAI_BASE_URL` or `--aibridge-openai-base-url`, defaults to `https://api.openai.com/v1/` -- `CODER_AIBRIDGE_ANTHROPIC_BASE_URL` or `--aibridge-anthropic-base-url`, defaults to `https://api.anthropic.com/` - -Bridge is compatible with _[Google Vertex AI](https://cloud.google.com/vertex-ai?hl=en)_, _[AWS Bedrock](https://aws.amazon.com/bedrock/)_, and other LLM brokers. You may specify the base URL(s) above to the appropriate API endpoint for your provider. - ---- - -> [!NOTE] -> See [Supported APIs](#supported-apis) section below for a comprehensive list. - -## Client Configuration - -Once AI Bridge is enabled on the server, your users need to configure their AI coding tools to use it. This section explains how users should configure their clients to connect to AI Bridge. - -### Setting Base URLs - -The exact configuration method varies by client — some use environment variables, others use configuration files or UI settings: - -- **OpenAI-compatible clients**: Set the base URL (commonly via the `OPENAI_BASE_URL` environment variable) to `https://coder.example.com/api/v2/aibridge/openai/v1` -- **Anthropic-compatible clients**: Set the base URL (commonly via the `ANTHROPIC_BASE_URL` environment variable) to `https://coder.example.com/api/v2/aibridge/anthropic` - -Replace `coder.example.com` with your actual Coder deployment URL. - -### Authentication - -Instead of distributing provider-specific API keys (OpenAI/Anthropic keys) to users, they authenticate to AI Bridge using their **Coder session token** or **API key**: - -- **OpenAI clients**: Users set `OPENAI_API_KEY` to their Coder session token or API key -- **Anthropic clients**: Users set `ANTHROPIC_API_KEY` to their Coder session token or API key - -Users can generate a Coder API key using: - -```sh -coder tokens create -``` - -Template admins can pre-configure authentication in templates using [`data.coder_workspace_owner.me.session_token`](https://registry.terraform.io/providers/coder/coder/latest/docs/data-sources/workspace_owner#session_token-1) to automatically configure the workspace owner's credentials. - -#### Compatibility Notes - -Most AI coding assistants that support custom base URLs can work with AI Bridge. However, client-specific configuration requirements vary: - -- Some clients require specific URL formats (e.g. try removing the `/v1` suffix) -- Some clients may proxy requests through their own servers, limiting compatibility (e.g. Cursor) -- Some clients may not support custom base URLs at all (e.g. Copilot CLI, Sourcegraph Amp) - -Consult your specific AI client's documentation for details on configuring custom API endpoints. - -## Collected Data - -Bridge collects: - -- The last `user` prompt of each request -- All token usage (associated with each prompt) -- Every tool invocation - -All of these records are associated to an "interception" record, which maps 1:1 with requests received from clients but may involve several interactions with upstream providers. Interceptions are associated with a Coder identity, allowing you to map consumption and cost with teams or individuals in your organization: - -![User Prompt logging](../images/aibridge/grafana_user_prompts_logging.png) - -These logs can be used to determine usage patterns, track costs, and evaluate tooling adoption. - -This data is currently accessible through the API and CLI, which we advise administrators export to their observability platform of choice. We've configured a Grafana dashboard to display Claude Code usage internally which can be imported as a starting point for your tooling adoption metrics. - -![User Leaderboard](../images/aibridge/grafana_user_leaderboard.png) - -We provide an example Grafana dashboard that you can import as a starting point for your tooling adoption metrics. See [here](https://github.com/coder/coder/blob/main/examples/monitoring/dashboards/grafana/aibridge/README.md). - -## Implementation Details - -`coderd` runs an in-memory instance of `aibridged`, whose logic is mostly contained in https://github.com/coder/aibridge. In future releases we will support running external instances for higher throughput and complete memory isolation from `coderd`. - -
-See a diagram of how Bridge interception works - -```mermaid - -sequenceDiagram - actor User - participant Client - participant Bridge - - User->>Client: Issues prompt - activate Client - - Note over User, Client: Coder session key used
as AI token - Client-->>Bridge: Sends request - - activate Bridge - Note over Client, Bridge: Coder session key
passed along - - Note over Bridge: Authenticate - Note over Bridge: Parse request - - alt Rejected - Bridge-->>Client: Send response - Client->>User: Display response - end - - Note over Bridge: If first request, establish
connection(s) with MCP server(s)
and list tools - - Note over Bridge: Inject MCP tools - - Bridge-->>AIProvider: Send modified request - - activate AIProvider - - AIProvider-->>Bridge: Send response - - Note over Client: Client is unaware of injected
tools and invocations,
just receives one long response - - alt Has injected tool calls - loop - Note over Bridge: Invoke injected tool - Bridge-->>AIProvider: Send tool result - AIProvider-->>Bridge: Send response - end - end - - deactivate AIProvider - - Bridge-->>Client: Relay response - deactivate Bridge - - Client->>User: Display response - deactivate Client -``` - -
- -## MCP - -[Model Context Protocol (MCP)](https://modelcontextprotocol.io/docs/getting-started/intro) is a mechanism for connecting AI applications to external systems. - -Bridge can connect to MCP servers and inject tools automatically, enabling you to centrally manage the list of tools you wish to grant your users. - -> [!NOTE] -> Only MCP servers which support OAuth2 Authorization are supported currently. In future releases we will support [optional authorization](https://modelcontextprotocol.io/specification/2025-06-18/basic/authorization#protocol-requirements). -> -> [_Streamable HTTP_](https://modelcontextprotocol.io/specification/2025-06-18/basic/transports#streamable-http) is the only supported transport currently. In future releases we will support the (now deprecated) [_Server-Sent Events_](https://modelcontextprotocol.io/specification/2025-06-18/basic/transports#backwards-compatibility) transport. - -Bridge makes use of [External Auth](../admin/external-auth/index.md) applications, as they define OAuth2 connections to upstream services. If your External Auth application hosts a remote MCP server, you can configure Bridge to connect to it, retrieve its tools and inject them into requests automatically - all while using each individual user's access token. - -For example, GitHub has a [remote MCP server](https://github.com/github/github-mcp-server?tab=readme-ov-file#remote-github-mcp-server) and we can use it as follows. - -```bash -CODER_EXTERNAL_AUTH_0_TYPE=github -CODER_EXTERNAL_AUTH_0_CLIENT_ID=... -CODER_EXTERNAL_AUTH_0_CLIENT_SECRET=... -# Tell Bridge where it can find this service's remote MCP server. -CODER_EXTERNAL_AUTH_0_MCP_URL=https://api.githubcopilot.com/mcp/ -``` - -See the diagram in [Implementation Details](#implementation-details) for more information. - -You can also control which tools are injected by using an allow and/or a deny regular expression on the tool names: - -```bash -CODER_EXTERNAL_AUTH_0_MCP_TOOL_ALLOW_REGEX=(.+_gist.*) -CODER_EXTERNAL_AUTH_0_MCP_TOOL_DENY_REGEX=(create_gist) -``` - -In the above example, all tools containing `_gist` in their name will be allowed, but `create_gist` is denied. - -The logic works as follows: - -- If neither the allow/deny patterns are defined, all tools will be injected. -- The deny pattern takes precedence. -- If only a deny pattern is defined, all tools are injected except those explicitly denied. - -In the above example, if you prompted your AI model with "list your available github tools by name", it would reply something like: - -> Certainly! Here are the GitHub-related tools that I have available: -> -> 1. `bmcp_github_update_gist` -> 2. `bmcp_github_list_gists` - -Bridge marks automatically injected tools with a prefix `bmcp_` ("bridged MCP"). It also namespaces all tool names by the ID of their associated External Auth application (in this case `github`). - -## Tool Injection - -If a model decides to invoke a tool and it has a `bmcp_` suffix and Bridge has a connection with the related MCP server, it will invoke the tool. The tool result will be passed back to the upstream AI provider, and this will loop until the model has all of its required data. These inner loops are not relayed back to the client; all it seems is the result of this loop. See [Implementation Details](#implementation-details). - -In contrast, tools which are defined by the client (i.e. the [`Bash` tool](https://docs.claude.com/en/docs/claude-code/settings#tools-available-to-claude) defined by _Claude Code_) cannot be invoked by Bridge, and the tool call from the model will be relayed to the client, after which it will invoke the tool. - -If you have the `oauth2` and `mcp-server-http` experiments enabled, Coder's own [internal MCP tools](mcp-server.md) will be injected automatically. - -### Troubleshooting - -- **Too many tools**: should you receive an error like `Invalid 'tools': array too long. Expected an array with maximum length 128, but got an array with length 132 instead`, you can reduce the number by filtering out tools using the allow/deny patterns documented in the [MCP](#mcp) section. - -- **Coder MCP tools not being injected**: in order for Coder MCP tools to be injected, the internal MCP server needs to be active. Follow the instructions in the [MCP Server](mcp-server.md) page to enable it. - -- **External Auth tools not being injected**: this is generally due to the requesting user not being authenticated against the External Auth app; when this is the case, no attempt is made to connect to the MCP server. - -## Known Issues / Limitations - -- Codex CLI currently does not work with Bridge due to a JSON marshaling issue: https://github.com/coder/aibridge/issues/19 -- Claude Code web searches do not report correctly: https://github.com/coder/aibridge/issues/11 - -## Supported APIs - -API support is broken down into two categories: - -- **Intercepted**: requests are intercepted, audited, and augmented - full Bridge functionality -- **Passthrough**: requests are proxied directly to the upstream, no auditing or augmentation takes place - -Where relevant, both streaming and non-streaming requests are supported. - -### OpenAI - -**Intercepted**: - -- [`/v1/chat/completions`](https://platform.openai.com/docs/api-reference/chat/create) - -**Passthrough**: - -- [`/v1/models(/*)`](https://platform.openai.com/docs/api-reference/models/list) -- [`/v1/responses`](https://platform.openai.com/docs/api-reference/responses/create) _(Interception support coming in **Beta**)_ - -### Anthropic - -**Intercepted**: - -- [`/v1/messages`](https://docs.claude.com/en/api/messages) - -**Passthrough**: - -- [`/v1/models(/*)`](https://docs.claude.com/en/api/models-list) - -## Troubleshooting - -To report a bug, file a feature request, or view a list of known issues, please visit our [GitHub repository for Bridge](https://github.com/coder/aibridge). If you encounter issues with Bridge during early access, please reach out to us via [Discord](https://discord.gg/coder). diff --git a/docs/ai-coder/ai-bridge/client-config.md b/docs/ai-coder/ai-bridge/client-config.md new file mode 100644 index 0000000000000..1fc348c935135 --- /dev/null +++ b/docs/ai-coder/ai-bridge/client-config.md @@ -0,0 +1,110 @@ +# Client Configuration + +Once AI Bridge is setup on your deployment, the AI coding tools used by your users will need to be configured to route requests via AI Bridge. + +## Base URLs + +Most AI coding tools allow the "base URL" to be customized. In other words, when a request is made to OpenAI's API from your coding tool, the API endpoint such as [/v1/chat/completions](https://platform.openai.com/docs/api-reference/chat) will be appended to the configured base. Therefore, instead of the default base URL of "https://api.openai.com/v1", you'll need to set it to "https://coder.example.com/api/v2/aibridge/openai/v1". + +The exact configuration method varies by client — some use environment variables, others use configuration files or UI settings: + +- **OpenAI-compatible clients**: Set the base URL (commonly via the `OPENAI_BASE_URL` environment variable) to `https://coder.example.com/api/v2/aibridge/openai/v1` +- **Anthropic-compatible clients**: Set the base URL (commonly via the `ANTHROPIC_BASE_URL` environment variable) to `https://coder.example.com/api/v2/aibridge/anthropic` + +Replace `coder.example.com` with your actual Coder deployment URL. + +## Authentication + +Instead of distributing provider-specific API keys (OpenAI/Anthropic keys) to users, they authenticate to AI Bridge using their **Coder session token** or **API key**: + +- **OpenAI clients**: Users set `OPENAI_API_KEY` to their Coder session token or API key +- **Anthropic clients**: Users set `ANTHROPIC_API_KEY` to their Coder session token or API key + +Again, the exact environment variable or setting naming may differ from tool to tool; consult your tool's documentation. + +## Configuring In-Workspace Tools + +Template admins can pre-configure workspaces to route all AI tool requests through AI Bridge, providing a seamless and secure experience for users. This can be done for both Coder Tasks and other AI tools running in the workspace. + +### Using Coder Tasks + +Agents like Claude Code can be configured to route through AI Bridge in any template by pre-configuring the agent with the session token. [Coder Tasks](../tasks.md) is particularly useful for this pattern, providing a framework for agents to complete background development operations autonomously. To route agents through AI Bridge in a Coder Tasks template, pre-configure it to install Claude Code and configure it with the session token: + +```hcl +data "coder_workspace_owner" "me" {} + +data "coder_workspace" "me" {} + +resource "coder_agent" "dev" { + arch = "amd64" + os = "linux" + dir = local.repo_dir + env = { + ANTHROPIC_BASE_URL : "${data.coder_workspace.me.url}/api/v2/aibridge/anthropic", + ANTHROPIC_AUTH_TOKEN : data.coder_workspace_owner.me.session_token + } + ... # other agent configuration +} + +# See https://registry.coder.com/modules/coder/claude-code for more information +module "claude-code" { + count = local.has_ai_prompt ? data.coder_workspace.me.start_count : 0 + source = "dev.registry.coder.com/coder/claude-code/coder" + version = ">= 3.4.0" + agent_id = coder_agent.dev.id + workdir = "/home/coder/project" + claude_api_key = data.coder_workspace_owner.me.session_token # Use the Coder session token to authenticate with AI Bridge + ai_prompt = data.coder_parameter.ai_prompt.value + ... # other claude-code configuration +} +``` + +### Other IDEs and Tools + +AI coding tools running inside a Coder workspace, such as IDE extensions, can be configured to use AI Bridge. + +While users can manually configure these tools with a long-lived API key, template admins can provide a more seamless experience by pre-configuring them. Similar to the Coder Tasks example, admins can use Terraform data sources like `data.coder_workspace_owner.me.session_token` to automatically inject the user's session token and the AI Bridge base URL into the workspace environment. + +This is the fastest way to bring existing agents like Roo Code, Cursor, or Claude Code into compliance without adopting Coder Tasks. + +## External and Desktop Clients + +You can also configure AI tools running outside of a Coder workspace, such as local IDE extensions or desktop applications, to connect to AI Bridge. + +The configuration is the same: point the tool to the AI Bridge [base URL](#base-urls) and use a Coder API key for authentication. + +Users can generate a long-lived API key from the Coder UI or CLI. Follow the instructions at [Sessions and API tokens](../../admin/users/sessions-tokens.md#generate-a-long-lived-api-token-on-behalf-of-yourself) to create one. + +## Compatibility + +The table below shows tested AI clients and their compatibility with AI Bridge. Click each client name for vendor-specific configuration instructions. Report issues or share compatibility updates in the [aibridge](https://github.com/coder/aibridge) issue tracker. + +| Client | OpenAI support | Anthropic support | Notes | +|-------------------------------------------------------------------------------------------------------------------------------------------|----------------|-------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| [Claude Code](https://docs.claude.com/en/docs/claude-code/settings#environment-variables) | N/A | ✅ | Works out of the box and can be preconfigured in templates. | +| Claude Code (VS Code) | N/A | ✅ | May require signing in once; afterwards respects workspace environment variables. | +| [Cursor](https://cursor.com/docs/settings/api-keys) | ⚠️ | ❌ | Only non-reasoning models like `gpt-4.1` are available when using a custom endpoint. Requests still transit Cursor's cloud. There is no central admin setting to configure this. | +| [Roo Code](https://docs.roocode.com/features/api-configuration-profiles#creating-and-managing-profiles) | ✅ | ✅ | Use the **OpenAI Compatible** provider with the legacy format to avoid `/v1/responses`. | +| [Codex CLI](https://github.com/openai/codex/blob/main/docs/config.md#model_providers) | ✅ | N/A | `gpt-5-codex` support is [in progress](https://github.com/coder/aibridge/issues/16). | +| [GitHub Copilot (VS Code)](https://docs.github.com/en/copilot/configuring-github-copilot/configuring-network-settings-for-github-copilot) | ✅ | ❌ | Requires the pre-release extension. Anthropic endpoints are not supported. | +| Goose | ❓ | ❓ | | +| Goose Desktop | ❓ | ✅ | | +| WindSurf | ❌ | — | No option to override the base URL. | +| Sourcegraph Amp | ❌ | — | No option to override the base URL. | +| Kiro | ❌ | — | No option to override the base URL. | +| [Copilot CLI](https://github.com/github/copilot-cli/issues/104) | ❌ | ❌ | No support for custom base URLs and uses a `GITHUB_TOKEN` for authentication. | +| [Kilo Code](https://kilocode.ai/docs/features/api-configuration-profiles#creating-and-managing-profiles) | ✅ | ✅ | Similar to Roo Code. | +| Gemini CLI | ❌ | ❌ | Not supported yet (`GOOGLE_GEMINI_BASE_URL`). | +| [Amazon Q CLI](https://aws.amazon.com/q/) | ❌ | ❌ | Limited to Amazon Q subscriptions; no custom endpoint support. | + +Legend: ✅ works, ⚠️ limited support, ❌ not supported, ❓ not yet verified, — not applicable. + +### Compatibility Overview + +Most AI coding assistants can use AI Bridge, provided they support custom base URLs. Client-specific requirements vary: + +- Some clients require specific URL formats (for example, removing the `/v1` suffix). +- Some clients proxy requests through their own servers, which limits compatibility. +- Some clients do not support custom base URLs. + +See the table in the [compatibility](#compatibility) section above for the combinations we have verified and any known issues. diff --git a/docs/ai-coder/ai-bridge/index.md b/docs/ai-coder/ai-bridge/index.md new file mode 100644 index 0000000000000..8368c2725419b --- /dev/null +++ b/docs/ai-coder/ai-bridge/index.md @@ -0,0 +1,39 @@ +# AI Bridge + +![AI bridge diagram](../../images/aibridge/aibridge_diagram.png) + +AI Bridge is a smart proxy for AI. It acts as a man-in-the-middle between your users' coding agents / IDEs +and providers like OpenAI and Anthropic. By intercepting all the AI traffic between these clients and +the upstream APIs, AI Bridge can record user prompts, token usage, and tool invocations. + +AI Bridge solves 3 key problems: + +1. **Centralized authn/z management**: no more issuing & managing API tokens for OpenAI/Anthropic usage. + Users use their Coder session or API tokens to authenticate with `coderd` (Coder control plane), and + `coderd` securely communicates with the upstream APIs on their behalf. +2. **Auditing and attribution**: all interactions with AI services, whether autonomous or human-initiated, + will be audited and attributed back to a user. +3. **Centralized MCP administration**: define a set of approved MCP servers and tools which your users may + use. + +## When to use AI Bridge + +As LLM adoption grows, administrators need centralized auditing, monitoring, and token management. AI Bridge enables organizations to manage AI tooling access for thousands of engineers from a single control plane. + +If you are an administrator or devops leader looking to: + +- Measure AI tooling adoption across teams or projects +- Establish an audit trail of prompts, issues, and tools invoked +- Manage token spend in a central dashboard +- Investigate opportunities for AI automation +- Uncover high-leverage use cases last + +AI Bridge is best suited for organizations facing these centralized management and observability challenges. + +## Next steps + +- [Set up AI Bridge](./setup.md) on your Coder deployment +- [Configure AI clients](./client-config.md) to use AI Bridge +- [Configure MCP servers](./mcp.md) for tool access +- [Monitor usage and metrics](./monitoring.md) +- [Reference documentation](./reference.md) diff --git a/docs/ai-coder/ai-bridge/mcp.md b/docs/ai-coder/ai-bridge/mcp.md new file mode 100644 index 0000000000000..498e8f3a36a0c --- /dev/null +++ b/docs/ai-coder/ai-bridge/mcp.md @@ -0,0 +1,66 @@ +# MCP + +[Model Context Protocol (MCP)](https://modelcontextprotocol.io/docs/getting-started/intro) is a mechanism for connecting AI applications to external systems. + +AI Bridge can connect to MCP servers and inject tools automatically, enabling you to centrally manage the list of tools you wish to grant your users. + +> [!NOTE] +> Only MCP servers which support OAuth2 Authorization are supported currently. +> +> [_Streamable HTTP_](https://modelcontextprotocol.io/specification/2025-06-18/basic/transports#streamable-http) is the only supported transport currently. In future releases we will support the (now deprecated) [_Server-Sent Events_](https://modelcontextprotocol.io/specification/2025-06-18/basic/transports#backwards-compatibility) transport. + +AI Bridge makes use of [External Auth](../../admin/external-auth/index.md) applications, as they define OAuth2 connections to upstream services. If your External Auth application hosts a remote MCP server, you can configure AI Bridge to connect to it, retrieve its tools and inject them into requests automatically - all while using each individual user's access token. + +For example, GitHub has a [remote MCP server](https://github.com/github/github-mcp-server?tab=readme-ov-file#remote-github-mcp-server) and we can use it as follows. + +```bash +CODER_EXTERNAL_AUTH_0_TYPE=github +CODER_EXTERNAL_AUTH_0_CLIENT_ID=... +CODER_EXTERNAL_AUTH_0_CLIENT_SECRET=... +# Tell AI Bridge where it can find this service's remote MCP server. +CODER_EXTERNAL_AUTH_0_MCP_URL=https://api.githubcopilot.com/mcp/ +``` + +See the diagram in [Implementation Details](./reference.md#implementation-details) for more information. + +You can also control which tools are injected by using an allow and/or a deny regular expression on the tool names: + +```env +CODER_EXTERNAL_AUTH_0_MCP_TOOL_ALLOW_REGEX=(.+_gist.*) +CODER_EXTERNAL_AUTH_0_MCP_TOOL_DENY_REGEX=(create_gist) +``` + +In the above example, all tools containing `_gist` in their name will be allowed, but `create_gist` is denied. + +The logic works as follows: + +- If neither the allow/deny patterns are defined, all tools will be injected. +- The deny pattern takes precedence. +- If only a deny pattern is defined, all tools are injected except those explicitly denied. + +In the above example, if you prompted your AI model with "list your available github tools by name", it would reply something like: + +> Certainly! Here are the GitHub-related tools that I have available: +> +> ```text +> 1. bmcp_github_update_gist +> 2. bmcp_github_list_gists +> ``` + +AI Bridge marks automatically injected tools with a prefix `bmcp_` ("bridged MCP"). It also namespaces all tool names by the ID of their associated External Auth application (in this case `github`). + +## Tool Injection + +If a model decides to invoke a tool and it has a `bmcp_` suffix and AI Bridge has a connection with the related MCP server, it will invoke the tool. The tool result will be passed back to the upstream AI provider, and this will loop until the model has all of its required data. These inner loops are not relayed back to the client; all it seems is the result of this loop. See [Implementation Details](./reference.md#implementation-details). + +In contrast, tools which are defined by the client (i.e. the [`Bash` tool](https://docs.claude.com/en/docs/claude-code/settings#tools-available-to-claude) defined by _Claude Code_) cannot be invoked by AI Bridge, and the tool call from the model will be relayed to the client, after which it will invoke the tool. + +If you have the `oauth2` and `mcp-server-http` experiments enabled, Coder's own [internal MCP tools](../mcp-server.md) will be injected automatically. + +### Troubleshooting + +- **Too many tools**: should you receive an error like `Invalid 'tools': array too long. Expected an array with maximum length 128, but got an array with length 132 instead`, you can reduce the number by filtering out tools using the allow/deny patterns documented in the [MCP](#mcp) section. + +- **Coder MCP tools not being injected**: in order for Coder MCP tools to be injected, the internal MCP server needs to be active. Follow the instructions in the [MCP Server](../mcp-server.md) page to enable it. + +- **External Auth tools not being injected**: this is generally due to the requesting user not being authenticated against the External Auth app; when this is the case, no attempt is made to connect to the MCP server. diff --git a/docs/ai-coder/ai-bridge/monitoring.md b/docs/ai-coder/ai-bridge/monitoring.md new file mode 100644 index 0000000000000..6fd04e09b6a39 --- /dev/null +++ b/docs/ai-coder/ai-bridge/monitoring.md @@ -0,0 +1,11 @@ +# Monitoring + +AI Bridge records the last `user` prompt, token usage, and every tool invocation for each intercepted request. Each capture is tied to a single "interception" that maps back to the authenticated Coder identity, making it easy to attribute spend and behaviour. + +![User Prompt logging](../../images/aibridge/grafana_user_prompts_logging.png) + +![User Leaderboard](../../images/aibridge/grafana_user_leaderboard.png) + +We provide an example Grafana dashboard that you can import as a starting point for your metrics. See [the Grafana dashboard README](https://github.com/coder/coder/blob/main/examples/monitoring/dashboards/grafana/aibridge/README.md). + +These logs and metrics can be used to determine usage patterns, track costs, and evaluate tooling adoption. diff --git a/docs/ai-coder/ai-bridge/reference.md b/docs/ai-coder/ai-bridge/reference.md new file mode 100644 index 0000000000000..731a4ee6ee287 --- /dev/null +++ b/docs/ai-coder/ai-bridge/reference.md @@ -0,0 +1,41 @@ +# Reference + +## Implementation Details + +`coderd` runs an in-memory instance of `aibridged`, whose logic is mostly contained in https://github.com/coder/aibridge. In future releases we will support running external instances for higher throughput and complete memory isolation from `coderd`. + +![AI Bridge implementation details](../../images/aibridge/aibridge-implementation-details.png) + +## Supported APIs + +API support is broken down into two categories: + +- **Intercepted**: requests are intercepted, audited, and augmented - full AI Bridge functionality +- **Passthrough**: requests are proxied directly to the upstream, no auditing or augmentation takes place + +Where relevant, both streaming and non-streaming requests are supported. + +### OpenAI + +**Intercepted**: + +- [`/v1/chat/completions`](https://platform.openai.com/docs/api-reference/chat/create) + +**Passthrough**: + +- [`/v1/models(/*)`](https://platform.openai.com/docs/api-reference/models/list) +- [`/v1/responses`](https://platform.openai.com/docs/api-reference/responses/create) _(Interception support coming in **Beta**)_ + +### Anthropic + +**Intercepted**: + +- [`/v1/messages`](https://docs.claude.com/en/api/messages) + +**Passthrough**: + +- [`/v1/models(/*)`](https://docs.claude.com/en/api/models-list) + +## Troubleshooting + +To report a bug, file a feature request, or view a list of known issues, please visit our [GitHub repository for AI Bridge](https://github.com/coder/aibridge). If you encounter issues with AI Bridge during early access, please reach out to us via [Discord](https://discord.gg/coder). diff --git a/docs/ai-coder/ai-bridge/setup.md b/docs/ai-coder/ai-bridge/setup.md new file mode 100644 index 0000000000000..a9d5004f1dee3 --- /dev/null +++ b/docs/ai-coder/ai-bridge/setup.md @@ -0,0 +1,62 @@ +# Setup + +AI Bridge runs inside the Coder control plane (`coderd`), requiring no separate compute to deploy or scale. Once enabled, `coderd` runs the `aibridged` in-memory and brokers traffic to your configured AI providers on behalf of authenticated users. + +**Required**: + +1. A **premium** licensed Coder deployment +1. Feature must be [enabled](#activation) using the server flag +1. One or more [providers](#configure-providers) API key(s) must be configured + +## Activation + +You will need to enable AI Bridge explicitly: + +```sh +CODER_AIBRIDGE_ENABLED=true coder server +# or +coder server --aibridge-enabled=true +``` + +## Configure Providers + +AI Bridge proxies requests to upstream LLM APIs. Configure at least one provider before exposing AI Bridge to end users. + +
+ +### OpenAI + +Set the following when routing [OpenAI-compatible](https://coder.com/docs/reference/cli/server#--aibridge-openai-key) traffic through AI Bridge: + +- `CODER_AIBRIDGE_OPENAI_KEY` or `--aibridge-openai-key` +- `CODER_AIBRIDGE_OPENAI_BASE_URL` or `--aibridge-openai-base-url` + +The default base URL (`https://api.openai.com/v1/`) works for the native OpenAI service. Point the base URL at your preferred OpenAI-compatible endpoint (for example, a hosted proxy or LiteLLM deployment) when needed. + +### Anthropic + +Set the following when routing [Anthropic-compatible](https://coder.com/docs/reference/cli/server#--aibridge-anthropic-key) traffic through AI Bridge: + +- `CODER_AIBRIDGE_ANTHROPIC_KEY` or `--aibridge-anthropic-key` +- `CODER_AIBRIDGE_ANTHROPIC_BASE_URL` or `--aibridge-anthropic-base-url` + +The default base URL (`https://api.anthropic.com/`) targets Anthropic's public API. Override it for Anthropic-compatible brokers. + +### Amazon Bedrock + +Set the following when routing [Amazon Bedrock](https://coder.com/docs/reference/cli/server#--aibridge-bedrock-region) traffic through AI Bridge: + +- `CODER_AIBRIDGE_BEDROCK_REGION` or `--aibridge-bedrock-region` +- `CODER_AIBRIDGE_BEDROCK_ACCESS_KEY` or `--aibridge-bedrock-access-key` +- `CODER_AIBRIDGE_BEDROCK_ACCESS_KEY_SECRET` or `--aibridge-bedrock-access-key-secret` +- `CODER_AIBRIDGE_BEDROCK_MODEL` or `--aibridge-bedrock-model` +- `CODER_AIBRIDGE_BEDROCK_SMALL_FAST_MODEL` or `--aibridge-bedrock-small-fast-model` + +### Additional providers and Model Proxies + +AI Bridge can relay traffic to other OpenAI- or Anthropic-compatible services or model proxies like LiteLLM by pointing the base URL variables above at the provider you operate. Share feedback or follow along in the [`aibridge`](https://github.com/coder/aibridge) issue tracker as we expand support for additional providers. + +
+ +> [!NOTE] +> See the [Supported APIs](./reference.md#supported-apis) section below for precise endpoint coverage and interception behavior. diff --git a/docs/images/aibridge/aibridge-implementation-details.png b/docs/images/aibridge/aibridge-implementation-details.png new file mode 100644 index 0000000000000..41c3c55e4aa32 Binary files /dev/null and b/docs/images/aibridge/aibridge-implementation-details.png differ diff --git a/docs/manifest.json b/docs/manifest.json index f0ead54182c8e..35598720fe834 100644 --- a/docs/manifest.json +++ b/docs/manifest.json @@ -927,9 +927,36 @@ { "title": "AI Bridge", "description": "Centralized LLM and MCP proxy for platform teams", - "path": "./ai-coder/ai-bridge.md", + "path": "./ai-coder/ai-bridge/index.md", "icon_path": "./images/icons/api.svg", - "state": ["premium", "early access"] + "state": ["premium", "early access"], + "children": [ + { + "title": "Setup", + "description": "How to set up and configure AI Bridge", + "path": "./ai-coder/ai-bridge/setup.md" + }, + { + "title": "Client Configuration", + "description": "How to configure your AI coding tools to use AI Bridge", + "path": "./ai-coder/ai-bridge/client-config.md" + }, + { + "title": "MCP", + "description": "How to use AI Bridge with MCP", + "path": "./ai-coder/ai-bridge/mcp.md" + }, + { + "title": "Monitoring", + "description": "How to monitor AI Bridge", + "path": "./ai-coder/ai-bridge/monitoring.md" + }, + { + "title": "Reference", + "description": "Technical reference for AI Bridge", + "path": "./ai-coder/ai-bridge/reference.md" + } + ] }, { "title": "Tasks CLI",