ton/pi_harness
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(coding-agent): merge origin/main into model runtime facade

Browse Source
This commit is contained in:
Mario Zechner
2026-07-15 12:25:36 +02:00
parent 9993c96907 f4e9ca7466
commit cd7cad4ee2
119 changed files with 4275 additions and 631 deletions
Download Patch File Download Diff File Expand all files Collapse all files
packages/ai/CHANGELOG.md
+24 -34
View File
@@ -8,45 +8,20 @@
- Removed the legacy built-in OAuth provider objects, global OAuth registry APIs, and public low-level built-in login/refresh functions. Use canonical `Provider.auth.oauth` methods instead; the `oauth` subpath now retains only extension compatibility types.
- Renamed the canonical login interaction interface from `AuthLoginCallbacks` to `AuthInteraction`; it exposes the provider-neutral `prompt()`/`notify()` protocol used by API-key and OAuth flows.
- Changed the `Models` request contract: `getAuth(model)` now includes model headers, while `getAuth(providerId)` remains provider-scoped, and Models stream options may include `transformHeaders`. Custom `Models` implementations must execute the transform after merging auth/model and explicit headers, then remove it before provider dispatch.
#### Models migration
Use the `getAuth()` argument to choose the desired scope:
```typescript
// Provider auth only
const providerAuth = await models.getAuth(model.provider);
// Provider auth plus model.headers
const modelAuth = await models.getAuth(model);
```
Use the Models-only `transformHeaders` option instead of resolving auth before streaming. It runs once on assembled headers and is not passed to `Provider.stream*()`:
```typescript
models.streamSimple(model, context, {
transformHeaders: async (headers) => ({
...headers,
"X-Request-ID": requestId,
}),
});
```
Custom `Models` implementations must apply the same order:
```text
getAuth(model) -> explicit options.headers -> transformHeaders -> Provider.stream*()
```
`Provider.stream*()` continues to accept ordinary `ApiStreamOptions`/`SimpleStreamOptions`; providers do not handle `transformHeaders`.
- Changed dynamic model refresh to `Models.refresh(options)`, which refreshes every configured dynamic provider and returns per-provider errors/cancellation state. `Provider.refreshModels(context)` now receives the effective credential, scoped model storage, network policy, and abort signal.
- Removed the `OpenAIResponsesCompat.sendSessionIdHeader` flag. Session-affinity behavior is now controlled by `compat.sessionAffinityFormat` (`"openai"`, `"openai-nosession"`, or `"openrouter"`). Replace `sendSessionIdHeader: false` with `sessionAffinityFormat: "openai-nosession"` ([#6366](https://github.com/earendil-works/pi/issues/6366)).
### Added
- Added provider-owned authentication and availability resolution to `Models`, including stored OAuth refresh and interactive login support through `CredentialStore`.
- Added async non-secret credential enumeration through `CredentialStore.list()` and credential-aware `Provider.filterModels()` availability policy.
- Added neutral auth-flow information/link events and provider-owned Amazon Bedrock and Google Vertex AI credential selection flows.
- Added a separate opt-in `max` thinking level, including native `xhigh` and `max` support for GPT-5.6 and Anthropic adaptive-thinking effort metadata matching Anthropic's documentation: `max` on all adaptive Claude models, native `xhigh` on Opus 4.7/4.8, Sonnet 5, and Fable 5 only.
- Added request-wide input-token pricing tiers to model cost metadata and usage cost calculation.
- Added `ModelsStore` with an in-memory default for restoring and persisting dynamic provider catalogs.
- Added the dynamic Radius `pi-messages` gateway provider with OAuth and credential-specific catalog refresh.
- Added cache-friendly dynamic tool loading. `ToolResultMessage.addedToolNames` marks where tools from `Context.tools` became available; Anthropic and OpenAI Responses use native deferred loading so late tools stay out of the cached prefix, while other providers continue using `Context.tools` normally ([#6474](https://github.com/earendil-works/pi-mono/pull/6474)).
- Added native `xhigh` and `max` thinking levels for Claude Fable 5 across all generated provider catalogs ([#6490](https://github.com/earendil-works/pi-mono/pull/6490) by [@davidbrai](https://github.com/davidbrai)).
- Added `toolChoice` support to OpenAI Codex Responses, including `"required"` to force a tool call.
- Added `toolChoice` support to OpenAI Responses, including required and named tool selection.
### Changed
@@ -56,9 +31,23 @@ getAuth(model) -> explicit options.headers -> transformHeaders -> Provider.strea
- Fixed Cloudflare Workers AI and AI Gateway streams to materialize account and gateway endpoint placeholders after auth resolution, including compat streaming with custom model objects.
- Fixed lazy provider streams to preserve their final assistant message when forwarding an inner stream.
- Fixed OpenRouter model context windows to use the top provider's actual context length ([#6481](https://github.com/earendil-works/pi-mono/pull/6481) by [@davidbrai](https://github.com/davidbrai)).
- Fixed the GitHub Copilot `mai-code-1-flash-picker` model to route through the `/responses` endpoint.
- Fixed Amazon Bedrock requests to use the generic `apiKey` stream option as a Bedrock bearer token.
- Fixed OpenRouter OpenAI-compatible session IDs to use the `x-session-id` header instead of OpenAI-specific session-affinity fields ([#6366](https://github.com/earendil-works/pi/issues/6366)).
## [0.80.6] - 2026-07-09
### Added
- Added a separate opt-in `max` thinking level, including native `xhigh` and `max` support for GPT-5.6 and Anthropic adaptive-thinking effort metadata matching Anthropic's documentation: `max` on all adaptive Claude models, native `xhigh` on Opus 4.7/4.8, Sonnet 5, and Fable 5 only.
- Added request-wide input-token pricing tiers to model cost metadata and usage cost calculation.
### Fixed
- Fixed post-compaction output-token budgeting to ignore stale assistant usage from before the compaction boundary ([#6464](https://github.com/earendil-works/pi/issues/6464)).
- Fixed GPT-5.4 and GPT-5.5 long-context cost accounting while retaining the intentional 272K default context limit for models that require an explicit override.
- Fixed GPT-5.6 metadata to keep direct OpenAI requests in the 272K short-context tier while exposing the Codex backend's 372K context window with long-context pricing.
- Fixed GPT-5.6 metadata to keep direct OpenAI requests in the 272K short-context tier while exposing the Codex backend's 372K context window with long-context pricing, and removed the nonexistent bare `gpt-5.6` alias from the OpenAI and Azure OpenAI Responses catalogs.
- Fixed Anthropic message conversion to preserve thinking blocks with empty thinking text but a valid signature instead of dropping them, avoiding thinking-block errors on newer Claude models ([#6457](https://github.com/earendil-works/pi/pull/6457) by [@davidbrai](https://github.com/davidbrai)).
## [0.80.5] - 2026-07-09
@@ -77,6 +66,7 @@ getAuth(model) -> explicit options.headers -> transformHeaders -> Provider.strea
- Fixed Amazon Bedrock Claude 5 prompt-cache pricing metadata by removing stale fallback overrides.
- Fixed DS4 server context overflow detection for `Prompt has ... tokens, but the configured context size is ... tokens` errors ([#6262](https://github.com/earendil-works/pi/issues/6262)).
- Fixed OpenAI Codex WebSocket sessions to rotate cached connections before the backend's 60-minute limit, avoiding connection-limit failures on long sessions ([#6268](https://github.com/earendil-works/pi/issues/6268)).
- Fixed Cloudflare Workers AI / AI Gateway auth to fall back to the ambient `CLOUDFLARE_ACCOUNT_ID` (and `CLOUDFLARE_GATEWAY_ID`) when the stored credential carries only the API key, so `/login`-style key-only credentials no longer leave the `{CLOUDFLARE_ACCOUNT_ID}` placeholder unresolved and return 404 ([#6021](https://github.com/earendil-works/pi/issues/6021)).
- Fixed OpenAI Completions and Responses providers to send `(no tool output)` instead of `(see attached image)` when a tool result has empty text and no image content, preventing the model from hallucinating image attachments.
- Fixed OpenAI Responses and Azure OpenAI Responses requests to avoid sending `max_output_tokens` values below the provider minimum ([#6265](https://github.com/earendil-works/pi/issues/6265)).
- Fixed retry classification for Cloudflare 524 timeout responses ([#6239](https://github.com/earendil-works/pi/issues/6239)).
packages/ai/README.md
+11 -5
View File
@@ -1029,21 +1029,26 @@ const tenantGateway = createProvider({
});
```
Dynamic model lists use `refreshModels`; the provider lists empty until the first `models.refresh()`:
Dynamic model lists use `fetchModels`. `Models.refresh()` refreshes every configured dynamic provider, passing its effective API-key or refreshed OAuth credential. A `ModelsStore` persists dynamic catalogs; both stores default to in-memory implementations.
```typescript
const models = createModels({ credentials, modelsStore });
const llamacpp = createProvider({
id: 'llamacpp',
auth: { apiKey: { name: 'llama.cpp', resolve: async () => ({ auth: {} }) } },
models: [],
refreshModels: async () => fetchModelsFromServer('http://localhost:8080'),
fetchModels: async ({ signal }) => fetchModelsFromServer('http://localhost:8080', signal),
api: openAICompletionsApi(),
});
models.setProvider(llamacpp);
await models.refresh('llamacpp');
const result = await models.refresh({ signal });
if (result.aborted) console.log('refresh cancelled');
for (const [provider, error] of result.errors) console.error(provider, error);
```
Use `models.refresh({ allowNetwork: false })` to restore persisted catalogs without network access. Model reads stay synchronous and return the last restored or refreshed list.
Custom models can carry `headers` (e.g. proxies behind bot detection) and `compat` flags. `Models.getAuth(model)` includes those model headers, and stream methods merge them before explicit request headers and `transformHeaders`. See [OpenAI Compatibility Settings](#openai-compatibility-settings).
Some OpenAI-compatible servers do not understand the `developer` role used for reasoning-capable models. For those providers, set `compat.supportsDeveloperRole` to `false` so the system prompt is sent as a `system` message instead. If the server also does not support `reasoning_effort`, set `compat.supportsReasoningEffort` to `false` too. This commonly applies to Ollama, vLLM, SGLang, and similar OpenAI-compatible servers.
@@ -1117,7 +1122,8 @@ interface OpenAICompletionsCompat {
supportsReasoningEffort?: boolean; // Whether provider supports `reasoning_effort` (default: true)
supportsUsageInStreaming?: boolean; // Whether provider supports `stream_options: { include_usage: true }` (default: true)
supportsStrictMode?: boolean; // Whether provider supports `strict` in tool definitions (default: true)
sendSessionAffinityHeaders?: boolean; // Whether to send `session_id`, `x-client-request-id`, and `x-session-affinity` from `sessionId` when caching is enabled (default: false)
sendSessionAffinityHeaders?: boolean; // Send session-affinity data from `sessionId` (default: false)
sessionAffinityFormat?: 'openai' | 'openai-nosession' | 'openrouter'; // Format for session affinity: 'openai' uses `prompt_cache_key`, `session_id`, `x-client-request-id`, and `x-session-affinity`; 'openai-nosession' uses `prompt_cache_key`, `x-client-request-id`, and `x-session-affinity`; 'openrouter' uses `x-session-id` (default: auto-detected)
maxTokensField?: 'max_completion_tokens' | 'max_tokens'; // Which field name to use (default: max_completion_tokens)
requiresToolResultName?: boolean; // Whether tool results require the `name` field (default: false)
requiresAssistantAfterToolResult?: boolean; // Whether tool results must be followed by an assistant message (default: false)
@@ -1132,7 +1138,7 @@ interface OpenAICompletionsCompat {
interface OpenAIResponsesCompat {
supportsDeveloperRole?: boolean; // Whether provider supports `developer` role vs `system` (default: true)
sendSessionIdHeader?: boolean; // Whether to send `session_id` from `sessionId` when caching is enabled (default: true)
sessionAffinityFormat?: 'openai' | 'openai-nosession' | 'openrouter'; // Session-affinity header format: 'openai' sends `session_id` and `x-client-request-id`; 'openai-nosession' sends `x-client-request-id`; 'openrouter' sends `x-session-id`. Does not affect the `prompt_cache_key` body param (default: auto-detected)
supportsLongCacheRetention?: boolean; // Whether provider supports `prompt_cache_retention: "24h"` (default: true)
}
```
packages/ai/package.json
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "@earendil-works/pi-ai",
"version": "0.80.5",
"version": "0.80.6",
"description": "Unified LLM API with automatic model discovery and provider configuration",
"type": "module",
"main": "./dist/index.js",
packages/ai/scripts/generate-models.ts
+38 -9
View File
@@ -9,7 +9,14 @@ import {
CLOUDFLARE_AI_GATEWAY_OPENAI_BASE_URL,
CLOUDFLARE_WORKERS_AI_BASE_URL,
} from "../src/api/cloudflare.ts";
import type { AnthropicMessagesCompat, Api, KnownProvider, Model, OpenAICompletionsCompat } from "../src/types.ts";
import type {
AnthropicMessagesCompat,
Api,
KnownProvider,
Model,
OpenAICompletionsCompat,
OpenAIResponsesCompat,
} from "../src/types.ts";
const __filename = fileURLToPath(import.meta.url);
const __dirname = dirname(__filename);
@@ -194,6 +201,15 @@ const ANT_LING_RING_THINKING_LEVEL_MAP = {
} as const;
const MODELS_DEV_OPENAI_UNSUPPORTED_MODEL_IDS = new Set(["gpt-5.6"]);
const OPENAI_TOOL_SEARCH_MODEL_IDS = new Set([
"gpt-5.4",
"gpt-5.4-mini",
"gpt-5.4-pro",
"gpt-5.5",
"gpt-5.6-sol",
"gpt-5.6-terra",
"gpt-5.6-luna",
]);
const OPENAI_LONG_CONTEXT_INPUT_THRESHOLD = 272000;
const OPENAI_SHORT_CONTEXT_CAPPED_MODEL_IDS = new Set([
"gpt-5.4",
@@ -482,6 +498,16 @@ function applyOpenAICompletionsCompatMetadata(model: Model<Api>): void {
}
}
function applyOpenAIToolSearchMetadata(model: Model<Api>): void {
const isOpenAIResponses = model.provider === "openai" && model.api === "openai-responses";
const isOpenAICodex = model.provider === "openai-codex" && model.api === "openai-codex-responses";
if (!(isOpenAIResponses || isOpenAICodex) || !OPENAI_TOOL_SEARCH_MODEL_IDS.has(model.id)) return;
model.compat = {
...(model.compat as OpenAIResponsesCompat | undefined),
supportsToolSearch: true,
};
}
function isGemini3ProModel(modelId: string): boolean {
return /gemini-3(?:\.\d+)?-pro/.test(modelId.toLowerCase());
}
@@ -545,10 +571,7 @@ function applyThinkingLevelMetadata(model: Model<any>): void {
) {
mergeThinkingLevelMap(model, { xhigh: "xhigh", max: "max" });
}
if (
(model.api === "anthropic-messages" || model.api === "bedrock-converse-stream") &&
model.id.includes("fable-5")
) {
if (model.id.includes("fable-5")) {
mergeThinkingLevelMap(model, { off: null, xhigh: "xhigh", max: "max" });
}
if (model.api === "anthropic-messages" && isAnthropicAdaptiveThinkingModel(model.id)) {
@@ -700,6 +723,8 @@ async function fetchOpenRouterModels(): Promise<Model<any>[]> {
const cacheReadCost = roundCost(parseFloat(model.pricing?.input_cache_read || "0") * 1_000_000);
const cacheWriteCost = roundCost(parseFloat(model.pricing?.input_cache_write || "0") * 1_000_000);
const contextWindow = model.top_provider?.context_length || model.context_length || 4096;
const normalizedModel: Model<any> = {
id: modelKey,
name: model.name,
@@ -714,7 +739,7 @@ async function fetchOpenRouterModels(): Promise<Model<any>[]> {
cacheRead: cacheReadCost,
cacheWrite: cacheWriteCost,
},
contextWindow: model.context_length || 4096,
contextWindow,
maxTokens: model.top_provider?.max_completion_tokens || 4096,
};
models.push(normalizedModel);
@@ -1345,11 +1370,12 @@ async function loadModelsDevData(): Promise<Model<any>[]> {
const npm = m.provider?.npm;
let api: Api;
let baseUrl: string;
let compat: OpenAICompletionsCompat | undefined;
let compat: OpenAICompletionsCompat | OpenAIResponsesCompat | undefined;
if (npm === "@ai-sdk/openai") {
api = "openai-responses";
baseUrl = `${variant.basePath}/v1`;
compat = { sessionAffinityFormat: "openai-nosession" };
} else if (npm === "@ai-sdk/anthropic") {
api = "anthropic-messages";
// Anthropic SDK appends /v1/messages to baseURL
@@ -1438,8 +1464,10 @@ async function loadModelsDevData(): Promise<Model<any>[]> {
// Claude 4.x and 5.x models route to Anthropic Messages API
const isCopilotClaude = /^claude-(haiku|sonnet|opus)-[45]([.\-]|$)/.test(modelId);
// gpt-5 models require responses API, others use completions
const needsResponsesApi = modelId.startsWith("gpt-5") || modelId.startsWith("oswe");
// gpt-5, oswe, and MAI-Code models are only served through the
// Copilot /responses endpoint.
const needsResponsesApi =
modelId.startsWith("gpt-5") || modelId.startsWith("oswe") || modelId.startsWith("mai-");
const api: Api = isCopilotClaude
? "anthropic-messages"
@@ -2179,6 +2207,7 @@ async function generateModels() {
for (const model of allModels) {
applyThinkingLevelMetadata(model);
applyOpenAICompletionsCompatMetadata(model);
applyOpenAIToolSearchMetadata(model);
}
// Group by provider and deduplicate by model ID
packages/ai/src/api/anthropic-messages.ts
+125 -54
View File
@@ -29,6 +29,7 @@ import type {
ToolCall,
ToolResultMessage,
} from "../types.ts";
import { splitDeferredTools } from "../utils/deferred-tools.ts";
import { AssistantMessageEventStream } from "../utils/event-stream.ts";
import { headersToRecord } from "../utils/headers.ts";
import { parseJsonWithRepair, parseStreamingJson } from "../utils/json-parse.ts";
@@ -177,9 +178,24 @@ function getAnthropicCompat(
supportsCacheControlOnTools: model.compat?.supportsCacheControlOnTools ?? true,
supportsTemperature: model.compat?.supportsTemperature ?? true,
allowEmptySignature: model.compat?.allowEmptySignature ?? false,
supportsToolReferences: model.compat?.supportsToolReferences ?? defaultSupportsToolReferences(model),
};
}
/**
* Default for `supportsToolReferences`: first-party Anthropic models except
* Haiku (rejects client-side tool_reference blocks) and models that predate
* tool search (Claude 3.x, Opus/Sonnet 4.0, Opus 4.1).
*/
function defaultSupportsToolReferences(model: Model<"anthropic-messages">): boolean {
if (model.provider !== "anthropic" || model.id.includes("haiku")) return false;
const version = model.id.match(/^claude-(?:opus|sonnet|fable)-(\d+)(?:-(\d+))?(?:-|$)/);
if (!version) return false;
const major = Number(version[1]);
const minor = version[2] && version[2].length < 8 ? Number(version[2]) : 0;
return major > 4 || (major === 4 && minor >= 5);
}
export interface AnthropicOptions extends StreamOptions {
/**
* Enable extended thinking.
@@ -687,25 +703,27 @@ export const stream: StreamFunction<"anthropic-messages", AnthropicOptions> = (
}
// Only update usage fields if present (not null).
// Preserves input_tokens from message_start when proxies omit it in message_delta.
if (event.usage.input_tokens != null) {
output.usage.input = event.usage.input_tokens;
}
if (event.usage.output_tokens != null) {
output.usage.output = event.usage.output_tokens;
}
if (event.usage.cache_read_input_tokens != null) {
output.usage.cacheRead = event.usage.cache_read_input_tokens;
}
if (event.usage.cache_creation_input_tokens != null) {
output.usage.cacheWrite = event.usage.cache_creation_input_tokens;
}
// Anthropic reports reasoning tokens in `output_tokens_details.thinking_tokens` on the
// final message_delta usage (a subset of output_tokens). SDK 0.91.1 omits the field from
// its Usage type, so read it through a narrow cast. Verified against the live API.
const thinkingTokens = (event.usage as { output_tokens_details?: { thinking_tokens?: number } })
.output_tokens_details?.thinking_tokens;
if (thinkingTokens != null) {
output.usage.reasoning = thinkingTokens;
if (event.usage) {
if (event.usage.input_tokens != null) {
output.usage.input = event.usage.input_tokens;
}
if (event.usage.output_tokens != null) {
output.usage.output = event.usage.output_tokens;
}
if (event.usage.cache_read_input_tokens != null) {
output.usage.cacheRead = event.usage.cache_read_input_tokens;
}
if (event.usage.cache_creation_input_tokens != null) {
output.usage.cacheWrite = event.usage.cache_creation_input_tokens;
}
// Anthropic reports reasoning tokens in `output_tokens_details.thinking_tokens` on the
// final message_delta usage (a subset of output_tokens). SDK 0.91.1 omits the field from
// its Usage type, so read it through a narrow cast. Verified against the live API.
const thinkingTokens = (event.usage as { output_tokens_details?: { thinking_tokens?: number } })
.output_tokens_details?.thinking_tokens;
if (thinkingTokens != null) {
output.usage.reasoning = thinkingTokens;
}
}
// Anthropic doesn't provide total_tokens, compute from components
output.usage.totalTokens =
@@ -907,9 +925,30 @@ function buildParams(
): MessageCreateParamsStreaming {
const { cacheControl } = getCacheControl(model, options?.cacheRetention, options?.env);
const compat = getAnthropicCompat(model);
const transformedMessages = transformMessages(context.messages, model, normalizeToolCallId);
const normalizeToolName = isOAuthToken ? toClaudeCodeName : (name: string) => name;
const toolPlacement = splitDeferredTools(
{ ...context, messages: transformedMessages },
compat.supportsToolReferences,
normalizeToolName,
);
let immediateTools = toolPlacement.immediate;
let deferredTools = [...toolPlacement.deferred.values()];
if (immediateTools.length === 0 && deferredTools.length > 0) {
immediateTools = deferredTools;
deferredTools = [];
}
const deferredToolNames = new Set(deferredTools.map((tool) => normalizeToolName(tool.name)));
const params: MessageCreateParamsStreaming = {
model: model.id,
messages: convertMessages(context.messages, model, isOAuthToken, cacheControl, compat.allowEmptySignature),
messages: convertMessages(
transformedMessages,
isOAuthToken,
cacheControl,
compat.allowEmptySignature,
deferredToolNames,
normalizeToolName,
),
max_tokens: options?.maxTokens ?? model.maxTokens,
stream: true,
};
@@ -946,13 +985,16 @@ function buildParams(
params.temperature = options.temperature;
}
if (context.tools && context.tools.length > 0) {
params.tools = convertTools(
context.tools,
isOAuthToken,
compat.supportsEagerToolInputStreaming,
compat.supportsCacheControlOnTools ? cacheControl : undefined,
);
if (immediateTools.length > 0 || deferredTools.length > 0) {
params.tools = [
...convertTools(
immediateTools,
isOAuthToken,
compat.supportsEagerToolInputStreaming,
compat.supportsCacheControlOnTools ? cacheControl : undefined,
),
...convertTools(deferredTools, isOAuthToken, compat.supportsEagerToolInputStreaming, undefined, true),
];
}
// Configure thinking mode: adaptive, budget-based, or explicitly disabled.
@@ -1009,17 +1051,51 @@ function normalizeToolCallId(id: string): string {
return id.replace(/[^a-zA-Z0-9_-]/g, "_").slice(0, 64);
}
function convertToolResult(
msg: ToolResultMessage,
isOAuthToken: boolean,
deferredToolNames: ReadonlySet<string>,
loadedToolNames: Set<string>,
normalizeToolName: (name: string) => string,
): { toolResult: ContentBlockParam; siblingContent: ContentBlockParam[] } {
const references: Array<{ type: "tool_reference"; tool_name: string }> = [];
for (const name of msg.addedToolNames ?? []) {
const normalizedName = normalizeToolName(name);
if (!deferredToolNames.has(normalizedName) || loadedToolNames.has(normalizedName)) continue;
loadedToolNames.add(normalizedName);
references.push({
type: "tool_reference",
tool_name: isOAuthToken ? toClaudeCodeName(name) : name,
});
}
const convertedContent = convertContentBlocks(msg.content);
// Anthropic rejects tool references mixed with ordinary tool-result content.
return {
toolResult: {
type: "tool_result",
tool_use_id: msg.toolCallId,
content: references.length > 0 ? references : convertedContent,
is_error: msg.isError,
},
siblingContent:
references.length === 0
? []
: typeof convertedContent === "string"
? [{ type: "text", text: convertedContent }]
: convertedContent,
};
}
function convertMessages(
messages: Message[],
model: Model<"anthropic-messages">,
transformedMessages: Message[],
isOAuthToken: boolean,
cacheControl?: CacheControlEphemeral,
allowEmptySignature = false,
deferredToolNames: ReadonlySet<string> = new Set(),
normalizeToolName: (name: string) => string = (name) => name,
): MessageParam[] {
const params: MessageParam[] = [];
// Transform messages for cross-provider compatibility
const transformedMessages = transformMessages(messages, model, normalizeToolCallId);
const loadedToolNames = new Set<string>();
for (let i = 0; i < transformedMessages.length; i++) {
const msg = transformedMessages[i];
@@ -1122,37 +1198,30 @@ function convertMessages(
content: blocks,
});
} else if (msg.role === "toolResult") {
// Collect all consecutive toolResult messages, needed for z.ai Anthropic endpoint
// Collect all consecutive toolResult messages, needed for z.ai Anthropic endpoint.
const toolResults: ContentBlockParam[] = [];
// Add the current tool result
toolResults.push({
type: "tool_result",
tool_use_id: msg.toolCallId,
content: convertContentBlocks(msg.content),
is_error: msg.isError,
});
// Look ahead for consecutive toolResult messages
let j = i + 1;
const siblingContent: ContentBlockParam[] = [];
let j = i;
while (j < transformedMessages.length && transformedMessages[j].role === "toolResult") {
const nextMsg = transformedMessages[j] as ToolResultMessage; // We know it's a toolResult
toolResults.push({
type: "tool_result",
tool_use_id: nextMsg.toolCallId,
content: convertContentBlocks(nextMsg.content),
is_error: nextMsg.isError,
});
const converted = convertToolResult(
transformedMessages[j] as ToolResultMessage,
isOAuthToken,
deferredToolNames,
loadedToolNames,
normalizeToolName,
);
toolResults.push(converted.toolResult);
siblingContent.push(...converted.siblingContent);
j++;
}
// Skip the messages we've already processed
// Skip the messages we've already processed.
i = j - 1;
// Add a single user message with all tool results
// Displaced reference-bearing results must follow every tool_result block.
params.push({
role: "user",
content: toolResults,
content: [...toolResults, ...siblingContent],
});
}
}
@@ -1193,6 +1262,7 @@ function convertTools(
isOAuthToken: boolean,
supportsEagerToolInputStreaming: boolean,
cacheControl?: CacheControlEphemeral,
deferLoading = false,
): Anthropic.Messages.Tool[] {
if (!tools) return [];
@@ -1208,6 +1278,7 @@ function convertTools(
properties: schema.properties ?? {},
required: schema.required ?? [],
},
...(deferLoading ? { defer_loading: true } : {}),
...(cacheControl && index === tools.length - 1 ? { cache_control: cacheControl } : {}),
};
});
packages/ai/src/api/bedrock-converse-stream.ts
+15 -8
View File
@@ -152,7 +152,10 @@ export const stream: StreamFunction<"bedrock-converse-stream", BedrockOptions> =
// Resolve bearer token for Bedrock API key auth.
const skipAuth = getProviderEnvValue("AWS_BEDROCK_SKIP_AUTH", options.env) === "1";
const bearerToken =
options.bearerToken || getProviderEnvValue("AWS_BEARER_TOKEN_BEDROCK", options.env) || undefined;
options.bearerToken ||
options.apiKey ||
getProviderEnvValue("AWS_BEARER_TOKEN_BEDROCK", options.env) ||
undefined;
const useBearerToken = bearerToken !== undefined && !skipAuth;
// in Node.js/Bun environment only
@@ -257,7 +260,11 @@ export const stream: StreamFunction<"bedrock-converse-stream", BedrockOptions> =
} else if (item.contentBlockStop) {
handleContentBlockStop(item.contentBlockStop, blocks, output, stream);
} else if (item.messageStop) {
output.stopReason = mapStopReason(item.messageStop.stopReason);
const { stopReason, errorMessage } = mapStopReason(item.messageStop.stopReason);
output.stopReason = stopReason;
if (errorMessage) {
output.errorMessage = errorMessage;
}
} else if (item.metadata) {
handleMetadata(item.metadata, model, output);
} else if (item.internalServerException) {
@@ -278,7 +285,7 @@ export const stream: StreamFunction<"bedrock-converse-stream", BedrockOptions> =
}
if (output.stopReason === "error" || output.stopReason === "aborted") {
throw new Error("An unknown error occurred");
throw new Error(output.errorMessage || "An unknown error occurred");
}
stream.push({ type: "done", reason: output.stopReason, message: output });
@@ -929,18 +936,18 @@ function convertToolConfig(
return { tools: bedrockTools, toolChoice: bedrockToolChoice };
}
function mapStopReason(reason: string | undefined): StopReason {
function mapStopReason(reason: string | undefined): { stopReason: StopReason; errorMessage?: string } {
switch (reason) {
case BedrockStopReason.END_TURN:
case BedrockStopReason.STOP_SEQUENCE:
return "stop";
return { stopReason: "stop" };
case BedrockStopReason.MAX_TOKENS:
case BedrockStopReason.MODEL_CONTEXT_WINDOW_EXCEEDED:
return "length";
return { stopReason: "length" };
case BedrockStopReason.TOOL_USE:
return "toolUse";
return { stopReason: "toolUse" };
default:
return "error";
return reason ? { stopReason: "error", errorMessage: reason } : { stopReason: "error" };
}
}
packages/ai/src/api/openai-codex-responses.ts
+8 -4
View File
@@ -36,6 +36,7 @@ import type {
Usage,
} from "../types.ts";
import { combineAbortSignals } from "../utils/abort-signals.ts";
import { splitDeferredTools } from "../utils/deferred-tools.ts";
import {
appendAssistantMessageDiagnostic,
createAssistantMessageDiagnostic,
@@ -84,6 +85,7 @@ export interface OpenAICodexResponsesOptions extends StreamOptions {
reasoningSummary?: "auto" | "concise" | "detailed" | "off" | "on" | null;
serviceTier?: ResponseCreateParamsStreaming["service_tier"];
textVerbosity?: "low" | "medium" | "high";
toolChoice?: "auto" | "none" | "required";
}
type CodexResponseStatus = "completed" | "incomplete" | "failed" | "cancelled" | "queued" | "in_progress";
@@ -96,7 +98,7 @@ interface RequestBody {
previous_response_id?: string;
input?: ResponseInput;
tools?: OpenAITool[];
tool_choice?: "auto";
tool_choice?: OpenAICodexResponsesOptions["toolChoice"];
parallel_tool_calls?: boolean;
temperature?: number;
reasoning?: { effort?: string; summary?: string };
@@ -481,8 +483,10 @@ function buildRequestBody(
context: Context,
options?: OpenAICodexResponsesOptions,
): RequestBody {
const toolPlacement = splitDeferredTools(context, model.compat?.supportsToolSearch ?? false);
const messages = convertResponsesMessages(model, context, CODEX_TOOL_CALL_PROVIDERS, {
includeSystemPrompt: false,
deferredTools: toolPlacement.deferred,
});
const body: RequestBody = {
@@ -494,7 +498,7 @@ function buildRequestBody(
text: { verbosity: options?.textVerbosity || "low" },
include: ["reasoning.encrypted_content"],
prompt_cache_key: clampOpenAIPromptCacheKey(options?.sessionId),
tool_choice: "auto",
tool_choice: options?.toolChoice ?? "auto",
parallel_tool_calls: true,
};
@@ -506,8 +510,8 @@ function buildRequestBody(
body.service_tier = options.serviceTier;
}
if (context.tools && context.tools.length > 0) {
body.tools = convertResponsesTools(context.tools, { strict: null });
if (toolPlacement.immediate.length > 0) {
body.tools = convertResponsesTools(toolPlacement.immediate, { strict: null });
}
if (options?.reasoningEffort !== undefined) {
packages/ai/src/api/openai-completions.ts
+11 -3
View File
@@ -519,9 +519,15 @@ function createClient(
}
if (sessionId && compat.sendSessionAffinityHeaders) {
headers.session_id = sessionId;
headers["x-client-request-id"] = sessionId;
headers["x-session-affinity"] = sessionId;
if (compat.sessionAffinityFormat === "openrouter") {
headers["x-session-id"] = sessionId;
} else {
if (compat.sessionAffinityFormat === "openai") {
headers.session_id = sessionId;
}
headers["x-client-request-id"] = sessionId;
headers["x-session-affinity"] = sessionId;
}
}
// Merge options headers last so they can override defaults
@@ -1250,6 +1256,7 @@ function detectCompat(model: Model<"openai-completions">): ResolvedOpenAIComplet
supportsStrictMode: !isMoonshot && !isTogether && !isCloudflareAiGateway && !isNvidia,
cacheControlFormat,
sendSessionAffinityHeaders: false,
sessionAffinityFormat: isOpenRouter ? "openrouter" : "openai",
supportsLongCacheRetention: !(
isTogether ||
isCloudflareWorkersAI ||
@@ -1289,6 +1296,7 @@ function getCompat(model: Model<"openai-completions">): ResolvedOpenAICompletion
supportsStrictMode: model.compat.supportsStrictMode ?? detected.supportsStrictMode,
cacheControlFormat: model.compat.cacheControlFormat ?? detected.cacheControlFormat,
sendSessionAffinityHeaders: model.compat.sendSessionAffinityHeaders ?? detected.sendSessionAffinityHeaders,
sessionAffinityFormat: model.compat.sessionAffinityFormat ?? detected.sessionAffinityFormat,
supportsLongCacheRetention: model.compat.supportsLongCacheRetention ?? detected.supportsLongCacheRetention,
};
}
packages/ai/src/api/openai-responses-shared.ts
+65 -8
View File
@@ -6,11 +6,13 @@ import type {
ResponseInput,
ResponseInputContent,
ResponseInputImage,
ResponseInputItem,
ResponseInputText,
ResponseOutputItem,
ResponseOutputMessage,
ResponseReasoningItem,
ResponseStreamEvent,
ResponseToolSearchOutputItemParam,
} from "openai/resources/responses/responses.js";
import { calculateCost } from "../models.ts";
import type {
@@ -77,12 +79,16 @@ export interface OpenAIResponsesStreamOptions {
export interface ConvertResponsesMessagesOptions {
includeSystemPrompt?: boolean;
deferredTools?: ReadonlyMap<string, Tool>;
}
export interface ConvertResponsesToolsOptions {
strict?: boolean | null;
deferLoading?: boolean;
}
type OpenAIFunctionTool = Extract<OpenAITool, { type: "function" }>;
// =============================================================================
// Message conversion
// =============================================================================
@@ -94,6 +100,7 @@ export function convertResponsesMessages<TApi extends Api>(
options?: ConvertResponsesMessagesOptions,
): ResponseInput {
const messages: ResponseInput = [];
const loadedToolNames = new Set<string>();
const normalizeIdPart = (part: string): string => {
const sanitized = part.replace(/[^a-zA-Z0-9_-]/g, "_");
@@ -259,6 +266,32 @@ export function convertResponsesMessages<TApi extends Api>(
call_id: callId,
output,
});
const deferredTools: Tool[] = [];
for (const name of msg.addedToolNames ?? []) {
const tool = options?.deferredTools?.get(name);
if (!tool || loadedToolNames.has(name)) continue;
loadedToolNames.add(name);
deferredTools.push(tool);
}
if (deferredTools.length > 0) {
const names = deferredTools.map((tool) => tool.name);
const searchCallId = `pi_tool_load_${shortHash(`${msg.toolCallId}:${names.join(",")}`)}`;
messages.push({
type: "tool_search_call",
call_id: searchCallId,
execution: "client",
status: "completed",
arguments: { query: names.join(" "), limit: names.length },
} satisfies ResponseInputItem);
messages.push({
type: "tool_search_output",
call_id: searchCallId,
execution: "client",
status: "completed",
tools: convertResponsesTools(deferredTools, { deferLoading: true }),
} satisfies ResponseToolSearchOutputItemParam);
}
}
msgIndex++;
}
@@ -270,15 +303,18 @@ export function convertResponsesMessages<TApi extends Api>(
// Tool conversion
// =============================================================================
export function convertResponsesTools(tools: Tool[], options?: ConvertResponsesToolsOptions): OpenAITool[] {
export function convertResponsesTools(tools: readonly Tool[], options?: ConvertResponsesToolsOptions): OpenAITool[] {
const strict = options?.strict === undefined ? false : options.strict;
return tools.map((tool) => ({
type: "function",
name: tool.name,
description: tool.description,
parameters: tool.parameters as any, // TypeBox already generates JSON Schema
strict,
}));
return tools.map(
(tool): OpenAIFunctionTool => ({
type: "function",
name: tool.name,
description: tool.description,
parameters: tool.parameters as Record<string, unknown>, // TypeBox already generates JSON Schema
strict,
...(options?.deferLoading ? { defer_loading: true } : {}),
}),
);
}
// =============================================================================
@@ -301,6 +337,7 @@ export async function processResponsesStream<TApi extends Api>(
): Promise<void> {
let sawTerminalResponseEvent = false;
const outputSlots = new Map<number, ResponsesOutputSlot>();
const reasoningBlocksById = new Map<string, ThinkingContent>();
const getSlot = <TType extends ResponsesOutputSlot["type"]>(
outputIndex: number,
type: TType,
@@ -352,10 +389,29 @@ export async function processResponsesStream<TApi extends Api>(
const getOrCreateSlot = (outputIndex: number, item: ResponseOutputItem): ResponsesOutputSlot | undefined => {
return outputSlots.get(outputIndex) ?? createSlot(outputIndex, item);
};
// Azure OpenAI can omit reasoning.encrypted_content from response.output_item.done
// and provide it only in response.completed.response.output. Backfill the
// persisted reasoning signature from the terminal response to keep store:false
// multi-turn replay stateless. See https://github.com/earendil-works/pi/issues/6409.
const backfillReasoningSignatures = (responseOutput: ResponseOutputItem[]): void => {
for (const item of responseOutput) {
if (item.type !== "reasoning" || !item.encrypted_content) continue;
const block = reasoningBlocksById.get(item.id);
if (!block?.thinkingSignature) continue;
const storedItem = JSON.parse(block.thinkingSignature) as ResponseReasoningItem;
if (storedItem.encrypted_content) continue;
block.thinkingSignature = JSON.stringify({
...storedItem,
encrypted_content: item.encrypted_content,
});
}
};
const finalizeResponse = (
response: Extract<ResponseStreamEvent, { type: "response.completed" | "response.incomplete" }>["response"],
): void => {
sawTerminalResponseEvent = true;
backfillReasoningSignatures(response.output ?? []);
if (response?.id) {
output.responseId = response.id;
}
@@ -483,6 +539,7 @@ export async function processResponsesStream<TApi extends Api>(
const contentText = item.content?.map((c) => c.text).join("\n\n") || "";
slot.block.thinking = summaryText || contentText || slot.block.thinking;
slot.block.thinkingSignature = JSON.stringify(item);
reasoningBlocksById.set(item.id, slot.block);
stream.push({
type: "thinking_end",
contentIndex: slot.contentIndex,
packages/ai/src/api/openai-responses.ts
+26 -8
View File
@@ -15,6 +15,7 @@ import type {
StreamOptions,
Usage,
} from "../types.ts";
import { splitDeferredTools } from "../utils/deferred-tools.ts";
import { formatProviderError, normalizeProviderError } from "../utils/error-body.ts";
import { AssistantMessageEventStream } from "../utils/event-stream.ts";
import { headersToRecord } from "../utils/headers.ts";
@@ -43,6 +44,10 @@ function getClientApiKey(provider: string, apiKey: string | undefined, headers:
throw new Error(`No API key for provider: ${provider}`);
}
function detectSessionAffinityFormat(model: Pick<Model<"openai-responses">, "provider" | "baseUrl">) {
return model.provider === "openrouter" || model.baseUrl.includes("openrouter.ai") ? "openrouter" : "openai";
}
/**
* Resolve cache retention preference.
* Defaults to "short" and uses PI_CACHE_RETENTION for backward compatibility.
@@ -60,8 +65,9 @@ function resolveCacheRetention(cacheRetention?: CacheRetention, env?: ProviderEn
function getCompat(model: Model<"openai-responses">): Required<OpenAIResponsesCompat> {
return {
supportsDeveloperRole: model.compat?.supportsDeveloperRole ?? true,
sendSessionIdHeader: model.compat?.sendSessionIdHeader ?? true,
sessionAffinityFormat: model.compat?.sessionAffinityFormat ?? detectSessionAffinityFormat(model),
supportsLongCacheRetention: model.compat?.supportsLongCacheRetention ?? true,
supportsToolSearch: model.compat?.supportsToolSearch ?? false,
};
}
@@ -81,6 +87,7 @@ export interface OpenAIResponsesOptions extends StreamOptions {
reasoningEffort?: "minimal" | "low" | "medium" | "high" | "xhigh" | "max";
reasoningSummary?: "auto" | "detailed" | "concise" | null;
serviceTier?: ResponseCreateParamsStreaming["service_tier"];
toolChoice?: ResponseCreateParamsStreaming["tool_choice"];
}
/**
@@ -200,10 +207,14 @@ function createClient(
}
if (sessionId) {
if (compat.sendSessionIdHeader) {
headers.session_id = sessionId;
if (compat.sessionAffinityFormat === "openrouter") {
headers["x-session-id"] = sessionId;
} else {
if (compat.sessionAffinityFormat === "openai") {
headers.session_id = sessionId;
}
headers["x-client-request-id"] = sessionId;
}
headers["x-client-request-id"] = sessionId;
}
// Merge options headers last so they can override defaults
@@ -220,10 +231,13 @@ function createClient(
}
function buildParams(model: Model<"openai-responses">, context: Context, options?: OpenAIResponsesOptions) {
const messages = convertResponsesMessages(model, context, OPENAI_TOOL_CALL_PROVIDERS);
const compat = getCompat(model);
const toolPlacement = splitDeferredTools(context, compat.supportsToolSearch);
const messages = convertResponsesMessages(model, context, OPENAI_TOOL_CALL_PROVIDERS, {
deferredTools: toolPlacement.deferred,
});
const cacheRetention = resolveCacheRetention(options?.cacheRetention, options?.env);
const compat = getCompat(model);
const params: ResponseCreateParamsStreaming = {
model: model.id,
input: messages,
@@ -245,8 +259,12 @@ function buildParams(model: Model<"openai-responses">, context: Context, options
params.service_tier = options.serviceTier;
}
if (context.tools && context.tools.length > 0) {
params.tools = convertResponsesTools(context.tools);
if (toolPlacement.immediate.length > 0) {
params.tools = convertResponsesTools(toolPlacement.immediate);
}
if (options?.toolChoice !== undefined) {
params.tool_choice = options.toolChoice;
}
if (model.reasoning) {
packages/ai/src/api/pi-messages.lazy.ts
+4
View File
@@ -0,0 +1,4 @@
import type { ProviderStreams } from "../types.ts";
import { lazyApi } from "./lazy.ts";
export const piMessagesApi = (): ProviderStreams => lazyApi(() => import("./pi-messages.ts"));
packages/ai/src/api/pi-messages.ts
+436
View File
@@ -0,0 +1,436 @@
/**
* pi-messages API implementation.
*
* Streams pi's own message protocol directly to a backend: the request is a
* single POST of `{ model, context, options }` to `<baseUrl>/messages`, the
* response is an SSE stream of serialized assistant-message events plus a
* terminal `done`/`error` event. This is the wire protocol spoken by the
* Radius gateway, but any backend implementing it can be used, e.g. via a
* models.json custom provider with `"api": "pi-messages"`.
*/
import type {
AssistantMessage,
AssistantMessageEvent,
CacheRetention,
Context,
Model,
ProviderEnv,
SimpleStreamOptions,
StreamFunction,
StreamOptions,
ThinkingLevel,
ToolCall,
} from "../types.ts";
import { appendAssistantMessageDiagnostic, createAssistantMessageDiagnostic } from "../utils/diagnostics.ts";
import { AssistantMessageEventStream } from "../utils/event-stream.ts";
import { headersToRecord, providerHeadersToRecord } from "../utils/headers.ts";
import { parseStreamingJson } from "../utils/json-parse.ts";
import { getProviderEnvValue } from "../utils/provider-env.ts";
export interface PiMessagesOptions extends StreamOptions {
reasoning?: ThinkingLevel;
toolChoice?: "auto" | "none" | "required" | { type: "function"; function: { name: string } };
/** Ask the backend for debug metadata (e.g. routing response headers). */
debug?: boolean;
}
type PiMessagesUsage = AssistantMessage["usage"];
type PiMessagesStopReason = AssistantMessage["stopReason"];
/** Impact summary of a server-side message rewrite (e.g. a gateway policy). */
export type PiMessagesRewriteImpact = {
policyId: string;
policyVersion: number;
changed: boolean;
tokenCountChange: number;
messageCountChange: number;
systemPromptChanged: boolean;
};
/** Serialized assistant-message event as sent by a pi-messages backend. */
export type PiMessagesEvent =
| { type: "start" }
| { type: "text_start"; contentIndex: number }
| { type: "text_delta"; contentIndex: number; delta: string }
| { type: "text_end"; contentIndex: number; content: string; contentSignature?: string }
| { type: "thinking_start"; contentIndex: number }
| { type: "thinking_delta"; contentIndex: number; delta: string }
| {
type: "thinking_end";
contentIndex: number;
content: string;
contentSignature?: string;
redacted?: boolean;
}
| { type: "toolcall_start"; contentIndex: number; id: string; toolName: string }
| { type: "toolcall_delta"; contentIndex: number; delta: string }
| { type: "toolcall_end"; contentIndex: number; toolCall: ToolCall }
| {
type: "done";
reason: Extract<PiMessagesStopReason, "stop" | "length" | "toolUse">;
usage: PiMessagesUsage;
responseId?: string;
rewrite?: PiMessagesRewriteImpact;
}
| {
type: "error";
reason: Extract<PiMessagesStopReason, "aborted" | "error">;
usage: PiMessagesUsage;
errorMessage?: string;
responseId?: string;
rewrite?: PiMessagesRewriteImpact;
};
type PiMessagesErrorBody = {
error?: {
message?: unknown;
code?: unknown;
details?: unknown;
[key: string]: unknown;
};
};
export class PiMessagesResponseError extends Error {
code?: string;
readonly diagnosticDetails: Record<string, unknown>;
constructor(message: string, code: string | undefined, diagnosticDetails: Record<string, unknown>) {
super(message);
this.name = "PiMessagesResponseError";
this.code = code;
this.diagnosticDetails = diagnosticDetails;
}
}
function isRecord(value: unknown): value is Record<string, unknown> {
return typeof value === "object" && value !== null && !Array.isArray(value);
}
function parsePiMessagesErrorBody(body: string): PiMessagesErrorBody | undefined {
try {
const parsed = JSON.parse(body) as unknown;
return isRecord(parsed) && isRecord(parsed.error) ? (parsed as PiMessagesErrorBody) : undefined;
} catch {
return undefined;
}
}
function truncateDiagnosticString(value: string): string {
const maxLength = 8192;
return value.length > maxLength ? `${value.slice(0, maxLength)}…` : value;
}
function formatPiMessagesResponseError(
response: Response,
body: string,
errorBody: PiMessagesErrorBody | undefined,
): string {
const message = typeof errorBody?.error?.message === "string" ? errorBody.error.message : undefined;
const code = typeof errorBody?.error?.code === "string" ? errorBody.error.code : undefined;
const suffix = message ?? body;
const codeSuffix = code ? ` (${code})` : "";
return `${response.status} ${response.statusText}: ${suffix}${codeSuffix}`;
}
function createPiMessagesResponseError(
model: Model<"pi-messages">,
url: URL,
response: Response,
body: string,
): PiMessagesResponseError {
const errorBody = parsePiMessagesErrorBody(body);
const code = typeof errorBody?.error?.code === "string" ? errorBody.error.code : undefined;
return new PiMessagesResponseError(formatPiMessagesResponseError(response, body, errorBody), code, {
version: 1,
provider: model.provider,
model: model.id,
url: url.toString(),
status: response.status,
statusText: response.statusText,
error: errorBody?.error,
body: errorBody ? undefined : truncateDiagnosticString(body),
timestampMs: Date.now(),
});
}
function createEmptyUsage(): PiMessagesUsage {
return {
input: 0,
output: 0,
cacheRead: 0,
cacheWrite: 0,
totalTokens: 0,
cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0, total: 0 },
};
}
function appendRewriteDiagnostic(message: AssistantMessage, rewrite: PiMessagesRewriteImpact | undefined): void {
if (!rewrite) {
return;
}
appendAssistantMessageDiagnostic(message, {
type: "pi_messages_rewrite",
timestamp: Date.now(),
details: { ...rewrite },
});
}
function createEventConverter(model: Model<"pi-messages">) {
const partial: AssistantMessage = {
role: "assistant",
content: [],
api: model.api,
provider: model.provider,
model: model.id,
usage: createEmptyUsage(),
stopReason: "stop",
timestamp: Date.now(),
};
const toolJson = new Map<number, string>();
return (event: PiMessagesEvent): AssistantMessageEvent => {
switch (event.type) {
case "done":
Object.assign(partial, {
stopReason: event.reason,
usage: event.usage,
responseId: event.responseId,
});
appendRewriteDiagnostic(partial, event.rewrite);
return { type: "done", reason: event.reason, message: partial };
case "error":
Object.assign(partial, {
stopReason: event.reason,
usage: event.usage,
errorMessage: event.errorMessage,
responseId: event.responseId,
});
appendRewriteDiagnostic(partial, event.rewrite);
return { type: "error", reason: event.reason, error: partial };
case "start":
break;
case "text_start":
partial.content[event.contentIndex] = { type: "text", text: "" };
break;
case "text_delta":
(partial.content[event.contentIndex] as { text: string }).text += event.delta;
break;
case "text_end":
Object.assign(partial.content[event.contentIndex]!, {
text: event.content,
textSignature: event.contentSignature,
});
break;
case "thinking_start":
partial.content[event.contentIndex] = { type: "thinking", thinking: "" };
break;
case "thinking_delta":
(partial.content[event.contentIndex] as { thinking: string }).thinking += event.delta;
break;
case "thinking_end":
Object.assign(partial.content[event.contentIndex]!, {
thinking: event.content,
thinkingSignature: event.contentSignature,
redacted: event.redacted,
});
break;
case "toolcall_start":
partial.content[event.contentIndex] = {
type: "toolCall",
id: event.id,
name: event.toolName,
arguments: {},
};
toolJson.set(event.contentIndex, "");
break;
case "toolcall_delta": {
const json = `${toolJson.get(event.contentIndex) ?? ""}${event.delta}`;
toolJson.set(event.contentIndex, json);
(partial.content[event.contentIndex] as ToolCall).arguments =
parseStreamingJson<ToolCall["arguments"]>(json);
break;
}
case "toolcall_end":
Object.assign(partial.content[event.contentIndex]!, event.toolCall);
toolJson.delete(event.contentIndex);
return {
type: "toolcall_end",
contentIndex: event.contentIndex,
toolCall: partial.content[event.contentIndex] as ToolCall,
partial,
};
}
return { ...event, partial } as AssistantMessageEvent;
};
}
async function* readPiMessagesEvents(stream: ReadableStream<Uint8Array>): AsyncGenerator<PiMessagesEvent> {
const decoder = new TextDecoder();
const reader = stream.getReader();
let buffer = "";
try {
while (true) {
const { done, value } = await reader.read();
buffer += done ? decoder.decode() : decoder.decode(value, { stream: true });
buffer = buffer.replace(/\r\n/g, "\n");
let split = buffer.indexOf("\n\n");
while (split !== -1) {
const event = parsePiMessagesEvent(buffer.slice(0, split));
if (event) {
yield event;
}
buffer = buffer.slice(split + 2);
split = buffer.indexOf("\n\n");
}
if (done) {
break;
}
}
if (buffer.trim()) {
const event = parsePiMessagesEvent(buffer);
if (event) {
yield event;
}
}
} finally {
reader.releaseLock();
}
}
function parsePiMessagesEvent(raw: string): PiMessagesEvent | undefined {
const data = raw
.split("\n")
.find((line) => line.startsWith("data:"))
?.slice(5)
.trim();
return data && data !== "[DONE]" ? (JSON.parse(data) as PiMessagesEvent) : undefined;
}
function createErrorEvent(model: Model<"pi-messages">, error: unknown, aborted: boolean): AssistantMessageEvent {
const reason = aborted ? "aborted" : "error";
const assistantMessage: AssistantMessage = {
role: "assistant",
content: [],
api: model.api,
provider: model.provider,
model: model.id,
usage: createEmptyUsage(),
stopReason: reason,
errorMessage: error instanceof Error ? error.message : String(error),
timestamp: Date.now(),
};
if (!aborted && error instanceof PiMessagesResponseError) {
appendAssistantMessageDiagnostic(
assistantMessage,
createAssistantMessageDiagnostic("pi_messages_response_failure", error, error.diagnosticDetails),
);
}
return { type: "error", reason, error: assistantMessage };
}
function resolveCacheRetention(cacheRetention?: CacheRetention, env?: ProviderEnv): CacheRetention | undefined {
if (cacheRetention) {
return cacheRetention;
}
// Backend defaults apply when unset; only the legacy env opt-in is mapped.
return getProviderEnvValue("PI_CACHE_RETENTION", env) === "long" ? "long" : undefined;
}
export const stream: StreamFunction<"pi-messages", PiMessagesOptions> = (
model: Model<"pi-messages">,
context: Context,
options?: PiMessagesOptions,
): AssistantMessageEventStream => {
const eventStream = new AssistantMessageEventStream();
const convertEvent = createEventConverter(model);
void (async () => {
try {
const apiKey = options?.apiKey;
if (!apiKey) {
throw new Error(`No API key provided for provider "${model.provider}"`);
}
const url = new URL(`${model.baseUrl.replace(/\/+$/u, "")}/messages`);
if (options?.debug) {
url.searchParams.set("debug", "1");
}
let payload: unknown = {
model: model.id,
context,
options: {
temperature: options?.temperature,
maxTokens: options?.maxTokens,
reasoning: options?.reasoning,
cacheRetention: resolveCacheRetention(options?.cacheRetention, options?.env),
sessionId: options?.sessionId,
toolChoice: options?.toolChoice,
},
};
const nextPayload = await options?.onPayload?.(payload, model);
if (nextPayload !== undefined) {
payload = nextPayload;
}
const response = await fetch(url, {
method: "POST",
headers: {
authorization: `Bearer ${apiKey}`,
accept: "text/event-stream",
"content-type": "application/json",
...providerHeadersToRecord(options?.headers),
},
body: JSON.stringify(payload),
signal: options?.signal,
});
await options?.onResponse?.({ status: response.status, headers: headersToRecord(response.headers) }, model);
if (!response.ok) {
const body = await response.text();
throw createPiMessagesResponseError(model, url, response, body);
}
if (!response.body) {
throw new Error(`${model.provider} response has no body`);
}
for await (const piEvent of readPiMessagesEvents(response.body)) {
const event = convertEvent(piEvent);
eventStream.push(event);
if (event.type === "done" || event.type === "error") {
return;
}
}
throw new Error(`${model.provider} stream ended without a terminal event`);
} catch (error) {
eventStream.push(createErrorEvent(model, error, options?.signal?.aborted ?? false));
}
})();
return eventStream;
};
export const streamSimple: StreamFunction<"pi-messages", SimpleStreamOptions> = (
model: Model<"pi-messages">,
context: Context,
options?: SimpleStreamOptions,
): AssistantMessageEventStream => {
const extra = options as PiMessagesOptions | undefined;
return stream(model, context, {
...options,
reasoning: options?.reasoning,
toolChoice: extra?.toolChoice,
debug: extra?.debug,
});
};
packages/ai/src/auth/oauth/load.ts
+7
View File
@@ -19,3 +19,10 @@ export const loadOpenAICodexOAuth = async (): Promise<OAuthAuth> =>
export const loadGitHubCopilotOAuth = async (): Promise<OAuthAuth> =>
((await importOAuthModule("./github-copilot.ts")) as { githubCopilotOAuth: OAuthAuth }).githubCopilotOAuth;
export const loadRadiusOAuth = async (options: { name: string; gateway: string }): Promise<OAuthAuth> =>
(
(await importOAuthModule("./radius.ts")) as {
createRadiusOAuth: (input: { name: string; gateway: string }) => OAuthAuth;
}
).createRadiusOAuth(options);
packages/ai/src/auth/oauth/radius.ts
+410
View File
@@ -0,0 +1,410 @@
/**
* Radius gateway OAuth flow.
*
* Radius is a pi-messages gateway. OAuth endpoints are discovered from the
* gateway (`/v1/oauth`); model catalog loading is owned by the Radius provider.
*
* NOTE: This module uses node:http for the OAuth callback server.
* It is only intended for CLI use, not browser environments.
*/
// NEVER convert to top-level imports - breaks browser/Vite builds
let _http: typeof import("node:http") | null = null;
if (typeof process !== "undefined" && (process.versions?.node || process.versions?.bun)) {
import("node:http").then((m) => {
_http = m;
});
}
import { normalizeRadiusGatewayUrl } from "../../providers/radius-config.ts";
import type { AuthInteraction, OAuthAuth, OAuthCredential } from "../types.ts";
import { pollOAuthDeviceCodeFlow } from "./device-code.ts";
import { oauthErrorHtml, oauthSuccessHtml } from "./oauth-page.ts";
import { generatePKCE } from "./pkce.ts";
const CALLBACK_HOST = "127.0.0.1";
const CALLBACK_PORT = 1456;
const CALLBACK_PATH = "/oauth/callback";
const REDIRECT_URI = `http://${CALLBACK_HOST}:${CALLBACK_PORT}${CALLBACK_PATH}`;
const TOKEN_EXPIRY_SKEW_MS = 60_000;
const LOGIN_METHOD_BROWSER = "browser";
const LOGIN_METHOD_DEVICE_CODE = "device-code";
type RadiusOAuthConfig = {
issuer: string;
authorizationEndpoint: string;
tokenEndpoint: string;
deviceAuthorizationEndpoint: string;
deviceAuthorizationEventsEndpoint: string;
verificationEndpoint: string;
clientId: string;
scope: string;
deviceCodeGrantType: string;
};
type DeviceAuthorizationResponse = {
device_code: string;
user_code: string;
verification_uri?: string;
verification_uri_complete?: string;
expires_in: number;
interval?: number;
};
async function loadRadiusOAuthConfig(gateway: string): Promise<RadiusOAuthConfig> {
const response = await fetch(new URL("/v1/oauth", gateway), {
headers: { accept: "application/json" },
});
if (!response.ok) {
throw new Error(
`Could not load Radius OAuth config from ${gateway}: ${response.status} ${await response.text()}`,
);
}
return (await response.json()) as RadiusOAuthConfig;
}
class OAuthResponseError extends Error {
readonly status: number;
readonly oauthError?: string;
constructor(status: number, oauthError: string | undefined, description: string | undefined, message: string) {
const detail = oauthError
? description
? `${oauthError}: ${description}`
: oauthError
: description || String(status);
super(`${message}: ${detail}`);
this.status = status;
this.oauthError = oauthError;
}
}
async function readOAuthResponseError(response: Response, message: string): Promise<OAuthResponseError> {
const text = await response.text().catch(() => "");
let oauthError: string | undefined;
let description: string | undefined;
if (text) {
try {
const data = JSON.parse(text) as { error?: unknown; error_description?: unknown };
oauthError = typeof data.error === "string" ? data.error : undefined;
description = typeof data.error_description === "string" ? data.error_description : undefined;
} catch {
description = text;
}
}
return new OAuthResponseError(response.status, oauthError, description, message);
}
async function requestOAuthToken(
oauth: RadiusOAuthConfig,
body: URLSearchParams,
signal?: AbortSignal,
): Promise<OAuthCredential> {
let response: Response;
try {
response = await fetch(oauth.tokenEndpoint, {
method: "POST",
headers: { accept: "application/json", "content-type": "application/x-www-form-urlencoded" },
body,
signal,
});
} catch (error) {
if (signal?.aborted) {
throw new Error("Login cancelled");
}
throw error;
}
if (!response.ok) {
throw await readOAuthResponseError(response, "Radius OAuth token request failed");
}
const data = (await response.json()) as {
access_token: string;
refresh_token: string;
expires_in: number;
scope?: string;
};
return {
type: "oauth",
access: data.access_token,
refresh: data.refresh_token,
expires: Date.now() + data.expires_in * 1000 - TOKEN_EXPIRY_SKEW_MS,
scope: data.scope,
};
}
type OAuthCallbackServer = {
waitForCode(): Promise<string | null>;
close(): void;
};
function startOAuthCallbackServer(
expectedState: string,
signal: AbortSignal | undefined,
): Promise<OAuthCallbackServer> {
if (!_http) {
throw new Error("Radius OAuth is only available in Node.js environments");
}
let settle: (code: string | null) => void = () => {};
let settled = false;
const wait = new Promise<string | null>((resolve) => {
settle = resolve;
});
const finish = (code: string | null) => {
if (settled) {
return;
}
settled = true;
signal?.removeEventListener("abort", onAbort);
settle(code);
};
const onAbort = () => finish(null);
signal?.addEventListener("abort", onAbort, { once: true });
const sendPage = (response: import("node:http").ServerResponse, status: number, html: string) => {
response.statusCode = status;
response.setHeader("content-type", "text/html; charset=utf-8");
response.end(html);
};
const server = _http.createServer((request, response) => {
const url = new URL(request.url ?? "/", REDIRECT_URI);
if (url.pathname !== CALLBACK_PATH) {
sendPage(response, 404, oauthErrorHtml("Callback route not found."));
return;
}
if (url.searchParams.get("state") !== expectedState) {
sendPage(response, 400, oauthErrorHtml("OAuth state mismatch."));
return;
}
const error = url.searchParams.get("error");
if (error) {
sendPage(response, 400, oauthErrorHtml(url.searchParams.get("error_description") ?? error));
finish(null);
return;
}
const code = url.searchParams.get("code");
if (!code) {
sendPage(response, 400, oauthErrorHtml("Missing authorization code."));
return;
}
sendPage(response, 200, oauthSuccessHtml("Signed in to Radius. You may now close this page."));
finish(code);
});
return new Promise((resolve) => {
server
.listen(CALLBACK_PORT, CALLBACK_HOST, () => {
resolve({
waitForCode: () => wait,
close: () => {
finish(null);
server.close();
},
});
})
.once("error", () => {
finish(null);
resolve({ waitForCode: async () => null, close: () => {} });
});
});
}
async function loginWithBrowser(oauth: RadiusOAuthConfig, interaction: AuthInteraction): Promise<OAuthCredential> {
const { verifier, challenge } = await generatePKCE();
const state = crypto.randomUUID();
const authorizeUrl = new URL(oauth.authorizationEndpoint);
authorizeUrl.search = new URLSearchParams({
response_type: "code",
client_id: oauth.clientId,
redirect_uri: REDIRECT_URI,
scope: oauth.scope,
code_challenge: challenge,
code_challenge_method: "S256",
handoff: "url",
state,
}).toString();
const callbackServer = await startOAuthCallbackServer(state, interaction.signal);
interaction.notify({ type: "progress", message: `Listening for OAuth callback on ${REDIRECT_URI}` });
interaction.notify({
type: "auth_url",
url: authorizeUrl.toString(),
instructions: "Continue in your browser.",
});
try {
const code = await callbackServer.waitForCode();
if (!code) {
if (interaction.signal?.aborted) {
throw new Error("Login cancelled");
}
throw new Error("OAuth callback did not complete.");
}
return await requestOAuthToken(
oauth,
new URLSearchParams({
grant_type: "authorization_code",
client_id: oauth.clientId,
redirect_uri: REDIRECT_URI,
code,
code_verifier: verifier,
}),
interaction.signal,
);
} finally {
callbackServer.close();
}
}
async function requestDeviceAuthorization(
oauth: RadiusOAuthConfig,
signal: AbortSignal | undefined,
): Promise<DeviceAuthorizationResponse> {
let response: Response;
try {
response = await fetch(oauth.deviceAuthorizationEndpoint, {
method: "POST",
headers: { accept: "application/json", "content-type": "application/x-www-form-urlencoded" },
body: new URLSearchParams({ client_id: oauth.clientId, scope: oauth.scope }),
signal,
});
} catch (error) {
if (signal?.aborted) {
throw new Error("Login cancelled");
}
throw error;
}
if (!response.ok) {
throw await readOAuthResponseError(response, "Radius OAuth device authorization failed");
}
const data = (await response.json()) as Partial<DeviceAuthorizationResponse>;
if (!data.device_code || !data.user_code || !data.expires_in) {
throw new Error("Radius OAuth device authorization response is missing required fields");
}
return {
device_code: data.device_code,
user_code: data.user_code,
verification_uri: data.verification_uri,
verification_uri_complete: data.verification_uri_complete,
expires_in: data.expires_in,
interval: data.interval,
};
}
async function loginWithDeviceCode(oauth: RadiusOAuthConfig, interaction: AuthInteraction): Promise<OAuthCredential> {
const device = await requestDeviceAuthorization(oauth, interaction.signal);
interaction.notify({
type: "device_code",
userCode: device.user_code,
verificationUri: device.verification_uri || oauth.verificationEndpoint,
intervalSeconds: device.interval,
expiresInSeconds: device.expires_in,
});
return pollOAuthDeviceCodeFlow<OAuthCredential>({
intervalSeconds: device.interval,
expiresInSeconds: device.expires_in,
signal: interaction.signal,
poll: async () => {
try {
const credentials = await requestOAuthToken(
oauth,
new URLSearchParams({
grant_type: oauth.deviceCodeGrantType,
client_id: oauth.clientId,
device_code: device.device_code,
}),
interaction.signal,
);
return { status: "complete", value: credentials };
} catch (error) {
if (!(error instanceof OAuthResponseError)) {
throw error;
}
switch (error.oauthError) {
case "authorization_pending":
return { status: "pending" };
case "slow_down":
return { status: "slow_down" };
case "expired_token":
return { status: "failed", message: "Device authorization expired." };
case "access_denied":
return { status: "failed", message: "Device authorization was denied." };
default:
throw error;
}
}
},
});
}
export interface RadiusOAuthOptions {
name: string;
gateway: string;
}
export function createRadiusOAuth(options: RadiusOAuthOptions): OAuthAuth {
const gateway = normalizeRadiusGatewayUrl(options.gateway);
return {
name: options.name,
async login(interaction): Promise<OAuthCredential> {
const oauth = await loadRadiusOAuthConfig(gateway);
const loginMethod = await interaction.prompt({
type: "select",
message: `Sign in to ${options.name}:`,
options: [
{ id: LOGIN_METHOD_BROWSER, label: "Sign in with browser (recommended)" },
{
id: LOGIN_METHOD_DEVICE_CODE,
label: "Sign in with device code (when signing in from another device)",
},
],
});
let credential: OAuthCredential;
if (loginMethod === LOGIN_METHOD_DEVICE_CODE) {
credential = await loginWithDeviceCode(oauth, interaction);
} else if (loginMethod === LOGIN_METHOD_BROWSER) {
credential = await loginWithBrowser(oauth, interaction);
} else {
throw new Error(`Unknown ${options.name} sign-in method: ${loginMethod}`);
}
return credential;
},
async refresh(credential, signal): Promise<OAuthCredential> {
const oauth = await loadRadiusOAuthConfig(gateway);
const refreshed = await requestOAuthToken(
oauth,
new URLSearchParams({
grant_type: "refresh_token",
client_id: oauth.clientId,
refresh_token: credential.refresh,
}),
signal,
);
return refreshed;
},
async toAuth(credential) {
return { apiKey: credential.access };
},
};
}
packages/ai/src/auth/types.ts
+1 -1
View File
@@ -196,7 +196,7 @@ export interface OAuthAuth {
* Exchange the refresh token. Network call; throws on failure
* (invalid_grant etc.). `Models` runs this under the store lock.
*/
refresh(credential: OAuthCredential): Promise<OAuthCredential>;
refresh(credential: OAuthCredential, signal?: AbortSignal): Promise<OAuthCredential>;
/**
* Side-effect-free derivation of request auth from a valid credential.
packages/ai/src/compat.ts
+8 -1
View File
@@ -19,6 +19,7 @@ export * from "./api/mistral-conversations.lazy.ts";
export * from "./api/openai-codex-responses.lazy.ts";
export * from "./api/openai-completions.lazy.ts";
export * from "./api/openai-responses.lazy.ts";
export * from "./api/pi-messages.lazy.ts";
export * from "./env-api-keys.ts";
export * from "./image-models.ts";
export * from "./images.ts";
@@ -36,9 +37,13 @@ import { mistralConversationsApi } from "./api/mistral-conversations.lazy.ts";
import { openAICodexResponsesApi } from "./api/openai-codex-responses.lazy.ts";
import { openAICompletionsApi } from "./api/openai-completions.lazy.ts";
import { openAIResponsesApi } from "./api/openai-responses.lazy.ts";
import { piMessagesApi } from "./api/pi-messages.lazy.ts";
import { getEnvApiKey } from "./env-api-keys.ts";
import type { ModelsApiStreamOptions } from "./models.ts";
import { builtinModels, getBuiltinModel, getBuiltinModels, getBuiltinProviders } from "./providers/all.ts";
export type { BuiltinProvider } from "./providers/all.ts";
import { createFauxCore, type FauxProviderRegistration, type RegisterFauxProviderOptions } from "./providers/faux.ts";
import type {
Api,
@@ -180,6 +185,7 @@ const BUILTIN_APIS: [Api, ProviderStreams][] = [
["google-vertex", googleVertexApi()],
["mistral-conversations", mistralConversationsApi()],
["bedrock-converse-stream", bedrockConverseStreamApi()],
["pi-messages", piMessagesApi()],
];
const builtinApiProviderInstances = new Map<Api, ReturnType<typeof getApiProvider>>();
@@ -207,6 +213,7 @@ export function resetApiProviders(): void {
registerBuiltInApiProviders();
const compatModels = builtinModels();
const AMBIENT_AUTH_MARKER = "<authenticated>";
function hasExplicitApiKey(apiKey: string | undefined): apiKey is string {
return typeof apiKey === "string" && apiKey.trim().length > 0;
@@ -218,7 +225,7 @@ function withEnvApiKey<TOptions extends StreamOptions>(
): TOptions | undefined {
if (hasExplicitApiKey(options?.apiKey)) return options;
const apiKey = getEnvApiKey(model.provider, options?.env);
if (!apiKey) return options;
if (!apiKey || apiKey === AMBIENT_AUTH_MARKER) return options;
return { ...options, apiKey } as TOptions;
}
packages/ai/src/env-api-keys.ts
+1
View File
@@ -82,6 +82,7 @@ function getApiKeyEnvVars(provider: string): readonly string[] | undefined {
groq: "GROQ_API_KEY",
cerebras: "CEREBRAS_API_KEY",
xai: "XAI_API_KEY",
radius: "RADIUS_API_KEY",
openrouter: "OPENROUTER_API_KEY",
"vercel-ai-gateway": "AI_GATEWAY_API_KEY",
zai: "ZAI_API_KEY",
packages/ai/src/index.ts
+2
View File
@@ -17,6 +17,7 @@ export type { MistralOptions } from "./api/mistral-conversations.ts";
export type { OpenAICodexResponsesOptions, OpenAICodexWebSocketDebugStats } from "./api/openai-codex-responses.ts";
export type { OpenAICompletionsOptions } from "./api/openai-completions.ts";
export type { OpenAIResponsesOptions } from "./api/openai-responses.ts";
export type { PiMessagesEvent, PiMessagesOptions, PiMessagesRewriteImpact } from "./api/pi-messages.ts";
export * from "./auth/context.ts";
export * from "./auth/credential-store.ts";
export * from "./auth/helpers.ts";
@@ -31,6 +32,7 @@ export type {
} from "./compat/extension-oauth-types.ts";
export * from "./images-models.ts";
export * from "./models.ts";
export * from "./models-store.ts";
export * from "./providers/faux.ts";
export * from "./session-resources.ts";
export * from "./types.ts";
packages/ai/src/models-store.ts
+35
View File
@@ -0,0 +1,35 @@
import type { Api, Model } from "./types.ts";
/** Persistent model catalogs keyed by provider ID. */
export interface ModelsStore {
read(providerId: string): Promise<readonly Model<Api>[] | undefined>;
write(providerId: string, models: readonly Model<Api>[]): Promise<void>;
delete(providerId: string): Promise<void>;
}
/** ModelsStore scoped to one provider. Providers cannot access other providers' catalogs. */
export interface ProviderModelsStore {
read(): Promise<readonly Model<Api>[] | undefined>;
write(models: readonly Model<Api>[]): Promise<void>;
delete(): Promise<void>;
}
export class InMemoryModelsStore implements ModelsStore {
private readonly models = new Map<string, readonly Model<Api>[]>();
async read(providerId: string): Promise<readonly Model<Api>[] | undefined> {
const models = this.models.get(providerId);
return models?.map((model) => structuredClone(model));
}
async write(providerId: string, models: readonly Model<Api>[]): Promise<void> {
this.models.set(
providerId,
models.map((model) => structuredClone(model)),
);
}
async delete(providerId: string): Promise<void> {
this.models.delete(providerId);
}
}
packages/ai/src/models.ts
+130 -40
View File
@@ -12,6 +12,7 @@ import type {
CredentialStore,
ProviderAuth,
} from "./auth/types.ts";
import { InMemoryModelsStore, type ModelsStore, type ProviderModelsStore } from "./models-store.ts";
import type {
Api,
ApiStreamOptions,
@@ -30,6 +31,26 @@ import type {
export { ModelsError, type ModelsErrorCode } from "./auth/resolve.ts";
export interface RefreshModelsContext {
/** Effective configured credential. OAuth credentials are refreshed before network access. */
credential?: Credential;
/** Persistent model storage scoped to this provider ID. */
store: ProviderModelsStore;
/** False during offline/cache-only initialization. */
allowNetwork: boolean;
signal?: AbortSignal;
}
export interface ModelsRefreshOptions {
allowNetwork?: boolean;
signal?: AbortSignal;
}
export interface ModelsRefreshResult {
aborted: boolean;
errors: ReadonlyMap<string, Error>;
}
export interface ModelsStreamTransforms {
/** Transform fully assembled model/auth/request headers before provider dispatch. */
transformHeaders?: (headers: ProviderHeaders) => ProviderHeaders | Promise<ProviderHeaders>;
@@ -72,13 +93,11 @@ export interface Provider<TApi extends Api = Api> {
getModels(): readonly Model<TApi>[];
/**
* Dynamic providers only: fetch and update the model list. Side-effect-free
* discovery (no loading/downloading); provider-specific model lifecycle
* belongs in app commands. Concurrent calls share one in-flight fetch.
* May reject (network); on rejection the model list stays at its last-known
* state and a later call retries.
* Dynamic providers only: restore the provider-scoped stored catalog and optionally fetch
* a newer list using the effective credential. Implementations must retain their previous
* list on failure and honor the shared abort signal for network requests.
*/
refreshModels?(): Promise<void>;
refreshModels?(context: RefreshModelsContext): Promise<void>;
/**
* Optional provider policy for credential-specific model availability.
@@ -118,12 +137,10 @@ export interface Models {
getModel(provider: string, id: string): Model<Api> | undefined;
/**
* Ask dynamic providers to re-fetch their model lists. With a provider id,
* rejects with `ModelsError` ("model_source") on that provider's fetch
* failure; without one, refreshes all providers concurrently best-effort.
* Static providers (no `refreshModels`) are no-ops.
* Refresh every configured dynamic provider concurrently. Provider errors and cancellation
* are returned without rejecting; static and unconfigured providers are skipped.
*/
refresh(provider?: string): Promise<void>;
refresh(options?: ModelsRefreshOptions): Promise<ModelsRefreshResult>;
/** Check whether a provider has complete auth configuration without refreshing OAuth. */
checkAuth(providerId: string): Promise<AuthCheck | undefined>;
@@ -174,6 +191,7 @@ export interface MutableModels extends Models {
export interface CreateModelsOptions {
credentials?: CredentialStore;
modelsStore?: ModelsStore;
authContext?: AuthContext;
}
@@ -196,10 +214,12 @@ function mergeHeaders(
class ModelsImpl implements MutableModels {
private providers = new Map<string, Provider>();
private credentials: CredentialStore;
private modelsStore: ModelsStore;
private authContext: AuthContext;
constructor(options?: CreateModelsOptions) {
this.credentials = options?.credentials ?? new InMemoryCredentialStore();
this.modelsStore = options?.modelsStore ?? new InMemoryModelsStore();
this.authContext = options?.authContext ?? defaultAuthContext();
}
@@ -249,22 +269,78 @@ class ModelsImpl implements MutableModels {
return this.getModels(provider).find((model) => model.id === id);
}
async refresh(provider?: string): Promise<void> {
if (provider !== undefined) {
const entry = this.providers.get(provider);
if (!entry?.refreshModels) return;
try {
await entry.refreshModels();
} catch (error) {
if (error instanceof ModelsError) throw error;
throw new ModelsError("model_source", `Model refresh failed for ${provider}`, { cause: error });
}
return;
async refresh(options: ModelsRefreshOptions = {}): Promise<ModelsRefreshResult> {
const allowNetwork = options.allowNetwork ?? true;
const errors = new Map<string, Error>();
const refreshable = Array.from(this.providers.values()).filter(
(provider): provider is Provider & Required<Pick<Provider, "refreshModels">> =>
provider.refreshModels !== undefined,
);
await Promise.all(
refreshable.map(async (provider) => {
if (options.signal?.aborted) return;
const store: ProviderModelsStore = {
read: () => this.modelsStore.read(provider.id),
write: (models) => this.modelsStore.write(provider.id, models),
delete: () => this.modelsStore.delete(provider.id),
};
let stored: Credential | undefined;
try {
stored = await this.readCredential(provider.id);
const credential = await this.resolveRefreshCredential(provider, stored, allowNetwork, options.signal);
if (!credential) return;
await provider.refreshModels({ credential, store, allowNetwork, signal: options.signal });
} catch (error) {
if (!options.signal?.aborted) {
errors.set(
provider.id,
error instanceof Error
? error
: new ModelsError("model_source", `Model refresh failed for ${provider.id}`, { cause: error }),
);
}
try {
await provider.refreshModels({
credential: stored,
store,
allowNetwork: false,
signal: options.signal,
});
} catch {
// Preserve the original auth/network error; cache restoration is best-effort here.
}
}
}),
);
return { aborted: options.signal?.aborted ?? false, errors };
}
private async resolveRefreshCredential(
provider: Provider,
stored: Credential | undefined,
allowNetwork: boolean,
signal?: AbortSignal,
): Promise<Credential | undefined> {
if (stored?.type === "oauth") {
const oauth = provider.auth.oauth;
if (!oauth) return undefined;
if (!allowNetwork || Date.now() < stored.expires) return stored;
if (signal?.aborted) return undefined;
const post = await this.credentials.modify(provider.id, async (current) => {
if (current?.type !== "oauth" || Date.now() < current.expires) return undefined;
return oauth.refresh(current, signal);
});
return post?.type === "oauth" ? post : undefined;
}
// Cannot reject: the async mapper turns even sync throws from ill-behaved
// providers into rejections, and allSettled captures all of them.
await Promise.allSettled(Array.from(this.providers.values(), async (entry) => entry.refreshModels?.()));
const apiKey = provider.auth.apiKey;
if (!apiKey) return undefined;
const credential = stored?.type === "api_key" ? stored : undefined;
const result = await apiKey.resolve({ ctx: this.authContext, credential });
if (!result) return undefined;
return { type: "api_key", key: result.auth.apiKey, env: result.env };
}
private async readCredential(providerId: string): Promise<Credential | undefined> {
@@ -452,16 +528,10 @@ export interface CreateProviderOptions<TApi extends Api = Api> {
headers?: ProviderHeaders;
/** Required — every provider has auth semantics, even ambient/keyless ones. */
auth: ProviderAuth;
/** Initial model list (empty for purely dynamic providers). */
/** Static baseline model list (empty for purely dynamic providers). */
models: readonly Model<TApi>[];
/**
* Dynamic providers: fetch the current list. Stored on success; concurrent
* calls share one in-flight fetch. May reject: the stored list then stays
* at its last-known state, the rejection propagates to the caller of
* `refreshModels()` (wrapped as ModelsError "model_source" by
* `Models.refresh(provider)`), and a later call retries.
*/
refreshModels?: () => Promise<readonly Model<TApi>[]>;
/** Fetch a dynamic model overlay. createProvider restores/persists it through ModelsStore. */
fetchModels?: (context: RefreshModelsContext) => Promise<readonly Model<TApi>[]>;
filterModels?: (models: readonly Model<TApi>[], credential: Credential | undefined) => readonly Model<TApi>[];
/** Single implementation, or map keyed by `model.api` for mixed-API providers. */
api: ProviderStreams | Partial<Record<TApi, ProviderStreams>>;
@@ -474,9 +544,19 @@ export interface CreateProviderOptions<TApi extends Api = Api> {
* produces a stream error.
*/
export function createProvider<TApi extends Api = Api>(input: CreateProviderOptions<TApi>): Provider<TApi> {
let models = input.models;
const baselineModels = input.models;
let dynamicModels: readonly Model<TApi>[] = [];
let inflightRefresh: Promise<void> | undefined;
const refreshModels = input.refreshModels;
const fetchModels = input.fetchModels;
const currentModels = (): readonly Model<TApi>[] => {
const merged = [...baselineModels];
for (const model of dynamicModels) {
const index = merged.findIndex((entry) => entry.id === model.id);
if (index >= 0) merged[index] = model;
else merged.push(model);
}
return merged;
};
const single =
typeof (input.api as ProviderStreams).stream === "function" ? (input.api as ProviderStreams) : undefined;
const byApi = single ? undefined : (input.api as Partial<Record<string, ProviderStreams>>);
@@ -502,12 +582,22 @@ export function createProvider<TApi extends Api = Api>(input: CreateProviderOpti
baseUrl: input.baseUrl,
headers: input.headers,
auth: input.auth,
getModels: () => models,
refreshModels: refreshModels
? () => {
getModels: currentModels,
refreshModels: fetchModels
? (context) => {
inflightRefresh ??= (async () => {
try {
models = await refreshModels();
const stored = await context.store.read();
if (stored) {
dynamicModels = stored
.filter((model) => model.provider === input.id)
.map((model) => model as Model<TApi>);
}
if (!context.allowNetwork || context.signal?.aborted) return;
const refreshed = await fetchModels(context);
if (context.signal?.aborted) return;
dynamicModels = refreshed;
await context.store.write(refreshed);
} finally {
inflightRefresh = undefined;
}
packages/ai/src/providers/all.ts
+15 -6
View File
@@ -1,7 +1,7 @@
import { createImagesModels, type ImagesProvider, type MutableImagesModels } from "../images-models.ts";
import { MODELS } from "../models.generated.ts";
import { type CreateModelsOptions, createModels, type MutableModels, type Provider } from "../models.ts";
import type { Api, KnownProvider, Model } from "../types.ts";
import type { Api, Model } from "../types.ts";
import { amazonBedrockProvider } from "./amazon-bedrock.ts";
import { antLingProvider } from "./ant-ling.ts";
import { anthropicProvider } from "./anthropic.ts";
@@ -29,6 +29,7 @@ import { opencodeProvider } from "./opencode.ts";
import { opencodeGoProvider } from "./opencode-go.ts";
import { openrouterProvider } from "./openrouter.ts";
import { openrouterImagesProvider } from "./openrouter-images.ts";
import { radiusProvider } from "./radius.ts";
import { togetherProvider } from "./together.ts";
import { vercelAIGatewayProvider } from "./vercel-ai-gateway.ts";
import { xaiProvider } from "./xai.ts";
@@ -39,13 +40,20 @@ import { xiaomiTokenPlanSgpProvider } from "./xiaomi-token-plan-sgp.ts";
import { zaiProvider } from "./zai.ts";
import { zaiCodingCnProvider } from "./zai-coding-cn.ts";
export { radiusProvider };
/** Providers present in the generated catalog. `KnownProvider` additionally
* includes purely dynamic providers (e.g. "radius") that have no static
* catalog entry. */
export type BuiltinProvider = keyof typeof MODELS;
type BuiltinModelApi<
TProvider extends KnownProvider,
TProvider extends BuiltinProvider,
TModelId extends keyof (typeof MODELS)[TProvider],
> = (typeof MODELS)[TProvider][TModelId] extends { api: infer TApi } ? (TApi extends Api ? TApi : never) : never;
/** Typed read of the generated built-in catalog. */
export function getBuiltinModel<TProvider extends KnownProvider, TModelId extends keyof (typeof MODELS)[TProvider]>(
export function getBuiltinModel<TProvider extends BuiltinProvider, TModelId extends keyof (typeof MODELS)[TProvider]>(
provider: TProvider,
modelId: TModelId,
): Model<BuiltinModelApi<TProvider, TModelId>> {
@@ -53,11 +61,11 @@ export function getBuiltinModel<TProvider extends KnownProvider, TModelId extend
return models?.[modelId as string] as Model<BuiltinModelApi<TProvider, TModelId>>;
}
export function getBuiltinProviders(): KnownProvider[] {
return Object.keys(MODELS) as KnownProvider[];
export function getBuiltinProviders(): BuiltinProvider[] {
return Object.keys(MODELS) as BuiltinProvider[];
}
export function getBuiltinModels<TProvider extends KnownProvider>(
export function getBuiltinModels<TProvider extends BuiltinProvider>(
provider: TProvider,
): Model<BuiltinModelApi<TProvider, keyof (typeof MODELS)[TProvider]>>[] {
const models = MODELS[provider] as Record<string, Model<Api>> | undefined;
@@ -95,6 +103,7 @@ export function builtinProviders(): Provider[] {
opencodeProvider(),
opencodeGoProvider(),
openrouterProvider(),
radiusProvider(),
togetherProvider(),
vercelAIGatewayProvider(),
xaiProvider(),
packages/ai/src/providers/cloudflare-auth.ts
+9 -5
View File
@@ -12,11 +12,15 @@ async function resolveValue(
ctx: AuthContext,
credential: ApiKeyCredential | undefined,
): Promise<string | undefined> {
if (credential) {
if (name === CLOUDFLARE_API_KEY) return credential.key;
return credential.env?.[name];
}
return ctx.env(name);
// Per-field merge: prefer the credential value, fall back to ambient env.
// A credential carrying only the API key must still pick up the account /
// gateway id from the environment.
const fromCredential = credential
? name === CLOUDFLARE_API_KEY
? credential.key
: credential.env?.[name]
: undefined;
return fromCredential ?? (await ctx.env(name));
}
async function resolveCloudflareEnv(
packages/ai/src/providers/github-copilot.models.ts
+3 -3
View File
@@ -13,6 +13,7 @@ export const GITHUB_COPILOT_MODELS = {
headers: {"User-Agent":"GitHubCopilotChat/0.35.0","Editor-Version":"vscode/1.107.0","Editor-Plugin-Version":"copilot-chat/0.35.0","Copilot-Integration-Id":"vscode-chat"},
compat: {"supportsStore":false,"supportsDeveloperRole":false,"supportsReasoningEffort":false},
reasoning: true,
thinkingLevelMap: {"off":null,"xhigh":"xhigh","max":"max"},
input: ["text", "image"],
cost: {
input: 10,
@@ -524,11 +525,10 @@ export const GITHUB_COPILOT_MODELS = {
"mai-code-1-flash-picker": {
id: "mai-code-1-flash-picker",
name: "MAI-Code-1-Flash",
api: "openai-completions",
api: "openai-responses",
provider: "github-copilot",
baseUrl: "https://api.individual.githubcopilot.com",
headers: {"User-Agent":"GitHubCopilotChat/0.35.0","Editor-Version":"vscode/1.107.0","Editor-Plugin-Version":"copilot-chat/0.35.0","Copilot-Integration-Id":"vscode-chat"},
compat: {"supportsStore":false,"supportsDeveloperRole":false,"supportsReasoningEffort":false},
reasoning: true,
input: ["text"],
cost: {
@@ -539,5 +539,5 @@ export const GITHUB_COPILOT_MODELS = {
},
contextWindow: 256000,
maxTokens: 128000,
} satisfies Model<"openai-completions">,
} satisfies Model<"openai-responses">,
} as const;
packages/ai/src/providers/openai-codex.models.ts
+6
View File
@@ -28,6 +28,7 @@ export const OPENAI_CODEX_MODELS = {
api: "openai-codex-responses",
provider: "openai-codex",
baseUrl: "https://chatgpt.com/backend-api",
compat: {"supportsToolSearch":true},
reasoning: true,
thinkingLevelMap: {"xhigh":"xhigh","minimal":"low"},
input: ["text", "image"],
@@ -47,6 +48,7 @@ export const OPENAI_CODEX_MODELS = {
api: "openai-codex-responses",
provider: "openai-codex",
baseUrl: "https://chatgpt.com/backend-api",
compat: {"supportsToolSearch":true},
reasoning: true,
thinkingLevelMap: {"xhigh":"xhigh","minimal":"low"},
input: ["text", "image"],
@@ -65,6 +67,7 @@ export const OPENAI_CODEX_MODELS = {
api: "openai-codex-responses",
provider: "openai-codex",
baseUrl: "https://chatgpt.com/backend-api",
compat: {"supportsToolSearch":true},
reasoning: true,
thinkingLevelMap: {"xhigh":"xhigh","minimal":"low"},
input: ["text", "image"],
@@ -84,6 +87,7 @@ export const OPENAI_CODEX_MODELS = {
api: "openai-codex-responses",
provider: "openai-codex",
baseUrl: "https://chatgpt.com/backend-api",
compat: {"supportsToolSearch":true},
reasoning: true,
thinkingLevelMap: {"xhigh":"xhigh","max":"max","minimal":"low"},
input: ["text", "image"],
@@ -103,6 +107,7 @@ export const OPENAI_CODEX_MODELS = {
api: "openai-codex-responses",
provider: "openai-codex",
baseUrl: "https://chatgpt.com/backend-api",
compat: {"supportsToolSearch":true},
reasoning: true,
thinkingLevelMap: {"xhigh":"xhigh","max":"max","minimal":"low"},
input: ["text", "image"],
@@ -122,6 +127,7 @@ export const OPENAI_CODEX_MODELS = {
api: "openai-codex-responses",
provider: "openai-codex",
baseUrl: "https://chatgpt.com/backend-api",
compat: {"supportsToolSearch":true},
reasoning: true,
thinkingLevelMap: {"xhigh":"xhigh","max":"max","minimal":"low"},
input: ["text", "image"],
packages/ai/src/providers/openai.models.ts
+7
View File
@@ -504,6 +504,7 @@ export const OPENAI_MODELS = {
api: "openai-responses",
provider: "openai",
baseUrl: "https://api.openai.com/v1",
compat: {"supportsToolSearch":true},
reasoning: true,
thinkingLevelMap: {"off":"none","xhigh":"xhigh"},
input: ["text", "image"],
@@ -523,6 +524,7 @@ export const OPENAI_MODELS = {
api: "openai-responses",
provider: "openai",
baseUrl: "https://api.openai.com/v1",
compat: {"supportsToolSearch":true},
reasoning: true,
thinkingLevelMap: {"off":"none","xhigh":"xhigh"},
input: ["text", "image"],
@@ -559,6 +561,7 @@ export const OPENAI_MODELS = {
api: "openai-responses",
provider: "openai",
baseUrl: "https://api.openai.com/v1",
compat: {"supportsToolSearch":true},
reasoning: true,
thinkingLevelMap: {"off":null,"xhigh":"xhigh"},
input: ["text", "image"],
@@ -578,6 +581,7 @@ export const OPENAI_MODELS = {
api: "openai-responses",
provider: "openai",
baseUrl: "https://api.openai.com/v1",
compat: {"supportsToolSearch":true},
reasoning: true,
thinkingLevelMap: {"off":"none","xhigh":"xhigh","minimal":null},
input: ["text", "image"],
@@ -616,6 +620,7 @@ export const OPENAI_MODELS = {
api: "openai-responses",
provider: "openai",
baseUrl: "https://api.openai.com/v1",
compat: {"supportsToolSearch":true},
reasoning: true,
thinkingLevelMap: {"off":"none","xhigh":"xhigh","max":"max"},
input: ["text", "image"],
@@ -635,6 +640,7 @@ export const OPENAI_MODELS = {
api: "openai-responses",
provider: "openai",
baseUrl: "https://api.openai.com/v1",
compat: {"supportsToolSearch":true},
reasoning: true,
thinkingLevelMap: {"off":"none","xhigh":"xhigh","max":"max"},
input: ["text", "image"],
@@ -654,6 +660,7 @@ export const OPENAI_MODELS = {
api: "openai-responses",
provider: "openai",
baseUrl: "https://api.openai.com/v1",
compat: {"supportsToolSearch":true},
reasoning: true,
thinkingLevelMap: {"off":"none","xhigh":"xhigh","max":"max"},
input: ["text", "image"],
packages/ai/src/providers/opencode.models.ts
+19
View File
@@ -392,6 +392,7 @@ export const OPENCODE_MODELS = {
api: "openai-responses",
provider: "opencode",
baseUrl: "https://opencode.ai/zen/v1",
compat: {"sessionAffinityFormat":"openai-nosession"},
reasoning: true,
thinkingLevelMap: {"off":null},
input: ["text", "image"],
@@ -410,6 +411,7 @@ export const OPENCODE_MODELS = {
api: "openai-responses",
provider: "opencode",
baseUrl: "https://opencode.ai/zen/v1",
compat: {"sessionAffinityFormat":"openai-nosession"},
reasoning: true,
thinkingLevelMap: {"off":null},
input: ["text", "image"],
@@ -428,6 +430,7 @@ export const OPENCODE_MODELS = {
api: "openai-responses",
provider: "opencode",
baseUrl: "https://opencode.ai/zen/v1",
compat: {"sessionAffinityFormat":"openai-nosession"},
reasoning: true,
thinkingLevelMap: {"off":null},
input: ["text", "image"],
@@ -446,6 +449,7 @@ export const OPENCODE_MODELS = {
api: "openai-responses",
provider: "opencode",
baseUrl: "https://opencode.ai/zen/v1",
compat: {"sessionAffinityFormat":"openai-nosession"},
reasoning: true,
thinkingLevelMap: {"off":null},
input: ["text", "image"],
@@ -464,6 +468,7 @@ export const OPENCODE_MODELS = {
api: "openai-responses",
provider: "opencode",
baseUrl: "https://opencode.ai/zen/v1",
compat: {"sessionAffinityFormat":"openai-nosession"},
reasoning: true,
thinkingLevelMap: {"off":null},
input: ["text", "image"],
@@ -482,6 +487,7 @@ export const OPENCODE_MODELS = {
api: "openai-responses",
provider: "opencode",
baseUrl: "https://opencode.ai/zen/v1",
compat: {"sessionAffinityFormat":"openai-nosession"},
reasoning: true,
thinkingLevelMap: {"off":null},
input: ["text", "image"],
@@ -500,6 +506,7 @@ export const OPENCODE_MODELS = {
api: "openai-responses",
provider: "opencode",
baseUrl: "https://opencode.ai/zen/v1",
compat: {"sessionAffinityFormat":"openai-nosession"},
reasoning: true,
thinkingLevelMap: {"off":null},
input: ["text", "image"],
@@ -518,6 +525,7 @@ export const OPENCODE_MODELS = {
api: "openai-responses",
provider: "opencode",
baseUrl: "https://opencode.ai/zen/v1",
compat: {"sessionAffinityFormat":"openai-nosession"},
reasoning: true,
thinkingLevelMap: {"off":null,"xhigh":"xhigh"},
input: ["text", "image"],
@@ -536,6 +544,7 @@ export const OPENCODE_MODELS = {
api: "openai-responses",
provider: "opencode",
baseUrl: "https://opencode.ai/zen/v1",
compat: {"sessionAffinityFormat":"openai-nosession"},
reasoning: true,
thinkingLevelMap: {"off":null,"xhigh":"xhigh"},
input: ["text", "image"],
@@ -554,6 +563,7 @@ export const OPENCODE_MODELS = {
api: "openai-responses",
provider: "opencode",
baseUrl: "https://opencode.ai/zen/v1",
compat: {"sessionAffinityFormat":"openai-nosession"},
reasoning: true,
thinkingLevelMap: {"off":null,"xhigh":"xhigh"},
input: ["text", "image"],
@@ -572,6 +582,7 @@ export const OPENCODE_MODELS = {
api: "openai-responses",
provider: "opencode",
baseUrl: "https://opencode.ai/zen/v1",
compat: {"sessionAffinityFormat":"openai-nosession"},
reasoning: true,
thinkingLevelMap: {"off":null,"xhigh":"xhigh"},
input: ["text", "image"],
@@ -590,6 +601,7 @@ export const OPENCODE_MODELS = {
api: "openai-responses",
provider: "opencode",
baseUrl: "https://opencode.ai/zen/v1",
compat: {"sessionAffinityFormat":"openai-nosession"},
reasoning: true,
thinkingLevelMap: {"off":null,"xhigh":"xhigh"},
input: ["text", "image"],
@@ -608,6 +620,7 @@ export const OPENCODE_MODELS = {
api: "openai-responses",
provider: "opencode",
baseUrl: "https://opencode.ai/zen/v1",
compat: {"sessionAffinityFormat":"openai-nosession"},
reasoning: true,
thinkingLevelMap: {"off":null,"xhigh":"xhigh"},
input: ["text", "image"],
@@ -626,6 +639,7 @@ export const OPENCODE_MODELS = {
api: "openai-responses",
provider: "opencode",
baseUrl: "https://opencode.ai/zen/v1",
compat: {"sessionAffinityFormat":"openai-nosession"},
reasoning: true,
thinkingLevelMap: {"off":null,"xhigh":"xhigh"},
input: ["text", "image"],
@@ -644,6 +658,7 @@ export const OPENCODE_MODELS = {
api: "openai-responses",
provider: "opencode",
baseUrl: "https://opencode.ai/zen/v1",
compat: {"sessionAffinityFormat":"openai-nosession"},
reasoning: true,
thinkingLevelMap: {"off":null,"xhigh":"xhigh"},
input: ["text", "image"],
@@ -662,6 +677,7 @@ export const OPENCODE_MODELS = {
api: "openai-responses",
provider: "opencode",
baseUrl: "https://opencode.ai/zen/v1",
compat: {"sessionAffinityFormat":"openai-nosession"},
reasoning: true,
thinkingLevelMap: {"off":null,"xhigh":"xhigh","minimal":null,"low":null},
input: ["text", "image"],
@@ -680,6 +696,7 @@ export const OPENCODE_MODELS = {
api: "openai-responses",
provider: "opencode",
baseUrl: "https://opencode.ai/zen/v1",
compat: {"sessionAffinityFormat":"openai-nosession"},
reasoning: true,
thinkingLevelMap: {"off":null,"xhigh":"xhigh","max":"max"},
input: ["text", "image"],
@@ -698,6 +715,7 @@ export const OPENCODE_MODELS = {
api: "openai-responses",
provider: "opencode",
baseUrl: "https://opencode.ai/zen/v1",
compat: {"sessionAffinityFormat":"openai-nosession"},
reasoning: true,
thinkingLevelMap: {"off":null,"xhigh":"xhigh","max":"max"},
input: ["text", "image"],
@@ -716,6 +734,7 @@ export const OPENCODE_MODELS = {
api: "openai-responses",
provider: "opencode",
baseUrl: "https://opencode.ai/zen/v1",
compat: {"sessionAffinityFormat":"openai-nosession"},
reasoning: true,
thinkingLevelMap: {"off":null,"xhigh":"xhigh","max":"max"},
input: ["text", "image"],
packages/ai/src/providers/openrouter.models.ts
+107 -88
View File
@@ -192,6 +192,7 @@ export const OPENROUTER_MODELS = {
baseUrl: "https://openrouter.ai/api/v1",
compat: {"thinkingFormat":"openrouter","cacheControlFormat":"anthropic"},
reasoning: true,
thinkingLevelMap: {"off":null,"xhigh":"xhigh","max":"max"},
input: ["text", "image"],
cost: {
input: 10,
@@ -461,6 +462,24 @@ export const OPENROUTER_MODELS = {
contextWindow: 262144,
maxTokens: 80000,
} satisfies Model<"openai-completions">,
"arcee-ai/trinity-mini": {
id: "arcee-ai/trinity-mini",
name: "Arcee AI: Trinity Mini",
api: "openai-completions",
provider: "openrouter",
baseUrl: "https://openrouter.ai/api/v1",
compat: {"supportsDeveloperRole":false,"thinkingFormat":"openrouter"},
reasoning: true,
input: ["text"],
cost: {
input: 0.045,
output: 0.15,
cacheRead: 0,
cacheWrite: 0,
},
contextWindow: 131072,
maxTokens: 131072,
} satisfies Model<"openai-completions">,
"arcee-ai/virtuoso-large": {
id: "arcee-ai/virtuoso-large",
name: "Arcee AI: Virtuoso Large",
@@ -638,7 +657,7 @@ export const OPENROUTER_MODELS = {
cacheRead: 0,
cacheWrite: 0,
},
contextWindow: 131072,
contextWindow: 128000,
maxTokens: 16000,
} satisfies Model<"openai-completions">,
"deepseek/deepseek-chat-v3-0324": {
@@ -669,8 +688,8 @@ export const OPENROUTER_MODELS = {
reasoning: true,
input: ["text"],
cost: {
input: 0.25,
output: 0.95,
input: 0.21,
output: 0.79,
cacheRead: 0.13,
cacheWrite: 0,
},
@@ -692,7 +711,7 @@ export const OPENROUTER_MODELS = {
cacheRead: 0,
cacheWrite: 0,
},
contextWindow: 163840,
contextWindow: 64000,
maxTokens: 16000,
} satisfies Model<"openai-completions">,
"deepseek/deepseek-r1-0528": {
@@ -746,7 +765,7 @@ export const OPENROUTER_MODELS = {
cacheRead: 0.02145,
cacheWrite: 0,
},
contextWindow: 131072,
contextWindow: 128000,
maxTokens: 64000,
} satisfies Model<"openai-completions">,
"deepseek/deepseek-v3.2-exp": {
@@ -1000,7 +1019,7 @@ export const OPENROUTER_MODELS = {
cacheRead: 0.2,
cacheWrite: 0.375,
},
contextWindow: 1048756,
contextWindow: 1048576,
maxTokens: 65536,
} satisfies Model<"openai-completions">,
"google/gemini-3.5-flash": {
@@ -1090,7 +1109,7 @@ export const OPENROUTER_MODELS = {
cacheRead: 0,
cacheWrite: 0,
},
contextWindow: 262144,
contextWindow: 131072,
maxTokens: 32768,
} satisfies Model<"openai-completions">,
"google/gemma-4-31b-it": {
@@ -1103,13 +1122,13 @@ export const OPENROUTER_MODELS = {
reasoning: true,
input: ["text", "image"],
cost: {
input: 0.06,
input: 0.12,
output: 0.35,
cacheRead: 0,
cacheRead: 0.09,
cacheWrite: 0,
},
contextWindow: 262144,
maxTokens: 8192,
maxTokens: 262144,
} satisfies Model<"openai-completions">,
"google/gemma-4-31b-it:free": {
id: "google/gemma-4-31b-it:free",
@@ -1127,7 +1146,7 @@ export const OPENROUTER_MODELS = {
cacheWrite: 0,
},
contextWindow: 262144,
maxTokens: 32768,
maxTokens: 8192,
} satisfies Model<"openai-completions">,
"ibm-granite/granite-4.1-8b": {
id: "ibm-granite/granite-4.1-8b",
@@ -1220,24 +1239,6 @@ export const OPENROUTER_MODELS = {
contextWindow: 262144,
maxTokens: 65536,
} satisfies Model<"openai-completions">,
"kwaipilot/kat-coder-air-v2.5": {
id: "kwaipilot/kat-coder-air-v2.5",
name: "Kwaipilot: KAT-Coder-Air V2.5",
api: "openai-completions",
provider: "openrouter",
baseUrl: "https://openrouter.ai/api/v1",
compat: {"supportsDeveloperRole":false,"thinkingFormat":"openrouter"},
reasoning: false,
input: ["text"],
cost: {
input: 0.15,
output: 0.6,
cacheRead: 0.03,
cacheWrite: 0,
},
contextWindow: 256000,
maxTokens: 80000,
} satisfies Model<"openai-completions">,
"kwaipilot/kat-coder-pro-v2": {
id: "kwaipilot/kat-coder-pro-v2",
name: "Kwaipilot: KAT-Coder-Pro V2",
@@ -1256,23 +1257,23 @@ export const OPENROUTER_MODELS = {
contextWindow: 256000,
maxTokens: 80000,
} satisfies Model<"openai-completions">,
"kwaipilot/kat-coder-pro-v2.5": {
id: "kwaipilot/kat-coder-pro-v2.5",
name: "Kwaipilot: KAT-Coder-Pro V2.5",
"liquid/lfm-2.5-1.2b-thinking:free": {
id: "liquid/lfm-2.5-1.2b-thinking:free",
name: "LiquidAI: LFM2.5-1.2B-Thinking (free)",
api: "openai-completions",
provider: "openrouter",
baseUrl: "https://openrouter.ai/api/v1",
compat: {"supportsDeveloperRole":false,"thinkingFormat":"openrouter"},
reasoning: false,
reasoning: true,
input: ["text"],
cost: {
input: 0.74,
output: 2.96,
cacheRead: 0.15,
input: 0,
output: 0,
cacheRead: 0,
cacheWrite: 0,
},
contextWindow: 256000,
maxTokens: 80000,
contextWindow: 32768,
maxTokens: 4096,
} satisfies Model<"openai-completions">,
"meta-llama/llama-3.1-70b-instruct": {
id: "meta-llama/llama-3.1-70b-instruct",
@@ -1343,7 +1344,7 @@ export const OPENROUTER_MODELS = {
cacheRead: 0,
cacheWrite: 0,
},
contextWindow: 131072,
contextWindow: 65536,
maxTokens: 4096,
} satisfies Model<"openai-completions">,
"meta-llama/llama-4-maverick": {
@@ -1356,8 +1357,8 @@ export const OPENROUTER_MODELS = {
reasoning: false,
input: ["text", "image"],
cost: {
input: 0.2,
output: 0.8,
input: 0.15,
output: 0.6,
cacheRead: 0,
cacheWrite: 0,
},
@@ -1379,7 +1380,7 @@ export const OPENROUTER_MODELS = {
cacheRead: 0,
cacheWrite: 0,
},
contextWindow: 10000000,
contextWindow: 327680,
maxTokens: 16384,
} satisfies Model<"openai-completions">,
"minimax/minimax-m1": {
@@ -1451,7 +1452,7 @@ export const OPENROUTER_MODELS = {
cacheRead: 0.05,
cacheWrite: 0,
},
contextWindow: 204800,
contextWindow: 196608,
maxTokens: 196608,
} satisfies Model<"openai-completions">,
"minimax/minimax-m2.7": {
@@ -1469,7 +1470,7 @@ export const OPENROUTER_MODELS = {
cacheRead: 0,
cacheWrite: 0,
},
contextWindow: 204800,
contextWindow: 196608,
maxTokens: 196608,
} satisfies Model<"openai-completions">,
"minimax/minimax-m3": {
@@ -1487,7 +1488,7 @@ export const OPENROUTER_MODELS = {
cacheRead: 0.06,
cacheWrite: 0,
},
contextWindow: 1048576,
contextWindow: 1000000,
maxTokens: 131072,
} satisfies Model<"openai-completions">,
"mistralai/codestral-2508": {
@@ -1865,7 +1866,7 @@ export const OPENROUTER_MODELS = {
cacheRead: 0.07,
cacheWrite: 0,
},
contextWindow: 262144,
contextWindow: 256000,
maxTokens: 4096,
} satisfies Model<"openai-completions">,
"moonshotai/kimi-k2.6": {
@@ -1896,9 +1897,9 @@ export const OPENROUTER_MODELS = {
reasoning: true,
input: ["text", "image"],
cost: {
input: 0.719,
input: 0.72,
output: 3.49,
cacheRead: 0.149,
cacheRead: 0.159,
cacheWrite: 0,
},
contextWindow: 262144,
@@ -2027,7 +2028,7 @@ export const OPENROUTER_MODELS = {
cacheRead: 0,
cacheWrite: 0,
},
contextWindow: 1000000,
contextWindow: 262144,
maxTokens: 4096,
} satisfies Model<"openai-completions">,
"nvidia/nemotron-3-super-120b-a12b:free": {
@@ -2045,7 +2046,7 @@ export const OPENROUTER_MODELS = {
cacheRead: 0,
cacheWrite: 0,
},
contextWindow: 1000000,
contextWindow: 262144,
maxTokens: 262144,
} satisfies Model<"openai-completions">,
"nvidia/nemotron-3-ultra-550b-a55b": {
@@ -2063,7 +2064,7 @@ export const OPENROUTER_MODELS = {
cacheRead: 0.1,
cacheWrite: 0,
},
contextWindow: 1000000,
contextWindow: 262144,
maxTokens: 16384,
} satisfies Model<"openai-completions">,
"nvidia/nemotron-3-ultra-550b-a55b:free": {
@@ -2456,11 +2457,11 @@ export const OPENROUTER_MODELS = {
cost: {
input: 0.05,
output: 0.4,
cacheRead: 0.005,
cacheRead: 0.01,
cacheWrite: 0,
},
contextWindow: 400000,
maxTokens: 128000,
maxTokens: 4096,
} satisfies Model<"openai-completions">,
"openai/gpt-5-pro": {
id: "openai/gpt-5-pro",
@@ -2492,7 +2493,7 @@ export const OPENROUTER_MODELS = {
cost: {
input: 1.25,
output: 10,
cacheRead: 0.125,
cacheRead: 0.13,
cacheWrite: 0,
},
contextWindow: 400000,
@@ -2976,8 +2977,26 @@ export const OPENROUTER_MODELS = {
reasoning: true,
input: ["text"],
cost: {
input: 0.03,
output: 0.15,
input: 0.036,
output: 0.18,
cacheRead: 0,
cacheWrite: 0,
},
contextWindow: 131072,
maxTokens: 4096,
} satisfies Model<"openai-completions">,
"openai/gpt-oss-120b:free": {
id: "openai/gpt-oss-120b:free",
name: "OpenAI: gpt-oss-120b (free)",
api: "openai-completions",
provider: "openrouter",
baseUrl: "https://openrouter.ai/api/v1",
compat: {"thinkingFormat":"openrouter"},
reasoning: true,
input: ["text"],
cost: {
input: 0,
output: 0,
cacheRead: 0,
cacheWrite: 0,
},
@@ -3341,7 +3360,7 @@ export const OPENROUTER_MODELS = {
cacheRead: 0,
cacheWrite: 0,
},
contextWindow: 131072,
contextWindow: 32768,
maxTokens: 16384,
} satisfies Model<"openai-completions">,
"qwen/qwen-2.5-7b-instruct": {
@@ -3359,7 +3378,7 @@ export const OPENROUTER_MODELS = {
cacheRead: 0,
cacheWrite: 0,
},
contextWindow: 131072,
contextWindow: 32768,
maxTokens: 32768,
} satisfies Model<"openai-completions">,
"qwen/qwen-plus": {
@@ -3431,7 +3450,7 @@ export const OPENROUTER_MODELS = {
cacheRead: 0,
cacheWrite: 0,
},
contextWindow: 131702,
contextWindow: 40960,
maxTokens: 40960,
} satisfies Model<"openai-completions">,
"qwen/qwen3-235b-a22b": {
@@ -3463,7 +3482,7 @@ export const OPENROUTER_MODELS = {
input: ["text"],
cost: {
input: 0.09,
output: 0.55,
output: 0.1,
cacheRead: 0,
cacheWrite: 0,
},
@@ -3485,7 +3504,7 @@ export const OPENROUTER_MODELS = {
cacheRead: 0,
cacheWrite: 0,
},
contextWindow: 262144,
contextWindow: 131072,
maxTokens: 4096,
} satisfies Model<"openai-completions">,
"qwen/qwen3-30b-a3b": {
@@ -3503,7 +3522,7 @@ export const OPENROUTER_MODELS = {
cacheRead: 0,
cacheWrite: 0,
},
contextWindow: 131072,
contextWindow: 40960,
maxTokens: 16384,
} satisfies Model<"openai-completions">,
"qwen/qwen3-30b-a3b-instruct-2507": {
@@ -3521,7 +3540,7 @@ export const OPENROUTER_MODELS = {
cacheRead: 0,
cacheWrite: 0,
},
contextWindow: 131072,
contextWindow: 128000,
maxTokens: 32000,
} satisfies Model<"openai-completions">,
"qwen/qwen3-30b-a3b-thinking-2507": {
@@ -3539,7 +3558,7 @@ export const OPENROUTER_MODELS = {
cacheRead: 0,
cacheWrite: 0,
},
contextWindow: 131072,
contextWindow: 81920,
maxTokens: 32768,
} satisfies Model<"openai-completions">,
"qwen/qwen3-32b": {
@@ -3557,7 +3576,7 @@ export const OPENROUTER_MODELS = {
cacheRead: 0,
cacheWrite: 0,
},
contextWindow: 131072,
contextWindow: 40960,
maxTokens: 16384,
} satisfies Model<"openai-completions">,
"qwen/qwen3-8b": {
@@ -3593,7 +3612,7 @@ export const OPENROUTER_MODELS = {
cacheRead: 0,
cacheWrite: 0,
},
contextWindow: 1048576,
contextWindow: 262144,
maxTokens: 65536,
} satisfies Model<"openai-completions">,
"qwen/qwen3-coder-30b-a3b-instruct": {
@@ -3683,7 +3702,7 @@ export const OPENROUTER_MODELS = {
cacheRead: 0,
cacheWrite: 0,
},
contextWindow: 1048576,
contextWindow: 262000,
maxTokens: 262000,
} satisfies Model<"openai-completions">,
"qwen/qwen3-max": {
@@ -3773,7 +3792,7 @@ export const OPENROUTER_MODELS = {
cacheRead: 0,
cacheWrite: 0,
},
contextWindow: 262144,
contextWindow: 131072,
maxTokens: 32768,
} satisfies Model<"openai-completions">,
"qwen/qwen3-vl-235b-a22b-instruct": {
@@ -3827,7 +3846,7 @@ export const OPENROUTER_MODELS = {
cacheRead: 0,
cacheWrite: 0,
},
contextWindow: 262144,
contextWindow: 131072,
maxTokens: 32768,
} satisfies Model<"openai-completions">,
"qwen/qwen3-vl-30b-a3b-thinking": {
@@ -3863,7 +3882,7 @@ export const OPENROUTER_MODELS = {
cacheRead: 0,
cacheWrite: 0,
},
contextWindow: 262144,
contextWindow: 131072,
maxTokens: 32768,
} satisfies Model<"openai-completions">,
"qwen/qwen3-vl-8b-instruct": {
@@ -3881,7 +3900,7 @@ export const OPENROUTER_MODELS = {
cacheRead: 0,
cacheWrite: 0,
},
contextWindow: 256000,
contextWindow: 131072,
maxTokens: 32768,
} satisfies Model<"openai-completions">,
"qwen/qwen3-vl-8b-thinking": {
@@ -3899,7 +3918,7 @@ export const OPENROUTER_MODELS = {
cacheRead: 0,
cacheWrite: 0,
},
contextWindow: 256000,
contextWindow: 131072,
maxTokens: 32768,
} satisfies Model<"openai-completions">,
"qwen/qwen3.5-122b-a10b": {
@@ -3971,7 +3990,7 @@ export const OPENROUTER_MODELS = {
cacheRead: 0.111,
cacheWrite: 0,
},
contextWindow: 256000,
contextWindow: 131072,
maxTokens: 4096,
} satisfies Model<"openai-completions">,
"qwen/qwen3.5-9b": {
@@ -4056,13 +4075,13 @@ export const OPENROUTER_MODELS = {
reasoning: true,
input: ["text", "image"],
cost: {
input: 0.289,
input: 0.285,
output: 2.4,
cacheRead: 0,
cacheRead: 0.15,
cacheWrite: 0,
},
contextWindow: 262144,
maxTokens: 131072,
contextWindow: 262140,
maxTokens: 262140,
} satisfies Model<"openai-completions">,
"qwen/qwen3.6-35b-a3b": {
id: "qwen/qwen3.6-35b-a3b",
@@ -4457,7 +4476,7 @@ export const OPENROUTER_MODELS = {
cacheRead: 0.028,
cacheWrite: 0,
},
contextWindow: 1048576,
contextWindow: 32000,
maxTokens: 4096,
} satisfies Model<"openai-completions">,
"xiaomi/mimo-v2.5-pro": {
@@ -4543,12 +4562,12 @@ export const OPENROUTER_MODELS = {
input: ["text"],
cost: {
input: 0.43,
output: 1.75,
output: 1.74,
cacheRead: 0.08,
cacheWrite: 0,
},
contextWindow: 200000,
maxTokens: 16384,
contextWindow: 202752,
maxTokens: 131072,
} satisfies Model<"openai-completions">,
"z-ai/glm-4.6v": {
id: "z-ai/glm-4.6v",
@@ -4620,7 +4639,7 @@ export const OPENROUTER_MODELS = {
cacheWrite: 0,
},
contextWindow: 202752,
maxTokens: 128000,
maxTokens: 4096,
} satisfies Model<"openai-completions">,
"z-ai/glm-5-turbo": {
id: "z-ai/glm-5-turbo",
@@ -4655,7 +4674,7 @@ export const OPENROUTER_MODELS = {
cacheRead: 0.1794,
cacheWrite: 0,
},
contextWindow: 202752,
contextWindow: 200000,
maxTokens: 128000,
} satisfies Model<"openai-completions">,
"z-ai/glm-5.2": {
@@ -4669,13 +4688,13 @@ export const OPENROUTER_MODELS = {
thinkingLevelMap: {"xhigh":"xhigh"},
input: ["text"],
cost: {
input: 0.924,
output: 2.904,
cacheRead: 0.1716,
input: 0.84,
output: 2.64,
cacheRead: 0.156,
cacheWrite: 0,
},
contextWindow: 1048576,
maxTokens: 131072,
contextWindow: 1024000,
maxTokens: 128000,
} satisfies Model<"openai-completions">,
"z-ai/glm-5v-turbo": {
id: "z-ai/glm-5v-turbo",
packages/ai/src/providers/radius-config.ts
+95
View File
@@ -0,0 +1,95 @@
import type { OAuthCredential } from "../auth/types.ts";
import type { Model, ThinkingLevelMap } from "../types.ts";
export const DEFAULT_RADIUS_GATEWAY = "https://radius.pi.dev";
export type RadiusGatewayModel = {
id: string;
name: string;
reasoning: boolean;
thinkingLevelMap?: ThinkingLevelMap;
input: ("text" | "image")[];
cost: Model<"pi-messages">["cost"];
contextWindow: number;
maxTokens: number;
};
export type RadiusGatewayConfig = {
baseUrl: string;
models: RadiusGatewayModel[];
};
export type RadiusOAuthCredential = OAuthCredential & {
gatewayConfig?: RadiusGatewayConfig;
};
function isRecord(value: unknown): value is Record<string, unknown> {
return typeof value === "object" && value !== null && !Array.isArray(value);
}
function isRadiusGatewayModel(value: unknown): value is RadiusGatewayModel {
return (
isRecord(value) &&
typeof value.id === "string" &&
typeof value.name === "string" &&
typeof value.reasoning === "boolean" &&
Array.isArray(value.input) &&
isRecord(value.cost) &&
typeof value.contextWindow === "number" &&
typeof value.maxTokens === "number"
);
}
function sanitizeRadiusGatewayConfig(config: unknown): RadiusGatewayConfig | undefined {
if (!isRecord(config) || typeof config.baseUrl !== "string" || !Array.isArray(config.models)) return undefined;
return {
baseUrl: config.baseUrl,
models: config.models.filter(isRadiusGatewayModel).map((model) => ({ ...model })),
};
}
export function normalizeRadiusGatewayUrl(value: string): string {
const withScheme = /^https?:\/\//iu.test(value) ? value : `https://${value}`;
return withScheme.replace(/\/+$/u, "");
}
export function getRadiusCredentialConfig(credential: OAuthCredential | undefined): RadiusGatewayConfig | undefined {
return sanitizeRadiusGatewayConfig((credential as RadiusOAuthCredential | undefined)?.gatewayConfig);
}
export function getRadiusModelsFromConfig(providerId: string, config: RadiusGatewayConfig): Model<"pi-messages">[] {
return config.models.map((model) => ({
...model,
api: "pi-messages",
provider: providerId,
baseUrl: config.baseUrl,
}));
}
export function getRadiusModels(providerId: string, credential: OAuthCredential | undefined): Model<"pi-messages">[] {
const config = getRadiusCredentialConfig(credential);
return config ? getRadiusModelsFromConfig(providerId, config) : [];
}
function truncateHttpBody(body: string): string {
const trimmed = body.trim();
return trimmed.length > 512 ? `${trimmed.slice(0, 512)}…` : trimmed;
}
export async function loadRadiusGatewayConfig(
gateway: string,
apiKey?: string,
signal?: AbortSignal,
): Promise<RadiusGatewayConfig> {
const headers: Record<string, string> = { accept: "application/json" };
if (apiKey) headers.authorization = `Bearer ${apiKey}`;
const response = await fetch(new URL("/v1/config", gateway), { headers, signal });
if (!response.ok) {
throw new Error(
`Could not load Radius config from ${gateway}: ${response.status}: ${truncateHttpBody(await response.text())}`,
);
}
const config = sanitizeRadiusGatewayConfig(await response.json());
if (!config) throw new Error(`Invalid Radius config from ${gateway}`);
return config;
}
packages/ai/src/providers/radius.ts
+67
View File
@@ -0,0 +1,67 @@
import { piMessagesApi } from "../api/pi-messages.lazy.ts";
import { envApiKeyAuth, lazyOAuth } from "../auth/helpers.ts";
import { loadRadiusOAuth } from "../auth/oauth/load.ts";
import type { Provider } from "../models.ts";
import {
DEFAULT_RADIUS_GATEWAY,
getRadiusModels,
getRadiusModelsFromConfig,
loadRadiusGatewayConfig,
normalizeRadiusGatewayUrl,
} from "./radius-config.ts";
export interface RadiusProviderOptions {
id?: string;
name?: string;
gateway?: string;
}
/** Radius gateway provider with a persisted, dynamically refreshed catalog. */
export function radiusProvider(options: RadiusProviderOptions = {}): Provider<"pi-messages"> {
const id = options.id ?? "radius";
const name = options.name ?? "Radius";
const gateway = normalizeRadiusGatewayUrl(options.gateway ?? DEFAULT_RADIUS_GATEWAY);
let models = getRadiusModels(id, undefined);
let inflightRefresh: Promise<void> | undefined;
const streams = piMessagesApi();
return {
id,
name,
auth: {
apiKey: envApiKeyAuth("Radius API key", ["RADIUS_API_KEY"]),
oauth: lazyOAuth({ name, load: () => loadRadiusOAuth({ name, gateway }) }),
},
getModels: () => models,
refreshModels: (context) => {
inflightRefresh ??= (async () => {
try {
const stored = await context.store.read();
if (stored) models = stored.filter((model) => model.provider === id) as typeof models;
// Import catalogs cached by the pre-ModelsStore Radius implementation.
if (!stored && context.credential?.type === "oauth") {
const legacy = getRadiusModels(id, context.credential);
if (legacy.length > 0) {
models = legacy;
await context.store.write(legacy);
}
}
if (!context.allowNetwork || context.signal?.aborted) return;
const apiKey =
context.credential?.type === "oauth" ? context.credential.access : context.credential?.key;
const config = await loadRadiusGatewayConfig(gateway, apiKey, context.signal);
if (context.signal?.aborted) return;
models = getRadiusModelsFromConfig(id, config);
await context.store.write(models);
} finally {
inflightRefresh = undefined;
}
})();
return inflightRefresh;
},
stream: (model, context, streamOptions) => streams.stream(model, context, streamOptions),
streamSimple: (model, context, streamOptions) => streams.streamSimple(model, context, streamOptions),
};
}
packages/ai/src/types.ts
+26 -5
View File
@@ -7,6 +7,7 @@ import type { MistralOptions } from "./api/mistral-conversations.ts";
import type { OpenAICodexResponsesOptions } from "./api/openai-codex-responses.ts";
import type { OpenAICompletionsOptions } from "./api/openai-completions.ts";
import type { OpenAIResponsesOptions } from "./api/openai-responses.ts";
import type { PiMessagesOptions } from "./api/pi-messages.ts";
import type { AssistantMessageDiagnostic } from "./utils/diagnostics.ts";
import type { AssistantMessageEventStream } from "./utils/event-stream.ts";
@@ -21,7 +22,8 @@ export type KnownApi =
| "anthropic-messages"
| "bedrock-converse-stream"
| "google-generative-ai"
| "google-vertex";
| "google-vertex"
| "pi-messages";
export type Api = KnownApi | (string & {});
@@ -38,6 +40,7 @@ export type KnownProvider =
| "openai"
| "azure-openai-responses"
| "openai-codex"
| "radius"
| "nvidia"
| "deepseek"
| "github-copilot"
@@ -100,6 +103,7 @@ export type Transport = "sse" | "websocket" | "websocket-cached" | "auto";
/** Provider-scoped environment overrides. Values take precedence over process.env. */
export type ProviderEnv = Record<string, string>;
export type ProviderHeaders = Record<string, string | null>;
export type SessionAffinityFormat = "openai" | "openai-nosession" | "openrouter";
export interface ProviderResponse {
status: number;
@@ -201,6 +205,7 @@ export interface ApiOptionsMap {
"google-vertex": GoogleVertexOptions;
"mistral-conversations": MistralOptions;
"bedrock-converse-stream": BedrockOptions;
"pi-messages": PiMessagesOptions;
}
/**
@@ -401,6 +406,12 @@ export interface ToolResultMessage<TDetails = any> {
toolName: string;
content: (TextContent | ImageContent)[]; // Supports text and images
details?: TDetails;
/**
* Names from `Context.tools` that became available after this result.
* Providers with native deferred tool loading use this as the load point;
* other providers ignore it and use `Context.tools` normally.
*/
addedToolNames?: string[];
isError: boolean;
timestamp: number; // Unix timestamp in milliseconds
}
@@ -511,8 +522,10 @@ export interface OpenAICompletionsCompat {
supportsStrictMode?: boolean;
/** Cache control convention for prompt caching. "anthropic" applies Anthropic-style `cache_control` markers to the system prompt, last tool definition, and last user/assistant text content. */
cacheControlFormat?: "anthropic";
/** Whether to send known session-affinity headers (`session_id`, `x-client-request-id`, `x-session-affinity`) from `options.sessionId` when caching is enabled. Default: false. */
/** Whether to send session-affinity data from `options.sessionId`. Default: false. */
sendSessionAffinityHeaders?: boolean;
/** Session-affinity header format: `openai` sends `session_id`, `x-client-request-id`, and `x-session-affinity`; `openai-nosession` sends `x-client-request-id` and `x-session-affinity`; `openrouter` sends `x-session-id`. Does not affect the `prompt_cache_key` body param, which is governed by cache retention. Default: auto-detected. */
sessionAffinityFormat?: SessionAffinityFormat;
/** Whether the provider supports long prompt cache retention (`prompt_cache_retention: "24h"` or Anthropic-style `cache_control.ttl: "1h"`, depending on format). Default: true. */
supportsLongCacheRetention?: boolean;
}
@@ -521,10 +534,12 @@ export interface OpenAICompletionsCompat {
export interface OpenAIResponsesCompat {
/** Whether the provider supports the `developer` role (vs `system`). Default: true. */
supportsDeveloperRole?: boolean;
/** Whether to send the OpenAI `session_id` cache-affinity header from `options.sessionId` when caching is enabled. Default: true. */
sendSessionIdHeader?: boolean;
/** Session-affinity header format: `openai` sends `session_id` and `x-client-request-id`; `openai-nosession` sends `x-client-request-id`; `openrouter` sends `x-session-id`. Does not affect the `prompt_cache_key` body param, which is governed by cache retention. Default: auto-detected. */
sessionAffinityFormat?: SessionAffinityFormat;
/** Whether the provider supports `prompt_cache_retention: "24h"`. Default: true. */
supportsLongCacheRetention?: boolean;
/** Whether the model supports client-executed tool search for deferred tools. Default: false. */
supportsToolSearch?: boolean;
}
/** Compatibility settings for Anthropic Messages-compatible APIs. */
@@ -573,6 +588,12 @@ export interface AnthropicMessagesCompat {
forceAdaptiveThinking?: boolean;
/** Whether to replay empty thinking signatures as `signature: ""` instead of converting thinking to text. Default: false. */
allowEmptySignature?: boolean;
/**
* Whether the provider supports deferred tools loaded by `tool_reference`
* blocks in tool results. Default: true for first-party Anthropic models
* except Haiku and models older than Claude 4.5; false for other providers.
*/
supportsToolReferences?: boolean;
}
/**
@@ -700,7 +721,7 @@ export interface Model<TApi extends Api> {
/** Compatibility overrides for OpenAI-compatible APIs. If not set, auto-detected from baseUrl. */
compat?: TApi extends "openai-completions"
? OpenAICompletionsCompat
: TApi extends "openai-responses"
: TApi extends "openai-responses" | "openai-codex-responses"
? OpenAIResponsesCompat
: TApi extends "anthropic-messages"
? AnthropicMessagesCompat
packages/ai/src/utils/deferred-tools.ts
+39
View File
@@ -0,0 +1,39 @@
import type { Context, Tool } from "../types.ts";
type ToolNameNormalizer = (name: string) => string;
const identityToolName: ToolNameNormalizer = (name) => name;
/** Split current tools into prefix and transcript-loaded definitions. */
export function splitDeferredTools(
context: Context,
enabled: boolean,
normalizeName: ToolNameNormalizer = identityToolName,
): { immediate: Tool[]; deferred: Map<string, Tool> } {
const uniqueTools = new Map<string, Tool>();
for (const tool of context.tools ?? []) uniqueTools.set(normalizeName(tool.name), tool);
if (!enabled) return { immediate: [...uniqueTools.values()], deferred: new Map() };
const deferredNames = new Set<string>();
const usedNames = new Set<string>();
for (const message of context.messages) {
if (message.role === "assistant") {
for (const block of message.content) {
if (block.type === "toolCall") usedNames.add(normalizeName(block.name));
}
} else if (message.role === "toolResult") {
for (const name of message.addedToolNames ?? []) {
const normalizedName = normalizeName(name);
if (!usedNames.has(normalizedName)) deferredNames.add(normalizedName);
}
}
}
const immediate: Tool[] = [];
const deferred = new Map<string, Tool>();
for (const [name, tool] of uniqueTools) {
if (deferredNames.has(name)) deferred.set(name, tool);
else immediate.push(tool);
}
return { immediate, deferred };
}
packages/ai/src/utils/estimate.ts
+23 -6
View File
@@ -1,4 +1,4 @@
import type { AssistantMessage, Context, ImageContent, Message, TextContent, Usage } from "../types.ts";
import type { AssistantMessage, Context, ImageContent, Message, TextContent, Tool, Usage } from "../types.ts";
export interface ContextUsageEstimate {
/** Estimated total context tokens. */
@@ -102,6 +102,11 @@ function estimateMessages(messages: readonly Message[]): ContextUsageEstimate {
return { tokens, usageTokens: 0, trailingTokens: tokens, lastUsageIndex: null };
}
function estimateToolsTokens(tools: readonly Tool[] | undefined): number {
if (!tools || tools.length === 0) return 0;
return estimateTextTokens(safeJsonStringify(tools));
}
function isMessageArray(value: Context | readonly Message[]): value is readonly Message[] {
return Array.isArray(value);
}
@@ -110,13 +115,25 @@ export function estimateContextTokens(context: Context | readonly Message[]): Co
if (isMessageArray(context)) return estimateMessages(context);
const estimate = estimateMessages(context.messages);
if (estimate.lastUsageIndex !== null) return estimate;
let prefixTokens = context.systemPrompt ? estimateTextTokens(context.systemPrompt) : 0;
if (context.tools && context.tools.length > 0) {
prefixTokens += estimateTextTokens(safeJsonStringify(context.tools));
if (estimate.lastUsageIndex !== null) {
const addedNames = new Set(
context.messages
.slice(estimate.lastUsageIndex + 1)
.filter((message) => message.role === "toolResult")
.flatMap((message) => message.addedToolNames ?? []),
);
const addedToolTokens = estimateToolsTokens(context.tools?.filter((tool) => addedNames.has(tool.name)));
return {
tokens: estimate.tokens + addedToolTokens,
usageTokens: estimate.usageTokens,
trailingTokens: estimate.trailingTokens + addedToolTokens,
lastUsageIndex: estimate.lastUsageIndex,
};
}
const prefixTokens =
(context.systemPrompt ? estimateTextTokens(context.systemPrompt) : 0) + estimateToolsTokens(context.tools);
return {
tokens: estimate.tokens + prefixTokens,
usageTokens: estimate.usageTokens,
packages/ai/test/anthropic-eager-tool-input-e2e.test.ts
+4 -4
View File
@@ -1,6 +1,6 @@
import { Type } from "typebox";
import { describe, expect, it } from "vitest";
import { complete, getModels, getProviders } from "../src/compat.ts";
import { type BuiltinProvider, complete, getModels, getProviders } from "../src/compat.ts";
import { getEnvApiKey } from "../src/env-api-keys.ts";
import type { Api, KnownProvider, Model, ProviderStreamOptions, Tool } from "../src/types.ts";
import { resolveApiKey } from "./oauth.ts";
@@ -19,7 +19,7 @@ const echoTool: Tool<typeof echoToolSchema> = {
interface AnthropicEagerE2ECase {
name: string;
provider: KnownProvider;
provider: BuiltinProvider;
model: Model<"anthropic-messages">;
apiKey: string | undefined;
}
@@ -31,7 +31,7 @@ function getE2EApiKey(provider: KnownProvider): string | undefined {
return getEnvApiKey(provider);
}
function getAnthropicMessagesModels(provider: KnownProvider): Model<"anthropic-messages">[] {
function getAnthropicMessagesModels(provider: BuiltinProvider): Model<"anthropic-messages">[] {
const models = getModels(provider) as Model<Api>[];
return models.filter((model) => model.api === "anthropic-messages") as Model<"anthropic-messages">[];
}
@@ -64,7 +64,7 @@ function getProbePriority(model: Model<"anthropic-messages">): number {
}
function selectOneCasePerProvider(cases: AnthropicEagerE2ECase[]): AnthropicEagerE2ECase[] {
const byProvider = new Map<KnownProvider, AnthropicEagerE2ECase[]>();
const byProvider = new Map<BuiltinProvider, AnthropicEagerE2ECase[]>();
for (const testCase of cases) {
const providerCases = byProvider.get(testCase.provider) ?? [];
providerCases.push(testCase);
packages/ai/test/anthropic-long-cache-retention-e2e.test.ts
+4 -4
View File
@@ -1,5 +1,5 @@
import { describe, expect, it } from "vitest";
import { complete, getModels, getProviders } from "../src/compat.ts";
import { type BuiltinProvider, complete, getModels, getProviders } from "../src/compat.ts";
import { getEnvApiKey } from "../src/env-api-keys.ts";
import type { Api, KnownProvider, Model, ProviderStreamOptions } from "../src/types.ts";
import { resolveApiKey } from "./oauth.ts";
@@ -8,7 +8,7 @@ const githubCopilotToken = await resolveApiKey("github-copilot");
interface AnthropicLongCacheRetentionE2ECase {
name: string;
provider: KnownProvider;
provider: BuiltinProvider;
model: Model<"anthropic-messages">;
apiKey: string | undefined;
}
@@ -20,7 +20,7 @@ function getE2EApiKey(provider: KnownProvider): string | undefined {
return getEnvApiKey(provider);
}
function getAnthropicMessagesModels(provider: KnownProvider): Model<"anthropic-messages">[] {
function getAnthropicMessagesModels(provider: BuiltinProvider): Model<"anthropic-messages">[] {
const models = getModels(provider) as Model<Api>[];
return models.filter((model) => model.api === "anthropic-messages") as Model<"anthropic-messages">[];
}
@@ -51,7 +51,7 @@ function getProbePriority(model: Model<"anthropic-messages">): number {
}
function selectOneCasePerProvider(cases: AnthropicLongCacheRetentionE2ECase[]): AnthropicLongCacheRetentionE2ECase[] {
const byProvider = new Map<KnownProvider, AnthropicLongCacheRetentionE2ECase[]>();
const byProvider = new Map<BuiltinProvider, AnthropicLongCacheRetentionE2ECase[]>();
for (const testCase of cases) {
const providerCases = byProvider.get(testCase.provider) ?? [];
providerCases.push(testCase);
packages/ai/test/anthropic-sse-parsing.test.ts
+28
View File
@@ -224,6 +224,34 @@ describe("Anthropic raw SSE parsing", () => {
expect(result.errorMessage).toBe(explanation);
});
it("treats message_delta without usage as a no-op for usage accumulation", async () => {
const model = getModel("anthropic", "claude-haiku-4-5");
const context: Context = {
messages: [{ role: "user", content: "Say hello.", timestamp: Date.now() }],
};
const response = createSseResponse(
minimalAnthropicEvents.map((event) =>
event.event === "message_delta"
? {
event: "message_delta",
data: JSON.stringify({ type: "message_delta", delta: { stop_reason: "end_turn" } }),
}
: event,
),
);
const stream = streamAnthropic(model, context, {
client: createFakeAnthropicClient(response),
});
const result = await stream.result();
expect(result.stopReason).toBe("stop");
expect(result.errorMessage).toBeUndefined();
expect(result.content).toEqual([{ type: "text", text: "Hello" }]);
expect(result.usage.input).toBe(12);
expect(result.usage.totalTokens).toBe(12);
});
it("ignores unknown SSE events after message_stop", async () => {
const model = getModel("anthropic", "claude-haiku-4-5");
const context: Context = {
packages/ai/test/azure-openai-responses-reasoning-replay.test.ts
+136
View File
@@ -0,0 +1,136 @@
import type { ResponseReasoningItem, ResponseStreamEvent } from "openai/resources/responses/responses.js";
import { describe, expect, it } from "vitest";
import { convertResponsesMessages, processResponsesStream } from "../src/api/openai-responses-shared.ts";
import type { AssistantMessage, Context, Model } from "../src/types.ts";
import { AssistantMessageEventStream } from "../src/utils/event-stream.ts";
function createModel(): Model<"azure-openai-responses"> {
return {
id: "gpt-5-mini",
name: "GPT-5 Mini",
api: "azure-openai-responses",
provider: "azure-openai-responses",
baseUrl: "https://example.invalid",
reasoning: true,
input: ["text"],
cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 },
contextWindow: 400000,
maxTokens: 128000,
};
}
function createOutput(model: Model<"azure-openai-responses">): AssistantMessage {
return {
role: "assistant",
content: [],
api: model.api,
provider: model.provider,
model: model.id,
usage: {
input: 0,
output: 0,
cacheRead: 0,
cacheWrite: 0,
totalTokens: 0,
cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0, total: 0 },
},
stopReason: "stop",
timestamp: Date.now(),
};
}
async function* createEvents(
doneItem: ResponseReasoningItem,
completedItem: ResponseReasoningItem,
): AsyncIterable<ResponseStreamEvent> {
yield {
type: "response.output_item.added",
output_index: 0,
sequence_number: 0,
item: { type: "reasoning", id: doneItem.id, summary: [] },
} as ResponseStreamEvent;
yield {
type: "response.output_item.done",
output_index: 0,
sequence_number: 1,
item: doneItem,
} as ResponseStreamEvent;
yield {
type: "response.completed",
sequence_number: 2,
response: {
id: "resp_test",
status: "completed",
output: [completedItem],
},
} as ResponseStreamEvent;
}
function getReplayedReasoning(model: Model<"azure-openai-responses">, assistant: AssistantMessage) {
const context: Context = {
messages: [
{ role: "user", content: "first", timestamp: Date.now() - 1 },
assistant,
{ role: "user", content: "follow-up", timestamp: Date.now() },
],
};
const input = convertResponsesMessages(model, context, new Set(["azure-openai-responses"]));
return input.find((item) => item.type === "reasoning");
}
describe("Azure OpenAI Responses reasoning replay", () => {
it("preserves existing encrypted_content from output_item.done", async () => {
const model = createModel();
const output = createOutput(model);
const doneItem: ResponseReasoningItem = {
type: "reasoning",
id: "rs_done",
summary: [],
encrypted_content: "from-output-item-done",
};
const completedItem: ResponseReasoningItem = {
...doneItem,
encrypted_content: "from-response-completed",
};
await processResponsesStream(
createEvents(doneItem, completedItem),
output,
new AssistantMessageEventStream(),
model,
);
expect(getReplayedReasoning(model, output)).toMatchObject({
type: "reasoning",
id: "rs_done",
encrypted_content: "from-output-item-done",
});
});
it("fills encrypted_content when output_item.done omitted it", async () => {
const model = createModel();
const output = createOutput(model);
const doneItem: ResponseReasoningItem = {
type: "reasoning",
id: "rs_missing",
summary: [],
};
const completedItem: ResponseReasoningItem = {
...doneItem,
encrypted_content: "from-response-completed",
};
await processResponsesStream(
createEvents(doneItem, completedItem),
output,
new AssistantMessageEventStream(),
model,
);
expect(getReplayedReasoning(model, output)).toMatchObject({
type: "reasoning",
id: "rs_missing",
encrypted_content: "from-response-completed",
});
});
});
packages/ai/test/bedrock-endpoint-resolution.test.ts
+26 -2
View File
@@ -44,8 +44,8 @@ vi.mock("@aws-sdk/client-bedrock-runtime", () => {
};
});
import { type BedrockOptions, stream as streamBedrock } from "../src/api/bedrock-converse-stream.ts";
import { getModel } from "../src/compat.ts";
import type { BedrockOptions } from "../src/api/bedrock-converse-stream.ts";
import { getModel, stream as streamBedrock } from "../src/compat.ts";
import type { Context, Model } from "../src/types.ts";
const context: Context = {
@@ -181,4 +181,28 @@ describe("bedrock endpoint resolution", () => {
expect(config.region).toBe("us-gov-west-1");
});
it("preserves ambient AWS auth for custom model IDs through compat dispatch", async () => {
process.env.AWS_PROFILE = "bedrock-profile";
const baseModel = getModel("amazon-bedrock", "us.anthropic.claude-opus-4-8");
const model: Model<"bedrock-converse-stream"> = {
...baseModel,
id: "arn:aws:bedrock:us-east-1:123456789012:application-inference-profile/example",
};
const config = await captureClientConfig(model);
expect(config.profile).toBe("bedrock-profile");
expect(config.token).toBeUndefined();
expect(config.authSchemePreference).toBeUndefined();
});
it("uses the generic API key option as a Bedrock bearer token", async () => {
const model = getModel("amazon-bedrock", "us.anthropic.claude-opus-4-8");
const config = await captureClientConfig(model, { apiKey: "bedrock-api-key" });
expect(config.token).toEqual({ token: "bedrock-api-key" });
expect(config.authSchemePreference).toEqual(["httpBearerAuth"]);
});
});
packages/ai/test/deferred-tools.test.ts
+387
View File
@@ -0,0 +1,387 @@
import { Type } from "typebox";
import { describe, expect, it } from "vitest";
import { getModel, streamSimple } from "../src/compat.ts";
import type { Api, AssistantMessage, Context, Model, Tool, ToolResultMessage, UserMessage } from "../src/types.ts";
import { estimateContextTokens } from "../src/utils/estimate.ts";
interface AnthropicToolPayload {
name: string;
description?: string;
defer_loading?: boolean;
}
interface AnthropicContentBlock {
type: string;
text?: string;
tool_use_id?: string;
content?: string | Array<{ type: string; tool_name?: string }>;
source?: {
type: string;
media_type: string;
data: string;
};
}
interface AnthropicPayload {
tools?: AnthropicToolPayload[];
messages: Array<{
content: string | AnthropicContentBlock[];
}>;
}
interface OpenAIToolSearchCall {
type: "tool_search_call";
call_id?: string | null;
execution?: string;
status?: string | null;
}
interface OpenAIToolSearchOutput {
type: "tool_search_output";
call_id?: string | null;
execution?: string;
status?: string | null;
tools: Array<{ type: string; name: string; defer_loading?: boolean }>;
}
interface OpenAIPayload {
tools?: Array<{ name?: string; function?: { name: string } }>;
input?: Array<OpenAIToolSearchCall | OpenAIToolSearchOutput | { type?: string }>;
}
class PayloadCaptured extends Error {}
function makeTool(name: string): Tool {
return {
name,
description: `The ${name} tool`,
parameters: Type.Object({ value: Type.String() }),
};
}
function makeUserMessage(timestamp: number): UserMessage {
return { role: "user", content: "Hello", timestamp };
}
function makeAssistantToolCall(): AssistantMessage {
return {
role: "assistant",
content: [{ type: "toolCall", id: "call_1", name: "base_tool", arguments: {} }],
api: "anthropic-messages",
provider: "anthropic",
model: "claude-opus-4-6",
usage: {
input: 0,
output: 0,
cacheRead: 0,
cacheWrite: 0,
totalTokens: 0,
cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0, total: 0 },
},
stopReason: "toolUse",
timestamp: 2,
};
}
function makeToolResult(addedToolNames: string[]): ToolResultMessage {
return {
role: "toolResult",
toolCallId: "call_1",
toolName: "base_tool",
content: [{ type: "text", text: "done" }],
addedToolNames,
isError: false,
timestamp: 3,
};
}
function makeContext(tools: Tool[], addedToolNames = ["late_tool"]): Context {
return {
messages: [makeUserMessage(1), makeAssistantToolCall(), makeToolResult(addedToolNames), makeUserMessage(4)],
tools,
};
}
async function capturePayload<T>(model: Model<Api>, context: Context, apiKey = "fake-key"): Promise<T> {
let captured: T | undefined;
const stream = streamSimple({ ...model, baseUrl: "http://127.0.0.1:9" }, context, {
apiKey,
onPayload: (payload) => {
captured = payload as T;
throw new PayloadCaptured();
},
});
await stream.result();
if (!captured) throw new Error("Expected payload capture");
return captured;
}
function findAnthropicToolResultContent(payload: AnthropicPayload): AnthropicContentBlock[] {
for (const message of payload.messages) {
if (typeof message.content !== "string" && message.content.some((block) => block.type === "tool_result")) {
return message.content;
}
}
throw new Error("No tool result in payload");
}
function findAnthropicToolResult(payload: AnthropicPayload): AnthropicContentBlock {
const result = findAnthropicToolResultContent(payload).find((block) => block.type === "tool_result");
if (!result) throw new Error("No tool result in payload");
return result;
}
function openAIToolNames(payload: OpenAIPayload): string[] {
return (payload.tools ?? []).map((tool) => tool.name ?? tool.function?.name ?? "");
}
function makeCodexToken(): string {
return `header.${btoa(JSON.stringify({ "https://api.openai.com/auth": { chatgpt_account_id: "account" } }))}.signature`;
}
describe("deferred tools", () => {
it("loads an Anthropic tool at its tool-result marker", async () => {
const context = makeContext([makeTool("base_tool"), makeTool("late_tool")]);
const payload = await capturePayload<AnthropicPayload>(getModel("anthropic", "claude-opus-4-6"), context);
expect(payload.tools).toMatchObject([{ name: "base_tool" }, { name: "late_tool", defer_loading: true }]);
expect(findAnthropicToolResult(payload).content).toEqual([{ type: "tool_reference", tool_name: "late_tool" }]);
});
it("preserves tool output as sibling content after emitting references", async () => {
const context = makeContext([makeTool("base_tool"), makeTool("late_tool")]);
const assistant = context.messages[1] as AssistantMessage;
assistant.content = [
{ type: "toolCall", id: "call_1", name: "base_tool", arguments: {} },
{ type: "toolCall", id: "call_2", name: "base_tool", arguments: {} },
];
const firstResult = context.messages[2] as ToolResultMessage;
firstResult.content = [
{ type: "text", text: "work completed" },
{ type: "image", mimeType: "image/png", data: "aW1hZ2U=" },
];
context.messages.splice(3, 0, {
...makeToolResult([]),
toolCallId: "call_2",
content: [{ type: "text", text: "second result" }],
});
const payload = await capturePayload<AnthropicPayload>(getModel("anthropic", "claude-opus-4-6"), context);
expect(findAnthropicToolResultContent(payload)).toMatchObject([
{
type: "tool_result",
tool_use_id: "call_1",
content: [{ type: "tool_reference", tool_name: "late_tool" }],
},
{ type: "tool_result", tool_use_id: "call_2", content: "second result" },
{ type: "text", text: "work completed" },
{
type: "image",
source: { type: "base64", media_type: "image/png", data: "aW1hZ2U=" },
},
]);
});
it("loads a tool introduced by OpenAI history after switching to Anthropic", async () => {
const context = makeContext([makeTool("base_tool"), makeTool("late_tool")]);
const assistant = context.messages[1] as AssistantMessage;
assistant.api = "openai-responses";
assistant.provider = "openai";
assistant.model = "gpt-5.4";
const payload = await capturePayload<AnthropicPayload>(getModel("anthropic", "claude-opus-4-8"), context);
expect(payload.tools).toMatchObject([{ name: "base_tool" }, { name: "late_tool", defer_loading: true }]);
expect(findAnthropicToolResult(payload).content).toEqual([{ type: "tool_reference", tool_name: "late_tool" }]);
});
it("does not resurrect a marked tool missing from Context.tools", async () => {
const context = makeContext([makeTool("base_tool")]);
const payload = await capturePayload<AnthropicPayload>(getModel("anthropic", "claude-opus-4-6"), context);
expect(payload.tools?.map((tool) => tool.name)).toEqual(["base_tool"]);
const content = findAnthropicToolResult(payload).content;
expect(Array.isArray(content) && content.some((block) => block.type === "tool_reference")).toBe(false);
});
it("keeps a tool immediate when it was used before its marker", async () => {
const context = makeContext([makeTool("base_tool"), makeTool("late_tool")]);
const assistant = context.messages[1] as AssistantMessage;
assistant.content = [{ type: "toolCall", id: "call_1", name: "late_tool", arguments: {} }];
const payload = await capturePayload<AnthropicPayload>(getModel("anthropic", "claude-opus-4-6"), context);
expect(payload.tools?.map((tool) => tool.name)).toEqual(["base_tool", "late_tool"]);
expect(payload.tools?.every((tool) => !tool.defer_loading)).toBe(true);
});
it("normalizes OAuth names before checking prior tool usage", async () => {
const context = makeContext([makeTool("base_tool"), makeTool("read")], ["read"]);
const assistant = context.messages[1] as AssistantMessage;
assistant.content = [{ type: "toolCall", id: "call_1", name: "Read", arguments: {} }];
const payload = await capturePayload<AnthropicPayload>(
getModel("anthropic", "claude-opus-4-6"),
context,
"sk-ant-oat-fake",
);
expect(payload.tools?.map((tool) => tool.name)).toEqual(["base_tool", "Read"]);
expect(payload.tools?.every((tool) => !tool.defer_loading)).toBe(true);
const content = findAnthropicToolResult(payload).content;
expect(Array.isArray(content) && content.some((block) => block.type === "tool_reference")).toBe(false);
});
it("matches OAuth-canonicalized markers to active tools", async () => {
const context = makeContext([makeTool("base_tool"), makeTool("read")], ["Read"]);
const payload = await capturePayload<AnthropicPayload>(
getModel("anthropic", "claude-opus-4-6"),
context,
"sk-ant-oat-fake",
);
expect(payload.tools).toMatchObject([{ name: "base_tool" }, { name: "Read", defer_loading: true }]);
const content = findAnthropicToolResult(payload).content;
expect(
Array.isArray(content) &&
content.some((block) => block.type === "tool_reference" && block.tool_name === "Read"),
).toBe(true);
});
it("deduplicates active tools after OAuth canonicalization", async () => {
const context: Context = {
messages: [makeUserMessage(1)],
tools: [makeTool("read"), { ...makeTool("Read"), description: "Canonical definition" }],
};
const payload = await capturePayload<AnthropicPayload>(
getModel("anthropic", "claude-opus-4-6"),
context,
"sk-ant-oat-fake",
);
expect(payload.tools).toMatchObject([{ name: "Read", description: "Canonical definition" }]);
});
it("uses the normal tool list when Anthropic tool references are unsupported", async () => {
const context = makeContext([makeTool("base_tool"), makeTool("late_tool")]);
const models: Model<"anthropic-messages">[] = [
getModel("anthropic", "claude-haiku-4-5"),
{ ...getModel("anthropic", "claude-opus-4-6"), id: "claude-sonnet-4-20250514" },
];
for (const model of models) {
const payload = await capturePayload<AnthropicPayload>(model, context);
expect(payload.tools?.map((tool) => tool.name)).toEqual(["base_tool", "late_tool"]);
expect(payload.tools?.every((tool) => !tool.defer_loading)).toBe(true);
}
});
it("keeps one immediate Anthropic tool when every current tool is marked", async () => {
const context = makeContext([makeTool("late_tool")]);
const payload = await capturePayload<AnthropicPayload>(getModel("anthropic", "claude-opus-4-6"), context);
expect(payload.tools).toMatchObject([{ name: "late_tool" }]);
expect(payload.tools?.[0]?.defer_loading).toBeUndefined();
const content = findAnthropicToolResult(payload).content;
expect(Array.isArray(content) && content.some((block) => block.type === "tool_reference")).toBe(false);
});
it("supports explicit Anthropic compatibility overrides", async () => {
const model: Model<"anthropic-messages"> = {
...getModel("anthropic", "claude-opus-4-6"),
provider: "anthropic-proxy",
compat: { supportsToolReferences: true },
};
const context = makeContext([makeTool("base_tool"), makeTool("late_tool")]);
const payload = await capturePayload<AnthropicPayload>(model, context);
expect(payload.tools?.find((tool) => tool.name === "late_tool")?.defer_loading).toBe(true);
});
it("loads an OpenAI Responses tool through client tool search", async () => {
const context = makeContext([makeTool("base_tool"), makeTool("late_tool")]);
const payload = await capturePayload<OpenAIPayload>(getModel("openai", "gpt-5.4"), context);
const searchCall = payload.input?.find((item): item is OpenAIToolSearchCall => item.type === "tool_search_call");
const searchOutput = payload.input?.find(
(item): item is OpenAIToolSearchOutput => item.type === "tool_search_output",
);
expect(openAIToolNames(payload)).toEqual(["base_tool"]);
expect(searchCall).toMatchObject({ execution: "client", status: "completed" });
expect(searchOutput?.call_id).toBe(searchCall?.call_id);
expect(searchOutput?.tools).toMatchObject([{ type: "function", name: "late_tool", defer_loading: true }]);
});
it.each(["gpt-5.2", "gpt-5.4-nano", "gpt-5.5-pro"] as const)(
"uses the normal tool list for unsupported OpenAI model %s",
async (modelId) => {
const context = makeContext([makeTool("base_tool"), makeTool("late_tool")]);
const payload = await capturePayload<OpenAIPayload>(getModel("openai", modelId), context);
expect(openAIToolNames(payload)).toEqual(["base_tool", "late_tool"]);
expect(payload.input?.some((item) => item.type === "tool_search_output")).toBe(false);
},
);
it("uses the normal tool list when OpenAI tool search is explicitly disabled", async () => {
const model: Model<"openai-responses"> = {
...getModel("openai", "gpt-5.4"),
provider: "openai-proxy",
compat: { supportsToolSearch: false },
};
const context = makeContext([makeTool("base_tool"), makeTool("late_tool")]);
const payload = await capturePayload<OpenAIPayload>(model, context);
expect(openAIToolNames(payload)).toEqual(["base_tool", "late_tool"]);
expect(payload.input?.some((item) => item.type === "tool_search_output")).toBe(false);
});
it("uses tool search only for supported Codex models", async () => {
const context = makeContext([makeTool("base_tool"), makeTool("late_tool")]);
const supported = await capturePayload<OpenAIPayload>(
getModel("openai-codex", "gpt-5.4"),
context,
makeCodexToken(),
);
const unsupported = await capturePayload<OpenAIPayload>(
getModel("openai-codex", "gpt-5.3-codex-spark"),
context,
makeCodexToken(),
);
expect(openAIToolNames(supported)).toEqual(["base_tool"]);
expect(supported.input?.some((item) => item.type === "tool_search_output")).toBe(true);
expect(openAIToolNames(unsupported)).toEqual(["base_tool", "late_tool"]);
expect(unsupported.input?.some((item) => item.type === "tool_search_output")).toBe(false);
});
it("leaves providers without deferred loading unchanged", async () => {
const context = makeContext([makeTool("base_tool"), makeTool("late_tool")]);
const payload = await capturePayload<OpenAIPayload>(getModel("groq", "llama-3.3-70b-versatile"), context);
expect(openAIToolNames(payload)).toEqual(["base_tool", "late_tool"]);
});
it("counts definitions marked after the latest usage checkpoint", () => {
const assistant: AssistantMessage = {
...makeAssistantToolCall(),
content: [{ type: "text", text: "done" }],
usage: {
input: 50,
output: 50,
cacheRead: 0,
cacheWrite: 0,
totalTokens: 100,
cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0, total: 0 },
},
stopReason: "stop",
};
const plain = estimateContextTokens({ messages: [assistant, makeUserMessage(4)], tools: [] });
const lateTool = { ...makeTool("late_tool"), description: "x".repeat(4000) };
const marked = estimateContextTokens({
messages: [assistant, makeToolResult(["late_tool"])],
tools: [lateTool],
});
expect(marked.tokens).toBeGreaterThan(plain.tokens + 500);
expect(marked.trailingTokens).toBeGreaterThan(plain.trailingTokens + 500);
});
});
packages/ai/test/models-runtime.test.ts
+120 -13
View File
@@ -1,7 +1,8 @@
import { describe, expect, it } from "vitest";
import { InMemoryCredentialStore } from "../src/auth/credential-store.ts";
import type { ApiKeyAuth, CredentialStore, OAuthAuth, ProviderAuth } from "../src/auth/types.ts";
import { calculateCost, createModels, hasApi, type Provider } from "../src/models.ts";
import { calculateCost, createModels, createProvider, hasApi, type Provider } from "../src/models.ts";
import { InMemoryModelsStore } from "../src/models-store.ts";
import type { Api, AssistantMessage, Context, Model, SimpleStreamOptions, StreamOptions, Usage } from "../src/types.ts";
import { AssistantMessageEventStream } from "../src/utils/event-stream.ts";
@@ -56,7 +57,7 @@ function testProvider(input: {
models?: Model<Api>[];
auth?: ProviderAuth;
getModels?: () => readonly Model<Api>[];
refreshModels?: () => Promise<void>;
refreshModels?: Provider["refreshModels"];
calls?: ProviderCall[];
}): Provider {
const models = input.models ?? [testModel(input.id, "model-a")];
@@ -215,7 +216,7 @@ describe("Models runtime", () => {
expect(() => models.getProvider("broken")?.getModels()).toThrow("boom");
});
it("refresh() updates dynamic providers; single-provider refresh failures reject", async () => {
it("refresh() updates every configured dynamic provider and reports failures", async () => {
let list = [testModel("dyn", "before")];
let refreshes = 0;
const models = createModels();
@@ -232,17 +233,12 @@ describe("Models runtime", () => {
models.setProvider(testProvider({ id: "static", models: [testModel("static", "s1")] }));
expect(models.getModel("dyn", "before")).toBeDefined();
await models.refresh("dyn");
const first = await models.refresh();
expect(first.errors.size).toBe(0);
expect(refreshes).toBe(1);
expect(models.getModel("dyn", "after")).toBeDefined();
expect(models.getModel("dyn", "before")).toBeUndefined();
// static providers are no-ops; refresh-all is best-effort
await models.refresh("static");
await models.refresh();
expect(refreshes).toBe(2);
// single-provider refresh failures reject with ModelsError
models.setProvider(
testProvider({
id: "flaky",
@@ -251,9 +247,120 @@ describe("Models runtime", () => {
},
}),
);
await expect(models.refresh("flaky")).rejects.toMatchObject({ code: "model_source" });
// refresh-all swallows the same failure
await expect(models.refresh()).resolves.toBeUndefined();
const second = await models.refresh();
expect(refreshes).toBe(2);
expect(second.errors.get("flaky")?.message).toBe("fetch failed");
});
it("persists dynamic catalogs and restores them without network access", async () => {
const credentials = new InMemoryCredentialStore();
const modelsStore = new InMemoryModelsStore();
await credentials.modify("dynamic", async () => ({ type: "api_key", key: "key" }));
const createDynamicProvider = (fetchModels: (() => Promise<readonly Model<Api>[]>) | undefined) =>
createProvider({
id: "dynamic",
auth: { apiKey: envKeyAuth(undefined) },
models: [],
fetchModels: fetchModels ? () => fetchModels() : undefined,
api: {
stream: () => new AssistantMessageEventStream(),
streamSimple: () => new AssistantMessageEventStream(),
},
});
const online = createModels({ credentials, modelsStore });
online.setProvider(createDynamicProvider(async () => [testModel("dynamic", "fetched")]));
expect((await online.refresh()).errors.size).toBe(0);
expect(online.getModel("dynamic", "fetched")).toBeDefined();
const offline = createModels({ credentials, modelsStore });
offline.setProvider(
createDynamicProvider(async () => {
throw new Error("must not fetch");
}),
);
expect((await offline.refresh({ allowNetwork: false })).errors.size).toBe(0);
expect(offline.getModel("dynamic", "fetched")).toBeDefined();
});
it("passes effective API-key credentials and skips unconfigured providers", async () => {
let effectiveCredential: unknown;
let unconfiguredRefreshes = 0;
const models = createModels();
models.setProvider(
testProvider({
id: "configured",
auth: { apiKey: envKeyAuth("ambient-key") },
refreshModels: async (context) => {
effectiveCredential = context.credential;
},
}),
);
models.setProvider(
testProvider({
id: "unconfigured",
auth: { apiKey: envKeyAuth(undefined) },
refreshModels: async () => {
unconfiguredRefreshes++;
},
}),
);
await models.refresh();
expect(effectiveCredential).toEqual({ type: "api_key", key: "ambient-key", env: undefined });
expect(unconfiguredRefreshes).toBe(0);
});
it("refreshes expired OAuth before refreshing models", async () => {
const credentials = new InMemoryCredentialStore();
let modelRefreshCredential: unknown;
await credentials.modify("oauth-dynamic", async () => ({
type: "oauth",
access: "expired",
refresh: "refresh",
expires: 0,
}));
const models = createModels({ credentials });
models.setProvider(
testProvider({
id: "oauth-dynamic",
auth: {
oauth: testOAuth({
refresh: async () => ({
type: "oauth",
access: "fresh",
refresh: "rotated",
expires: Date.now() + 60_000,
}),
}),
},
refreshModels: async (context) => {
modelRefreshCredential = context.credential;
},
}),
);
expect((await models.refresh()).errors.size).toBe(0);
expect(modelRefreshCredential).toMatchObject({ type: "oauth", access: "fresh", refresh: "rotated" });
expect(await credentials.read("oauth-dynamic")).toMatchObject({ access: "fresh", refresh: "rotated" });
});
it("returns aborted state without reporting cancellation as a provider error", async () => {
const controller = new AbortController();
const models = createModels();
models.setProvider(
testProvider({
id: "dynamic",
refreshModels: async ({ signal }) => {
controller.abort();
if (signal?.aborted) return;
},
}),
);
const result = await models.refresh({ signal: controller.signal });
expect(result.aborted).toBe(true);
expect(result.errors.size).toBe(0);
});
it("resolves auth: stored credential owns the provider, ambient only when nothing stored", async () => {
packages/ai/test/openai-codex-stream.test.ts
+56
View File
@@ -2,6 +2,7 @@ import { mkdtempSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { zstdDecompressSync } from "node:zlib";
import { Type } from "typebox";
import { afterEach, describe, expect, it, vi } from "vitest";
import {
closeOpenAICodexWebSocketSessions,
@@ -700,6 +701,61 @@ describe("openai-codex streaming", () => {
expect(requestedReasoning).toEqual({ effort: "xhigh", summary: "auto" });
});
it("forwards required tool choice", async () => {
const token = mockToken();
const encoder = new TextEncoder();
const sse = buildSSEPayload({ status: "completed" });
let requestedToolChoice: unknown;
vi.stubGlobal(
"fetch",
vi.fn(async (_input: string | URL, init?: RequestInit) => {
requestedToolChoice = decodeCodexRequestBody(init?.body)?.tool_choice;
return new Response(
new ReadableStream<Uint8Array>({
start(controller) {
controller.enqueue(encoder.encode(sse));
controller.close();
},
}),
{ status: 200, headers: { "content-type": "text/event-stream" } },
);
}),
);
const model: Model<"openai-codex-responses"> = {
id: "gpt-5.5",
name: "GPT-5.5",
api: "openai-codex-responses",
provider: "openai-codex",
baseUrl: "https://chatgpt.com/backend-api",
reasoning: true,
input: ["text"],
cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 },
contextWindow: 400000,
maxTokens: 128000,
};
await streamOpenAICodexResponses(
model,
{
messages: [
{ role: "user", content: "Do not call ping. Respond with text instead.", timestamp: Date.now() },
],
tools: [
{
name: "ping",
description: "Ping",
parameters: Type.Object({ value: Type.String() }),
},
],
},
{ apiKey: token, transport: "sse", toolChoice: "required" },
).result();
expect(requestedToolChoice).toBe("required");
});
it.each(["gpt-5.3-codex", "gpt-5.4", "gpt-5.5"])("clamps %s minimal reasoning effort to low", async (modelId) => {
const tempDir = mkdtempSync(join(tmpdir(), "pi-codex-stream-"));
process.env.PI_CODING_AGENT_DIR = tempDir;
packages/ai/test/openai-completions-prompt-cache.test.ts
+58
View File
@@ -13,6 +13,7 @@ interface FakeOpenAIClientOptions {
interface CapturedCompletionsPayload {
prompt_cache_key?: string;
prompt_cache_retention?: "24h" | "in-memory" | null;
session_id?: string;
}
const mockState = vi.hoisted(() => ({
@@ -170,6 +171,63 @@ describe("openai-completions prompt caching", () => {
expect(headers["x-session-affinity"]).toBe("session-affinity");
});
it("uses OpenAI no-session format when configured", async () => {
const model = createModel({
compat: { sendSessionAffinityHeaders: true, sessionAffinityFormat: "openai-nosession" },
});
const { payload, headers } = await captureRequest({ sessionId: "session-nosession" }, model);
expect(payload?.session_id).toBeUndefined();
expect(payload?.prompt_cache_key).toBe("session-nosession");
expect(headers.session_id).toBeUndefined();
expect(headers["x-client-request-id"]).toBe("session-nosession");
expect(headers["x-session-affinity"]).toBe("session-nosession");
expect(headers["x-session-id"]).toBeUndefined();
});
it("uses OpenRouter session-affinity header when configured", async () => {
const model = createModel({
baseUrl: "https://proxy.example.com/v1",
compat: { sendSessionAffinityHeaders: true, sessionAffinityFormat: "openrouter" },
});
const { payload, headers } = await captureRequest({ sessionId: "session-proxy" }, model);
expect(payload?.session_id).toBeUndefined();
expect(payload?.prompt_cache_key).toBeUndefined();
expect(headers["x-session-id"]).toBe("session-proxy");
expect(headers.session_id).toBeUndefined();
expect(headers["x-client-request-id"]).toBeUndefined();
expect(headers["x-session-affinity"]).toBeUndefined();
});
it("auto-detects OpenRouter session-affinity header for OpenRouter endpoints", async () => {
const model = createModel({
provider: "openrouter",
baseUrl: "https://openrouter.ai/api/v1",
compat: { sendSessionAffinityHeaders: true },
});
const { payload, headers } = await captureRequest({ sessionId: "session-openrouter" }, model);
expect(payload?.session_id).toBeUndefined();
expect(payload?.prompt_cache_key).toBeUndefined();
expect(headers["x-session-id"]).toBe("session-openrouter");
expect(headers.session_id).toBeUndefined();
expect(headers["x-client-request-id"]).toBeUndefined();
expect(headers["x-session-affinity"]).toBeUndefined();
});
it("omits OpenRouter session-affinity data when disabled", async () => {
const model = createModel({
provider: "openrouter",
baseUrl: "https://openrouter.ai/api/v1",
});
const { payload, headers } = await captureRequest({ sessionId: "session-openrouter" }, model);
expect(payload?.session_id).toBeUndefined();
expect(payload?.prompt_cache_key).toBeUndefined();
expect(headers["x-session-id"]).toBeUndefined();
});
it("omits session-affinity headers when cacheRetention is none", async () => {
const model = createModel({
baseUrl: "https://proxy.example.com/v1",
packages/ai/test/openai-completions-thinking-as-text.test.ts
+1
View File
@@ -39,6 +39,7 @@ const compat = {
supportsStrictMode: true,
cacheControlFormat: undefined,
sendSessionAffinityHeaders: false,
sessionAffinityFormat: "openai",
supportsLongCacheRetention: true,
} satisfies Required<Omit<OpenAICompletionsCompat, "cacheControlFormat">> & {
cacheControlFormat?: OpenAICompletionsCompat["cacheControlFormat"];
packages/ai/test/openai-completions-tool-choice.test.ts
+1
View File
@@ -1247,6 +1247,7 @@ describe("openai-completions tool_choice", () => {
zaiToolStream: false,
supportsStrictMode: true,
sendSessionAffinityHeaders: false,
sessionAffinityFormat: "openai",
supportsLongCacheRetention: true,
},
);
packages/ai/test/openai-completions-tool-result-images.test.ts
+1
View File
@@ -37,6 +37,7 @@ const compat: Required<OpenAICompletionsCompat> = {
supportsStrictMode: true,
cacheControlFormat: "anthropic",
sendSessionAffinityHeaders: false,
sessionAffinityFormat: "openai",
supportsLongCacheRetention: true,
};
packages/ai/test/openai-responses-copilot-provider.test.ts → packages/ai/test/openai-responses-compat.test.ts
+197 -54
View File
@@ -1,3 +1,4 @@
import { Type } from "typebox";
import { afterEach, describe, expect, it, vi } from "vitest";
import { stream as streamOpenAIResponses } from "../src/api/openai-responses.ts";
import { getModel } from "../src/compat.ts";
@@ -5,6 +6,11 @@ import type { Model } from "../src/types.ts";
type CapturedHeaders = Headers | string[][] | Record<string, string | readonly string[]> | undefined;
interface CapturedResponsesPayload {
prompt_cache_key?: string;
session_id?: string;
}
function getHeader(headers: CapturedHeaders, name: string): string | null {
if (!headers) return null;
if (headers instanceof Headers) return headers.get(name);
@@ -24,11 +30,20 @@ function getHeader(headers: CapturedHeaders, name: string): string | null {
async function captureOpenAIResponseHeaders(
options: Parameters<typeof streamOpenAIResponses>[2],
model: Model<"openai-responses"> = getModel("openai", "gpt-5.4"),
): Promise<{ sessionId: string | null; clientRequestId: string | null }> {
const captured = { sessionId: null as string | null, clientRequestId: null as string | null };
): Promise<{
sessionId: string | null;
clientRequestId: string | null;
xSessionId: string | null;
}> {
const captured = {
sessionId: null as string | null,
clientRequestId: null as string | null,
xSessionId: null as string | null,
};
vi.spyOn(globalThis, "fetch").mockImplementation(async (_input, init) => {
captured.sessionId = getHeader(init?.headers, "session_id");
captured.clientRequestId = getHeader(init?.headers, "x-client-request-id");
captured.xSessionId = getHeader(init?.headers, "x-session-id");
return new Response("data: [DONE]\n\n", {
status: 200,
headers: { "content-type": "text/event-stream" },
@@ -91,6 +106,53 @@ describe("openai-responses provider defaults", () => {
});
});
it("forwards required tool choice", async () => {
let capturedPayload: unknown;
vi.spyOn(globalThis, "fetch").mockResolvedValue(
new Response("data: [DONE]\n\n", {
status: 200,
headers: { "content-type": "text/event-stream" },
}),
);
const stream = streamOpenAIResponses(
getModel("openai", "gpt-5.4"),
{
messages: [
{
role: "user",
content: "Do not call ping. Respond with text instead.",
timestamp: Date.now(),
},
],
tools: [
{
name: "ping",
description: "Ping",
parameters: Type.Object({ value: Type.String() }),
},
],
},
{
apiKey: "test-key",
toolChoice: "required",
onPayload: (payload) => {
capturedPayload = payload;
},
},
);
for await (const event of stream) {
if (event.type === "done" || event.type === "error") break;
}
expect(capturedPayload).toMatchObject({
tool_choice: "required",
tools: [expect.objectContaining({ name: "ping" })],
});
});
it.each([
"gpt-5.1",
"gpt-5.2",
@@ -176,12 +238,13 @@ describe("openai-responses provider defaults", () => {
it("sets cache-affinity headers for official OpenAI Responses requests with a sessionId", async () => {
const captured = await captureOpenAIResponseHeaders({ sessionId: "session-123" });
expect(captured).toEqual({ sessionId: "session-123", clientRequestId: "session-123" });
expect(captured.sessionId).toBe("session-123");
expect(captured.clientRequestId).toBe("session-123");
});
it("clamps prompt_cache_key to OpenAI's 64-character limit", async () => {
const sessionId = "x".repeat(67);
let capturedPayload: { prompt_cache_key?: string } | undefined;
let capturedPayload: Pick<CapturedResponsesPayload, "prompt_cache_key"> | undefined;
vi.spyOn(globalThis, "fetch").mockResolvedValue(
new Response("data: [DONE]\n\n", {
status: 200,
@@ -199,7 +262,7 @@ describe("openai-responses provider defaults", () => {
apiKey: "test-key",
sessionId,
onPayload: (payload) => {
capturedPayload = payload as { prompt_cache_key?: string };
capturedPayload = payload as Pick<CapturedResponsesPayload, "prompt_cache_key">;
},
},
);
@@ -219,19 +282,125 @@ describe("openai-responses provider defaults", () => {
};
const captured = await captureOpenAIResponseHeaders({ sessionId: "session-123" }, proxyModel);
expect(captured).toEqual({ sessionId: "session-123", clientRequestId: "session-123" });
expect(captured.sessionId).toBe("session-123");
expect(captured.clientRequestId).toBe("session-123");
});
it("can omit the session_id header while preserving other cache-affinity headers", async () => {
it("uses OpenRouter session-affinity header when configured", async () => {
const proxyModel: Model<"openai-responses"> = {
...getModel("openai", "gpt-5.4"),
provider: "proxy",
baseUrl: "https://proxy.example.com/v1",
compat: { sessionAffinityFormat: "openrouter" },
};
let capturedPayload: CapturedResponsesPayload | undefined;
const captured = await captureOpenAIResponseHeaders(
{
sessionId: "session-proxy",
onPayload: (payload) => {
capturedPayload = payload as CapturedResponsesPayload;
},
},
proxyModel,
);
expect(captured.sessionId).toBeNull();
expect(captured.clientRequestId).toBeNull();
expect(captured.xSessionId).toBe("session-proxy");
expect(capturedPayload?.session_id).toBeUndefined();
expect(capturedPayload?.prompt_cache_key).toBe("session-proxy");
});
it("auto-detects OpenRouter session-affinity header for OpenRouter Responses endpoints", async () => {
const openRouterModel: Model<"openai-responses"> = {
...getModel("openai", "gpt-5.4"),
provider: "openrouter",
baseUrl: "https://openrouter.ai/api/v1",
};
let capturedPayload: CapturedResponsesPayload | undefined;
const captured = await captureOpenAIResponseHeaders(
{
sessionId: "session-openrouter",
onPayload: (payload) => {
capturedPayload = payload as CapturedResponsesPayload;
},
},
openRouterModel,
);
expect(captured.sessionId).toBeNull();
expect(captured.clientRequestId).toBeNull();
expect(captured.xSessionId).toBe("session-openrouter");
expect(capturedPayload?.session_id).toBeUndefined();
expect(capturedPayload?.prompt_cache_key).toBe("session-openrouter");
});
it("uses OpenAI no-session format when configured", async () => {
const proxyModel: Model<"openai-responses"> = {
...getModel("openai", "gpt-5.4"),
provider: "proxy",
baseUrl: "https://proxy.example.com/v1",
compat: { sessionAffinityFormat: "openai-nosession" },
};
let capturedPayload: CapturedResponsesPayload | undefined;
const captured = await captureOpenAIResponseHeaders(
{
sessionId: "session-proxy",
onPayload: (payload) => {
capturedPayload = payload as CapturedResponsesPayload;
},
},
proxyModel,
);
expect(captured.sessionId).toBeNull();
expect(captured.clientRequestId).toBe("session-proxy");
expect(captured.xSessionId).toBeNull();
expect(capturedPayload?.session_id).toBeUndefined();
expect(capturedPayload?.prompt_cache_key).toBe("session-proxy");
});
it("uses OpenAI no-session format for OpenCode Responses models", async () => {
const model = getModel("opencode", "gpt-5.4");
let capturedPayload: CapturedResponsesPayload | undefined;
const captured = await captureOpenAIResponseHeaders(
{
sessionId: "session-opencode",
onPayload: (payload) => {
capturedPayload = payload as CapturedResponsesPayload;
},
},
model,
);
expect(model.compat?.sessionAffinityFormat).toBe("openai-nosession");
expect(captured.sessionId).toBeNull();
expect(captured.clientRequestId).toBe("session-opencode");
expect(captured.xSessionId).toBeNull();
expect(capturedPayload?.prompt_cache_key).toBe("session-opencode");
});
it("can omit OpenAI session_id header while preserving other affinity data", async () => {
const proxyModel: Model<"openai-responses"> = {
...getModel("openai", "gpt-5.4"),
provider: "opencode",
baseUrl: "https://proxy.example.com/v1",
compat: { sendSessionIdHeader: false },
compat: { sessionAffinityFormat: "openai-nosession" },
};
const captured = await captureOpenAIResponseHeaders({ sessionId: "session-123" }, proxyModel);
let capturedPayload: CapturedResponsesPayload | undefined;
const captured = await captureOpenAIResponseHeaders(
{
sessionId: "session-123",
onPayload: (payload) => {
capturedPayload = payload as CapturedResponsesPayload;
},
},
proxyModel,
);
expect(captured).toEqual({ sessionId: null, clientRequestId: "session-123" });
expect(captured.sessionId).toBeNull();
expect(captured.clientRequestId).toBe("session-123");
expect(capturedPayload?.prompt_cache_key).toBe("session-123");
});
it("lets explicit headers override the default OpenAI cache-affinity headers", async () => {
@@ -243,22 +412,25 @@ describe("openai-responses provider defaults", () => {
},
});
expect(captured).toEqual({ sessionId: "override-session", clientRequestId: "override-request" });
expect(captured.sessionId).toBe("override-session");
expect(captured.clientRequestId).toBe("override-request");
});
it("omits OpenAI cache-affinity headers when cacheRetention is none", async () => {
const captured = await captureOpenAIResponseHeaders({ cacheRetention: "none", sessionId: "session-123" });
expect(captured).toEqual({ sessionId: null, clientRequestId: null });
expect(captured.sessionId).toBeNull();
expect(captured.clientRequestId).toBeNull();
});
async function streamServiceTierUsage(
modelId: "gpt-5.4" | "gpt-5.5",
serviceTier: "priority" | "flex",
inputTokens: number,
outputTokens: number,
) {
it.each([
["gpt-5.4", "priority", 2],
["gpt-5.5", "priority", 2.5],
["gpt-5.5", "flex", 0.5],
] as const)("applies %s %s service-tier cost multiplier", async (modelId, serviceTier, multiplier) => {
const model = getModel("openai", modelId);
const tokenCount = 100_000;
const tokenScale = tokenCount / 1_000_000;
const sse = `${[
`data: ${JSON.stringify({
type: "response.completed",
@@ -266,9 +438,9 @@ describe("openai-responses provider defaults", () => {
status: "completed",
service_tier: serviceTier,
usage: {
input_tokens: inputTokens,
output_tokens: outputTokens,
total_tokens: inputTokens + outputTokens,
input_tokens: tokenCount,
output_tokens: tokenCount,
total_tokens: tokenCount * 2,
input_tokens_details: { cached_tokens: 0 },
},
},
@@ -291,39 +463,10 @@ describe("openai-responses provider defaults", () => {
{ apiKey: "test-key", serviceTier },
);
return { model, result: await stream.result() };
}
const result = await stream.result();
it.each([
["gpt-5.4", "priority", 2],
["gpt-5.5", "priority", 2.5],
["gpt-5.5", "flex", 0.5],
] as const)("applies %s %s service-tier cost multiplier", async (modelId, serviceTier, multiplier) => {
// Stay below the 272K long-context tier threshold so base rates apply.
const inputTokens = 200000;
const outputTokens = 100000;
const { model, result } = await streamServiceTierUsage(modelId, serviceTier, inputTokens, outputTokens);
const expectedInput = (model.cost.input / 1_000_000) * inputTokens * multiplier;
const expectedOutput = (model.cost.output / 1_000_000) * outputTokens * multiplier;
expect(result.usage.cost.input).toBe(expectedInput);
expect(result.usage.cost.output).toBe(expectedOutput);
expect(result.usage.cost.total).toBe(expectedInput + expectedOutput);
});
it("applies the service-tier multiplier on top of long-context tier pricing", async () => {
// Above the 272K input threshold the long-context tier rates apply, then the multiplier.
const inputTokens = 1000000;
const outputTokens = 100000;
const multiplier = 2;
const { model, result } = await streamServiceTierUsage("gpt-5.4", "priority", inputTokens, outputTokens);
const tier = model.cost.tiers?.find((entry) => inputTokens > entry.inputTokensAbove);
if (!tier) throw new Error("expected gpt-5.4 to define a long-context pricing tier");
const expectedInput = (tier.input / 1_000_000) * inputTokens * multiplier;
const expectedOutput = (tier.output / 1_000_000) * outputTokens * multiplier;
expect(result.usage.cost.input).toBe(expectedInput);
expect(result.usage.cost.output).toBe(expectedOutput);
expect(result.usage.cost.total).toBe(expectedInput + expectedOutput);
expect(result.usage.cost.input).toBe(model.cost.input * multiplier * tokenScale);
expect(result.usage.cost.output).toBe(model.cost.output * multiplier * tokenScale);
expect(result.usage.cost.total).toBe((model.cost.input + model.cost.output) * multiplier * tokenScale);
});
});
packages/ai/test/pi-messages.test.ts
+243
View File
@@ -0,0 +1,243 @@
import { createServer, type IncomingMessage, type Server, type ServerResponse } from "node:http";
import type { AddressInfo } from "node:net";
import { afterEach, describe, expect, it } from "vitest";
import { type PiMessagesOptions, stream, streamSimple } from "../src/api/pi-messages.ts";
import type { Api, AssistantMessageEvent, Context, Model } from "../src/types.ts";
type RecordedRequest = {
url: string;
headers: IncomingMessage["headers"];
body: unknown;
};
type ResponderOptions = {
status?: number;
headers?: Record<string, string>;
events?: unknown[];
rawBody?: string;
};
let server: Server | undefined;
afterEach(() => {
server?.close();
server = undefined;
});
async function startServer(options: ResponderOptions): Promise<{ baseUrl: string; requests: RecordedRequest[] }> {
const requests: RecordedRequest[] = [];
server = createServer((request: IncomingMessage, response: ServerResponse) => {
const chunks: Buffer[] = [];
request.on("data", (chunk: Buffer) => chunks.push(chunk));
request.on("end", () => {
const raw = Buffer.concat(chunks).toString("utf-8");
requests.push({
url: request.url ?? "",
headers: request.headers,
body: raw ? JSON.parse(raw) : undefined,
});
if (options.status && options.status !== 200) {
response.statusCode = options.status;
response.setHeader("content-type", "application/json");
response.end(options.rawBody ?? "{}");
return;
}
response.statusCode = 200;
response.setHeader("content-type", "text/event-stream");
for (const [name, value] of Object.entries(options.headers ?? {})) {
response.setHeader(name, value);
}
for (const event of options.events ?? []) {
response.write(`data: ${JSON.stringify(event)}\n\n`);
}
response.end();
});
});
await new Promise<void>((resolve) => {
server!.listen(0, "127.0.0.1", () => resolve());
});
const address = server!.address() as AddressInfo;
return { baseUrl: `http://127.0.0.1:${address.port}/v1`, requests };
}
function createModel(baseUrl: string): Model<"pi-messages"> {
return {
id: "auto",
name: "Radius Auto",
api: "pi-messages",
provider: "radius",
baseUrl,
reasoning: false,
input: ["text"],
cost: { input: 1, output: 2, cacheRead: 0.1, cacheWrite: 0.2 },
contextWindow: 128000,
maxTokens: 16384,
};
}
const context: Context = {
messages: [{ role: "user", content: "Hello", timestamp: Date.now() }],
};
const usage = {
input: 10,
output: 5,
cacheRead: 0,
cacheWrite: 0,
totalTokens: 15,
cost: { input: 0.1, output: 0.2, cacheRead: 0, cacheWrite: 0, total: 0.3 },
};
describe("pi-messages", () => {
it("streams text and tool calls and resolves the terminal message", async () => {
const { baseUrl, requests } = await startServer({
events: [
{ type: "start" },
{ type: "text_start", contentIndex: 0 },
{ type: "text_delta", contentIndex: 0, delta: "Hel" },
{ type: "text_delta", contentIndex: 0, delta: "lo" },
{ type: "text_end", contentIndex: 0, content: "Hello" },
{ type: "toolcall_start", contentIndex: 1, id: "call_1", toolName: "read" },
{ type: "toolcall_delta", contentIndex: 1, delta: '{"path":' },
{ type: "toolcall_delta", contentIndex: 1, delta: '"a.txt"}' },
{
type: "toolcall_end",
contentIndex: 1,
toolCall: { type: "toolCall", id: "call_1", name: "read", arguments: { path: "a.txt" } },
},
{ type: "done", reason: "toolUse", usage, responseId: "resp_1" },
],
});
const model = createModel(baseUrl);
const events: AssistantMessageEvent[] = [];
const eventStream = stream(model, context, {
apiKey: "test-key",
sessionId: "session-1",
toolChoice: "auto",
maxTokens: 100,
headers: { "x-custom": "1" },
});
for await (const event of eventStream) {
events.push(event);
}
const message = await eventStream.result();
expect(message.stopReason).toBe("toolUse");
expect(message.usage).toEqual(usage);
expect(message.responseId).toBe("resp_1");
expect(message.model).toBe("auto");
expect(message.provider).toBe("radius");
expect(message.content).toEqual([
{ type: "text", text: "Hello", textSignature: undefined },
{ type: "toolCall", id: "call_1", name: "read", arguments: { path: "a.txt" } },
]);
expect(events.some((event) => event.type === "text_delta")).toBe(true);
expect(events.filter((event) => event.type === "toolcall_end")).toHaveLength(1);
expect(requests).toHaveLength(1);
const request = requests[0];
expect(request.url).toBe("/v1/messages");
expect(request.headers.authorization).toBe("Bearer test-key");
expect(request.headers["x-custom"]).toBe("1");
expect(request.body).toEqual({
model: "auto",
context,
options: { maxTokens: 100, sessionId: "session-1", toolChoice: "auto" },
});
});
it("appends debug=1 and reports response headers via onResponse", async () => {
const { baseUrl, requests } = await startServer({
headers: { "x-pi-gateway-upstream-provider": "anthropic" },
events: [{ type: "done", reason: "stop", usage }],
});
const model = createModel(baseUrl);
let observedHeaders: Record<string, string> | undefined;
const options: PiMessagesOptions = {
apiKey: "test-key",
debug: true,
onResponse: (response) => {
observedHeaders = response.headers;
},
};
const message = await streamSimple(model, context, options).result();
expect(message.stopReason).toBe("stop");
expect(requests[0].url).toBe("/v1/messages?debug=1");
expect(observedHeaders?.["x-pi-gateway-upstream-provider"]).toBe("anthropic");
});
it("surfaces backend error responses with diagnostics", async () => {
const { baseUrl } = await startServer({
status: 401,
rawBody: JSON.stringify({ error: { message: "Token expired", code: "unauthorized" } }),
});
const model = createModel(baseUrl);
const message = await stream(model, context, { apiKey: "stale" }).result();
expect(message.stopReason).toBe("error");
expect(message.errorMessage).toContain("401");
expect(message.errorMessage).toContain("Token expired");
expect(message.errorMessage).toContain("unauthorized");
expect(message.diagnostics?.[0]?.type).toBe("pi_messages_response_failure");
expect(message.diagnostics?.[0]?.details?.status).toBe(401);
});
it("propagates server-sent error events", async () => {
const { baseUrl } = await startServer({
events: [{ type: "start" }, { type: "error", reason: "error", usage, errorMessage: "Upstream failed" }],
});
const model = createModel(baseUrl);
const message = await stream(model, context, { apiKey: "test-key" }).result();
expect(message.stopReason).toBe("error");
expect(message.errorMessage).toBe("Upstream failed");
expect(message.usage).toEqual(usage);
});
it("errors when no API key is provided", async () => {
const model = createModel("http://127.0.0.1:1/v1");
const message = await stream(model, context).result();
expect(message.stopReason).toBe("error");
expect(message.errorMessage).toContain("No API key provided");
});
it("errors when the stream ends without a terminal event", async () => {
const { baseUrl } = await startServer({
events: [
{ type: "start" },
{ type: "text_start", contentIndex: 0 },
{ type: "text_delta", contentIndex: 0, delta: "partial" },
],
});
const model = createModel(baseUrl);
const message = await stream(model, context, { apiKey: "test-key" }).result();
expect(message.stopReason).toBe("error");
expect(message.errorMessage).toContain("stream ended without a terminal event");
});
});
describe("pi-messages api registration", () => {
it("is registered as a builtin api provider", async () => {
const { getApiProvider } = await import("../src/compat.ts");
expect(getApiProvider("pi-messages")).toBeDefined();
});
it("is a known api usable on models", () => {
const api: Api = "pi-messages";
expect(api).toBe("pi-messages");
});
});
packages/ai/test/providers.test.ts
+17 -5
View File
@@ -2,6 +2,7 @@ import { describe, expect, it } from "vitest";
import { envApiKeyAuth } from "../src/auth/helpers.ts";
import type { AuthContext, AuthEvent } from "../src/auth/types.ts";
import { createModels, createProvider } from "../src/models.ts";
import { InMemoryModelsStore } from "../src/models-store.ts";
import { builtinModels, builtinProviders } from "../src/providers/all.ts";
import { amazonBedrockProvider } from "../src/providers/amazon-bedrock.ts";
import { anthropicProvider } from "../src/providers/anthropic.ts";
@@ -34,10 +35,11 @@ describe("builtin providers", () => {
const all = models.getModels();
expect(all.length).toBeGreaterThan(500);
// every provider lists at least one model and owns its models
// Static providers list models immediately; Radius is purely dynamic.
for (const provider of providers) {
const list = models.getModels(provider.id);
expect(list.length).toBeGreaterThan(0);
if (provider.id === "radius") expect(list).toEqual([]);
else expect(list.length).toBeGreaterThan(0);
expect(list.every((m) => m.provider === provider.id)).toBe(true);
}
});
@@ -345,7 +347,7 @@ describe("createProvider", () => {
id: "dynamic",
auth: { apiKey: { name: "Test", resolve: async () => ({ auth: {} }) } },
models: [],
refreshModels: async () => {
fetchModels: async () => {
fetches++;
await new Promise((resolve) => setTimeout(resolve, 5));
return [testModel("api-a", "listed")];
@@ -353,13 +355,23 @@ describe("createProvider", () => {
api: recordingStreams("a", []),
});
const store = new InMemoryModelsStore();
const refreshContext = {
credential: { type: "api_key" as const },
store: {
read: () => store.read("dynamic"),
write: (listed: readonly Model<Api>[]) => store.write("dynamic", listed),
delete: () => store.delete("dynamic"),
},
allowNetwork: true,
};
expect(provider.getModels()).toEqual([]);
await Promise.all([provider.refreshModels?.(), provider.refreshModels?.()]);
await Promise.all([provider.refreshModels?.(refreshContext), provider.refreshModels?.(refreshContext)]);
expect(fetches).toBe(1);
expect(provider.getModels().map((m) => m.id)).toEqual(["listed"]);
// a later refresh fetches again
await provider.refreshModels?.();
await provider.refreshModels?.(refreshContext);
expect(fetches).toBe(2);
});
});