Stream implementations move from src/providers/ to src/api/, renamed by
API id (anthropic.ts -> anthropic-messages.ts, google.ts ->
google-generative-ai.ts, mistral.ts -> mistral-conversations.ts,
amazon-bedrock.ts -> bedrock-converse-stream.ts). Every module now
exports exactly stream/streamSimple; shared helpers move alongside.
New ProviderStreams dispatch contract in types.ts, lazyApi() wrapper in
api/lazy.ts, and one .lazy.ts wrapper per API. Bedrock's wrapper keeps
the node-only variable-specifier import and setBedrockProviderModule()
(now taking ProviderStreams).
providers/register-builtins.ts deleted; interim until the compat
entrypoint lands, builtin api-registry registration lives in stream.ts
and lazy wrappers are exported from the root barrel. Old per-API lazy
exports (streamAnthropic, ...) are gone; package.json subpaths retarget
to dist/api/.
Fireworks prompt caching is enabled by default (automatic prefix matching),
but on serverless infrastructure, requests hit random replicas. Without
session affinity, the per-replica cache misses, negating cache hit rates
and the discounted cacheRead pricing.
Changes:
- Add sendSessionAffinityHeaders and supportsCacheControlOnTools
to AnthropicMessagesCompat interface
- Send x-session-affinity header for Fireworks (and Cloudflare AI
Gateway Anthropic) when sessionId is available and caching is enabled
- Omit cache_control on tool definitions for Fireworks (unsupported
per https://docs.fireworks.ai/tools-sdks/anthropic-compatibility)
- Default supportsEagerToolInputStreaming to false for Fireworks
(unsupported field)
- Default supportsLongCacheRetention to false for Fireworks
(cache_control.ttl not supported)
- Add compat settings to Fireworks models in generate-models.ts
- Update generated models with Fireworks compat settings
- Add integration tests for session affinity and tool compat
Refs: https://docs.fireworks.ai/guides/prompt-caching
Refs: https://docs.fireworks.ai/tools-sdks/anthropic-compatibility
Mario Zechner