feat(amazon-bedrock): conditionally omit maxTokens from inferenceConfig (#3400)
When model.maxTokens is 0 (unset/unknown), omit maxTokens from the Bedrock ConverseStream inferenceConfig instead of sending 0. Bedrock's inferenceConfig.maxTokens is optional — when omitted, the model uses its own internal default. This is optimal for Bedrock's token quota management: - At request start, Bedrock reserves input_tokens + max_tokens from your TPM quota - For Claude 3.7+ models, output tokens have a 5x burndown rate - Sending an unnecessarily high maxTokens wastes TPM capacity during the reservation window, reducing concurrent request throughput - Omitting it lets Bedrock use the model default (~4096 for Claude), matching expected output sizes and maximizing quota utilization Also applies the same conditional pattern to temperature — only include it in inferenceConfig when explicitly set. Changes: - simple-options.ts: Use ?? instead of || so explicit 0 is preserved; return undefined when model.maxTokens is 0 (unset) - amazon-bedrock.ts: Spread maxTokens and temperature into inferenceConfig only when defined Fixes #3399
This commit is contained in:
@@ -3,7 +3,7 @@ import type { Api, Model, SimpleStreamOptions, StreamOptions, ThinkingBudgets, T
|
||||
export function buildBaseOptions(model: Model<Api>, options?: SimpleStreamOptions, apiKey?: string): StreamOptions {
|
||||
return {
|
||||
temperature: options?.temperature,
|
||||
maxTokens: options?.maxTokens || Math.min(model.maxTokens, 32000),
|
||||
maxTokens: options?.maxTokens ?? (model.maxTokens > 0 ? Math.min(model.maxTokens, 32000) : undefined),
|
||||
signal: options?.signal,
|
||||
apiKey: apiKey || options?.apiKey,
|
||||
cacheRetention: options?.cacheRetention,
|
||||
|
||||
Reference in New Issue
Block a user