diff --git a/packages/coding-agent/CHANGELOG.md b/packages/coding-agent/CHANGELOG.md index fe2ae5e1..5e6099bb 100644 --- a/packages/coding-agent/CHANGELOG.md +++ b/packages/coding-agent/CHANGELOG.md @@ -4,6 +4,7 @@ ### Added +- Added built-in llama.cpp router support with `/login` connection setup and `/llama` model downloads, explicit loading, unloading, and live progress. See [llama.cpp](docs/llama-cpp.md). - Added extension registration for complete pi-ai providers, including native authentication, model refresh, filtering, and streaming behavior. ### Fixed diff --git a/packages/coding-agent/README.md b/packages/coding-agent/README.md index c452fd36..3454fac2 100644 --- a/packages/coding-agent/README.md +++ b/packages/coding-agent/README.md @@ -135,7 +135,9 @@ For each built-in provider, pi maintains a list of tool-capable models. Configur - Xiaomi MiMo Token Plan (Amsterdam) - Xiaomi MiMo Token Plan (Singapore) -See [docs/providers.md](docs/providers.md) for detailed setup instructions. +Pi also supports the llama.cpp router server. Configure it with `/login llama.cpp`, manage downloads and loaded models with `/llama`, then select a loaded model with `/model`. See [docs/llama-cpp.md](docs/llama-cpp.md) for setup and usage. + +See [docs/providers.md](docs/providers.md) for other provider setup instructions. **Custom providers & models:** Add providers via `~/.pi/agent/models.json` if they speak a supported API (OpenAI, Anthropic, Google). For custom APIs or OAuth, use extensions. See [docs/models.md](docs/models.md) and [docs/custom-provider.md](docs/custom-provider.md). @@ -173,7 +175,8 @@ Type `/` in the editor to trigger commands. [Extensions](#extensions) can regist | Command | Description | |---------|-------------| -| `/login`, `/logout` | OAuth authentication | +| `/login`, `/logout` | Manage provider credentials | +| [`/llama`](docs/llama-cpp.md) | Download, load, and unload llama.cpp router models | | `/model` | Switch models | | `/scoped-models` | Enable/disable models for Ctrl+P cycling | | `/settings` | Thinking level, theme, message delivery, transport | diff --git a/packages/coding-agent/docs/index.md b/packages/coding-agent/docs/index.md index b5ee017e..6b1e419e 100644 --- a/packages/coding-agent/docs/index.md +++ b/packages/coding-agent/docs/index.md @@ -41,6 +41,7 @@ For the full first-run flow, see [Quickstart](quickstart.md). - [Quickstart](quickstart.md) - install, authenticate, and run a first session. - [Using Pi](usage.md) - interactive mode, slash commands, context files, and CLI reference. - [Providers](providers.md) - subscription and API-key setup for built-in providers. +- [llama.cpp](llama-cpp.md) - run a local router and manage models with `/llama`. - [Security](security.md) - project trust, sandbox boundaries, and vulnerability reporting. - [Containerization](containerization.md) - sandbox pi with Gondolin, Docker, or OpenShell. - [Settings](settings.md) - global and project settings. diff --git a/packages/coding-agent/docs/llama-cpp.md b/packages/coding-agent/docs/llama-cpp.md new file mode 100644 index 00000000..ffdd4e37 --- /dev/null +++ b/packages/coding-agent/docs/llama-cpp.md @@ -0,0 +1,97 @@ +# llama.cpp + +Pi supports the [llama.cpp](https://github.com/ggml-org/llama.cpp) router server. The router discovers multiple GGUF models and loads or unloads them on demand. + +Use a current llama.cpp build with router support. Follow the [build instructions](https://github.com/ggml-org/llama.cpp/blob/master/docs/build.md) or install a [prebuilt release](https://github.com/ggml-org/llama.cpp/releases) for your platform. + +## Start the router + +Start `llama-server` without `--model` or `-m`. Passing a model starts single-model mode instead of router mode. + +```bash +llama-server \ + --models-dir ~/models \ + --no-models-autoload \ + --jinja \ + --host 127.0.0.1 \ + --port 8080 \ + -ngl 999 \ + -c 32768 +``` + +Important options: + +- `--models-dir ~/models` discovers local GGUF files. +- `--no-models-autoload` keeps loading explicit through `/llama`. +- `--jinja` enables compatible chat templates and tool calling. +- `-ngl 999` offloads as many layers as possible to the GPU. +- `-c 32768` sets the context window for each loaded model. Omit it to use the model's native context, which may require substantially more memory. + +A single-file model can sit directly in the model directory. Put multimodal and multi-shard models in separate subdirectories: + +```text +~/models/ +├── llama-3.2-1b-Q4_K_M.gguf +├── gemma-3-4b-it-Q4_K_M/ +│ ├── gemma-3-4b-it-Q4_K_M.gguf +│ └── mmproj-F16.gguf +└── large-model-Q4_K_M/ + ├── large-model-Q4_K_M-00001-of-00003.gguf + ├── large-model-Q4_K_M-00002-of-00003.gguf + └── large-model-Q4_K_M-00003-of-00003.gguf +``` + +Restart the router after manually adding files. For per-model context sizes and other options, use [llama.cpp model presets](https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md#model-presets). + +## Configure Pi + +Start Pi and configure the provider: + +```text +/login llama.cpp +``` + +Enter the router URL and optional API key. The default URL is `http://127.0.0.1:8080`. + +Environment variables can configure the same values without `/login`: + +```bash +export LLAMA_BASE_URL=http://127.0.0.1:8080 +export LLAMA_API_KEY=optional-secret +pi +``` + +If the server uses an API key, start `llama-server` with the matching `--api-key` value. Keep `--host 127.0.0.1` for local-only access. + +## Manage models + +Run: + +```text +/llama +``` + +- Select an unloaded model to load it. +- Select a loaded model to unload it. +- Select **Download model…** and enter `owner/repository[:quant]` to download from Hugging Face. +- Press Escape during a load or download to confirm cancellation. + +If other models are loaded, Pi asks whether to unload them first or keep them loaded. Pi does not silently unload models and never deletes model files. The router may be shared with other clients, so `/llama` always displays the router's current state. + +Only loaded models appear in `/model`. After loading a model, run `/model` to select it for the current Pi session. + +If the router disconnects, `/llama` shows **Retry** and **Close**. Retry reconnects and refreshes model state without replaying the interrupted operation. + +## Troubleshooting + +Check that the router is reachable: + +```bash +curl http://127.0.0.1:8080/health +curl http://127.0.0.1:8080/models +``` + +- **No models in `/llama`:** Check `--models-dir`, the directory layout, and restart the router. +- **Model missing from `/model`:** Load it with `/llama` first. +- **Load fails or uses too much memory:** Lower `-c` or unload another model. +- **Server is not in router mode:** Start it without `--model`, `-m`, or `-hf`. diff --git a/packages/coding-agent/docs/providers.md b/packages/coding-agent/docs/providers.md index d3f832ef..9517dd0c 100644 --- a/packages/coding-agent/docs/providers.md +++ b/packages/coding-agent/docs/providers.md @@ -8,6 +8,7 @@ Pi supports subscription-based providers via OAuth and API key providers via env - [API Keys](#api-keys) - [Auth File](#auth-file) - [Cloud Providers](#cloud-providers) +- [llama.cpp](#llamacpp) - [Custom Providers](#custom-providers) - [Resolution Order](#resolution-order) @@ -274,6 +275,12 @@ export GOOGLE_CLOUD_LOCATION=us-central1 Or set `GOOGLE_APPLICATION_CREDENTIALS` to a service account key file. +## llama.cpp + +Pi supports the llama.cpp router server. Configure it with `/login llama.cpp`, manage loaded models with `/llama`, and select a loaded model with `/model`. + +See [llama.cpp](llama-cpp.md) for server setup, model directory layout, environment variables, and command usage. + ## Custom Providers **Via models.json:** Add Ollama, LM Studio, vLLM, or any provider that speaks a supported API (OpenAI Completions, OpenAI Responses, Anthropic Messages, Google Generative AI). See [models.md](models.md). diff --git a/packages/coding-agent/docs/usage.md b/packages/coding-agent/docs/usage.md index 7d0b85ec..48cf16b1 100644 --- a/packages/coding-agent/docs/usage.md +++ b/packages/coding-agent/docs/usage.md @@ -37,6 +37,7 @@ Type `/` in the editor to open command completion. Extensions can register custo | Command | Description | |---------|-------------| | `/login`, `/logout` | Manage OAuth or API-key credentials | +| [`/llama`](llama-cpp.md) | Download, load, and unload llama.cpp router models | | `/model` | Switch models | | `/scoped-models` | Enable/disable models for Ctrl+P cycling | | `/settings` | Thinking level, theme, message delivery, transport | diff --git a/packages/coding-agent/src/core/extensions/types.ts b/packages/coding-agent/src/core/extensions/types.ts index 6eb12466..514f7af6 100644 --- a/packages/coding-agent/src/core/extensions/types.ts +++ b/packages/coding-agent/src/core/extensions/types.ts @@ -1482,6 +1482,8 @@ export type InlineExtension = /** Display name shown as `` in the startup Extensions list. */ name: string; factory: ExtensionFactory; + /** Omit this extension from the startup Extensions list. */ + hidden?: boolean; }; // ============================================================================ @@ -1649,6 +1651,7 @@ export interface ExtensionRuntime extends ExtensionRuntimeState, ExtensionAction export interface Extension { path: string; resolvedPath: string; + hidden?: boolean; sourceInfo: SourceInfo; handlers: Map; tools: Map; diff --git a/packages/coding-agent/src/core/resource-loader.ts b/packages/coding-agent/src/core/resource-loader.ts index e66d9771..c8b6e2c2 100644 --- a/packages/coding-agent/src/core/resource-loader.ts +++ b/packages/coding-agent/src/core/resource-loader.ts @@ -899,6 +899,7 @@ export class DefaultResourceLoader implements ResourceLoader { const extensionPath = ``; try { const extension = await loadExtensionFromFactory(factory, this.cwd, this.eventBus, runtime, extensionPath); + extension.hidden = isNamed && input.hidden; extensions.push(extension); } catch (error) { const message = error instanceof Error ? error.message : "failed to load extension"; diff --git a/packages/coding-agent/src/extensions/index.ts b/packages/coding-agent/src/extensions/index.ts new file mode 100644 index 00000000..a734df7d --- /dev/null +++ b/packages/coding-agent/src/extensions/index.ts @@ -0,0 +1,4 @@ +import type { InlineExtension } from "../core/extensions/types.ts"; +import llamaExtension from "./llama/index.ts"; + +export const builtInExtensions: InlineExtension[] = [{ name: "llama.cpp", factory: llamaExtension, hidden: true }]; diff --git a/packages/coding-agent/src/extensions/llama/client.ts b/packages/coding-agent/src/extensions/llama/client.ts new file mode 100644 index 00000000..0e58e1bd --- /dev/null +++ b/packages/coding-agent/src/extensions/llama/client.ts @@ -0,0 +1,330 @@ +export type LlamaModelStatus = "unloaded" | "loading" | "loaded" | "downloading" | "sleeping"; + +export interface LlamaModelInfo { + id: string; + aliases?: string[]; + status: { + value: LlamaModelStatus; + args?: string[]; + failed?: boolean; + exit_code?: number; + progress?: Record; + }; + architecture?: { + input_modalities?: string[]; + output_modalities?: string[]; + }; + source?: string; + meta?: { + n_ctx?: number; + n_ctx_train?: number; + size?: number; + ftype?: string; + }; +} + +export interface LlamaModelsResponse { + data: LlamaModelInfo[]; + object?: string; +} + +export interface LlamaModelEvent { + model: string; + event: string; + data?: unknown; +} + +export interface LlamaProgress { + message: string; + ratio?: number; + detail?: string; +} + +function errorMessage(payload: unknown, fallback: string): string { + if (typeof payload !== "object" || payload === null) return fallback; + const error = (payload as { error?: unknown }).error; + if (typeof error !== "object" || error === null) return fallback; + const message = (error as { message?: unknown }).message; + return typeof message === "string" && message ? message : fallback; +} + +function isModelInfo(value: unknown): value is LlamaModelInfo { + if (typeof value !== "object" || value === null) return false; + const candidate = value as { id?: unknown; status?: { value?: unknown } }; + return typeof candidate.id === "string" && typeof candidate.status?.value === "string"; +} + +function linkSignal(source: AbortSignal | undefined, target: AbortController): () => void { + if (!source) return () => {}; + if (source.aborted) { + target.abort(source.reason); + return () => {}; + } + const abort = () => target.abort(source.reason); + source.addEventListener("abort", abort, { once: true }); + return () => source.removeEventListener("abort", abort); +} + +function sleep(ms: number, signal?: AbortSignal): Promise { + return new Promise((resolve, reject) => { + if (signal?.aborted) { + reject(signal.reason ?? new Error("Cancelled")); + return; + } + const abort = () => { + clearTimeout(timeout); + reject(signal?.reason ?? new Error("Cancelled")); + }; + const timeout = setTimeout(() => { + signal?.removeEventListener("abort", abort); + resolve(); + }, ms); + signal?.addEventListener("abort", abort, { once: true }); + }); +} + +function parseLoadProgress(data: unknown): LlamaProgress | undefined { + if (typeof data !== "object" || data === null) return undefined; + const progress = (data as { progress?: unknown }).progress; + if (typeof progress !== "object" || progress === null) return undefined; + const value = progress as { stages?: unknown; current?: unknown; stage?: unknown; value?: unknown }; + const stage = + typeof value.current === "string" ? value.current : typeof value.stage === "string" ? value.stage : undefined; + const stages = Array.isArray(value.stages) + ? value.stages.filter((entry): entry is string => typeof entry === "string") + : []; + const stageRatio = typeof value.value === "number" ? Math.max(0, Math.min(1, value.value)) : undefined; + let ratio = stageRatio; + if (stage && stages.length > 0) { + const index = stages.indexOf(stage); + if (index >= 0) ratio = (index + (stageRatio ?? 0)) / stages.length; + } + return { + message: stage ? `Loading ${stage.replaceAll("_", " ")}` : "Loading model", + ratio, + }; +} + +function parseDownloadProgress(data: unknown): LlamaProgress | undefined { + if (typeof data !== "object" || data === null) return undefined; + let done = 0; + let total = 0; + for (const value of Object.values(data as Record)) { + if (typeof value !== "object" || value === null) continue; + const entry = value as { done?: unknown; total?: unknown }; + if (typeof entry.done !== "number" || typeof entry.total !== "number") continue; + done += entry.done; + total += entry.total; + } + if (total <= 0) return undefined; + return { + message: "Downloading model", + ratio: done / total, + detail: `${formatBytes(done)} / ${formatBytes(total)}`, + }; +} + +export function formatBytes(bytes: number): string { + if (bytes < 1024) return `${bytes} B`; + const units = ["KiB", "MiB", "GiB", "TiB"]; + let value = bytes / 1024; + let unit = units[0]!; + for (let index = 1; index < units.length && value >= 1024; index++) { + value /= 1024; + unit = units[index]!; + } + return `${value >= 10 ? value.toFixed(1) : value.toFixed(2)} ${unit}`; +} + +export function normalizeLlamaServerUrl(value: string): string { + const url = new URL(value.trim()); + if (url.protocol !== "http:" && url.protocol !== "https:") { + throw new Error("Server URL must use http or https"); + } + url.hash = ""; + url.search = ""; + url.pathname = url.pathname.replace(/\/+$/u, "").replace(/\/v1$/u, "") || "/"; + return url.toString().replace(/\/$/u, ""); +} + +export function llamaInferenceUrl(serverUrl: string): string { + return `${normalizeLlamaServerUrl(serverUrl)}/v1`; +} + +export class LlamaClient { + readonly serverUrl: string; + private readonly apiKey: string | undefined; + + constructor(serverUrl: string, apiKey?: string) { + this.serverUrl = normalizeLlamaServerUrl(serverUrl); + this.apiKey = apiKey; + } + + private async request(path: string, init: RequestInit = {}): Promise { + const headers = new Headers(init.headers); + if (init.body !== undefined) headers.set("Content-Type", "application/json"); + if (this.apiKey) headers.set("Authorization", `Bearer ${this.apiKey}`); + const timeout = AbortSignal.timeout(15_000); + const signal = init.signal ? AbortSignal.any([init.signal, timeout]) : timeout; + const response = await fetch(`${this.serverUrl}${path}`, { ...init, headers, signal }); + let payload: unknown; + try { + payload = await response.json(); + } catch { + payload = undefined; + } + if (!response.ok) throw new Error(errorMessage(payload, `llama.cpp returned HTTP ${response.status}`)); + return payload; + } + + async list(options: { reload?: boolean; signal?: AbortSignal } = {}): Promise { + const payload = await this.request(`/models${options.reload ? "?reload=1" : ""}`, { signal: options.signal }); + if (typeof payload !== "object" || payload === null || !Array.isArray((payload as { data?: unknown }).data)) { + throw new Error("llama.cpp returned an invalid model catalog"); + } + const data = (payload as { data: unknown[] }).data; + if (!data.every(isModelInfo)) throw new Error("Server is not running in llama.cpp router mode"); + return data; + } + + async load(model: string, signal?: AbortSignal): Promise { + await this.request("/models/load", { method: "POST", body: JSON.stringify({ model }), signal }); + } + + async unload(model: string, signal?: AbortSignal): Promise { + await this.request("/models/unload", { method: "POST", body: JSON.stringify({ model }), signal }); + } + + async unloadAndWait(model: string, signal?: AbortSignal): Promise { + await this.unload(model, signal); + while (true) { + const entry = (await this.list({ signal })).find((candidate) => candidate.id === model); + if (!entry || entry.status.value === "unloaded") return; + await sleep(100, signal); + } + } + + async download(model: string, signal?: AbortSignal): Promise { + await this.request("/models", { method: "POST", body: JSON.stringify({ model }), signal }); + } + + async watch(onEvent: (event: LlamaModelEvent) => void, signal?: AbortSignal): Promise { + const headers = new Headers(); + if (this.apiKey) headers.set("Authorization", `Bearer ${this.apiKey}`); + const response = await fetch(`${this.serverUrl}/models/sse`, { headers, signal }); + if (!response.ok || !response.body) throw new Error(`llama.cpp SSE returned HTTP ${response.status}`); + const reader = response.body.getReader(); + const decoder = new TextDecoder(); + let buffer = ""; + while (true) { + const chunk = await reader.read(); + if (chunk.done) break; + buffer += decoder.decode(chunk.value, { stream: true }).replaceAll("\r\n", "\n"); + let boundary = buffer.indexOf("\n\n"); + while (boundary >= 0) { + const frame = buffer.slice(0, boundary); + buffer = buffer.slice(boundary + 2); + const data = frame + .split("\n") + .filter((line) => line.startsWith("data:")) + .map((line) => line.slice(5).trimStart()) + .join("\n"); + if (data) { + try { + const event = JSON.parse(data) as LlamaModelEvent; + if (event && typeof event.model === "string" && typeof event.event === "string") onEvent(event); + } catch { + // Ignore malformed events; catalog polling remains authoritative. + } + } + boundary = buffer.indexOf("\n\n"); + } + } + } + + async loadAndWait( + model: string, + onProgress: (progress: LlamaProgress) => void, + signal?: AbortSignal, + ): Promise { + const watcher = new AbortController(); + const unlink = linkSignal(signal, watcher); + let eventLoaded = false; + let eventError: string | undefined; + void this.watch((event) => { + if (event.model !== model) return; + if (event.event !== "model_status" && event.event !== "status_change") return; + const data = event.data as { status?: unknown } | undefined; + if (data?.status === "loaded") eventLoaded = true; + if (data?.status === "unloaded") eventError = "Model failed to load"; + const progress = parseLoadProgress(event.data); + if (progress) onProgress(progress); + }, watcher.signal).catch(() => {}); + try { + await this.load(model, signal); + onProgress({ message: "Loading model" }); + while (true) { + if (signal?.aborted) throw signal.reason ?? new Error("Cancelled"); + const entry = (await this.list({ signal })).find((candidate) => candidate.id === model); + if (entry?.status.value === "loaded") return entry; + if (eventLoaded && !entry) return { id: model, status: { value: "loaded" } }; + if (entry?.status.failed || eventError) { + throw new Error( + entry?.status.exit_code === undefined + ? (eventError ?? "Model failed to load") + : `Model exited with code ${entry.status.exit_code}`, + ); + } + await sleep(250, signal); + } + } finally { + unlink(); + watcher.abort(); + } + } + + async downloadAndWait( + model: string, + onProgress: (progress: LlamaProgress) => void, + signal?: AbortSignal, + ): Promise { + const watcher = new AbortController(); + const unlink = linkSignal(signal, watcher); + let finished = false; + let failure: string | undefined; + let sawDownloading = false; + let polls = 0; + void this.watch((event) => { + if (event.model !== model) return; + if (event.event === "download_finished") finished = true; + if (event.event === "download_failed") failure = errorMessage(event.data, "Download failed"); + if (event.event === "download_progress") { + sawDownloading = true; + const progress = parseDownloadProgress(event.data); + if (progress) onProgress(progress); + } + }, watcher.signal).catch(() => {}); + try { + await this.download(model, signal); + onProgress({ message: "Downloading model" }); + while (true) { + if (signal?.aborted) throw signal.reason ?? new Error("Cancelled"); + if (failure) throw new Error(failure); + const models = await this.list({ signal }); + polls++; + const entry = models.find((candidate) => candidate.id === model); + if (entry?.status.value === "downloading") { + sawDownloading = true; + const progress = parseDownloadProgress(entry.status.progress); + if (progress) onProgress(progress); + } else if (finished || (entry && (sawDownloading || polls >= 2))) { + return this.list({ reload: true, signal }); + } + await sleep(500, signal); + } + } finally { + unlink(); + watcher.abort(); + } + } +} diff --git a/packages/coding-agent/src/extensions/llama/index.ts b/packages/coding-agent/src/extensions/llama/index.ts new file mode 100644 index 00000000..23dd635d --- /dev/null +++ b/packages/coding-agent/src/extensions/llama/index.ts @@ -0,0 +1,188 @@ +import type { ExtensionAPI, ExtensionCommandContext } from "../../core/extensions/types.ts"; +import { LlamaClient, type LlamaModelInfo, normalizeLlamaServerUrl } from "./client.ts"; +import { createLlamaProvider, LLAMA_PROVIDER_ID } from "./provider.ts"; +import { type LlamaUi, runWithProgress, showLlamaUi } from "./ui.ts"; + +function modelIsLoaded(model: LlamaModelInfo): boolean { + return model.status.value === "loaded" || model.status.value === "sleeping"; +} + +function isConnectionError(error: unknown): boolean { + if (!(error instanceof Error)) return false; + const message = `${error.name} ${error.message}`.toLowerCase(); + return message.includes("fetch failed") || message.includes("timeout") || message.includes("network"); +} + +function connectionErrorMessage(error: unknown): string { + if (isConnectionError(error)) return "Could not connect to the server."; + return error instanceof Error ? error.message : String(error); +} + +async function configuredClient(ctx: ExtensionCommandContext): Promise { + const result = await ctx.modelRegistry.getProviderAuth(LLAMA_PROVIDER_ID); + if (!result) { + ctx.ui.notify(`Configure llama.cpp with /login ${LLAMA_PROVIDER_ID}`, "warning"); + return undefined; + } + const configuredUrl = result.env?.LLAMA_BASE_URL; + const serverUrl = normalizeLlamaServerUrl( + typeof configuredUrl === "string" && configuredUrl ? configuredUrl : (result.auth.baseUrl ?? ""), + ); + return new LlamaClient(serverUrl, result.auth.apiKey); +} + +export default function llamaExtension(pi: ExtensionAPI): void { + const provider = createLlamaProvider(); + pi.registerProvider(provider.provider); + + const syncCatalog = async ( + ctx: ExtensionCommandContext, + client: LlamaClient, + catalog?: LlamaModelInfo[], + ): Promise => { + const current = catalog ?? (await client.list()); + provider.setCatalog(current, client.serverUrl); + await ctx.modelRegistry.refresh(); + return current; + }; + + const loadModel = async ( + ctx: ExtensionCommandContext, + ui: LlamaUi, + client: LlamaClient, + catalog: LlamaModelInfo[], + target: LlamaModelInfo, + ): Promise => { + const loaded = catalog.filter((model) => model.id !== target.id && modelIsLoaded(model)); + let replace = false; + if (loaded.length > 0) { + const choice = await ui.select(`${loaded.length} model${loaded.length === 1 ? " is" : "s are"} loaded`, [ + "Unload all and load", + "Keep loaded and load", + "Cancel", + ]); + if (!choice || choice === "Cancel") return; + replace = choice === "Unload all and load"; + } + + const restoreLoaded = async (): Promise => { + ctx.ui.notify("Restoring previously loaded models"); + for (const model of loaded) await client.loadAndWait(model.id, () => {}); + await syncCatalog(ctx, client); + }; + if (replace) { + for (const model of loaded) await client.unloadAndWait(model.id); + } + + try { + const result = await runWithProgress(ui, { + title: "Loading model", + model: target.id, + initialMessage: "Starting…", + cancelTitle: "Stop loading?", + cancelMessage: `Stop loading ${target.id}?`, + run: (signal, update) => client.loadAndWait(target.id, update, signal), + cancel: () => client.unload(target.id), + }); + if (result.cancelled) { + if (replace) await restoreLoaded(); + return; + } + const refreshed = await syncCatalog(ctx, client); + const loadedModel = refreshed.find((model) => model.id === target.id); + ctx.ui.notify( + loadedModel?.status.value === "loaded" ? `Loaded ${target.id}` : `Load started for ${target.id}`, + ); + } catch (error) { + if (replace) { + try { + await restoreLoaded(); + } catch { + // Preserve the original load error. + } + } + throw error; + } + }; + + const unloadModel = async ( + ctx: ExtensionCommandContext, + ui: LlamaUi, + client: LlamaClient, + model: LlamaModelInfo, + ): Promise => { + if (!(await ui.confirm("Unload model?", `Unload ${model.id}?`))) return; + await client.unloadAndWait(model.id); + await syncCatalog(ctx, client); + ctx.ui.notify(`Unloaded ${model.id}`); + }; + + const downloadModel = async (ctx: ExtensionCommandContext, ui: LlamaUi, client: LlamaClient): Promise => { + const model = (await ui.input("Download llama.cpp model", "owner/repository[:quant]"))?.trim(); + if (!model) return; + if (/\s/u.test(model) || !model.includes("/")) { + ctx.ui.notify("Use owner/repository[:quant]", "error"); + return; + } + const result = await runWithProgress(ui, { + title: "Downloading model", + model, + initialMessage: "Starting…", + cancelTitle: "Stop download?", + cancelMessage: `Stop downloading ${model}?`, + run: (signal, update) => client.downloadAndWait(model, update, signal), + cancel: () => client.unload(model), + }); + if (result.cancelled) return; + await syncCatalog(ctx, client, result.value); + ctx.ui.notify(`Downloaded ${model}`); + }; + + pi.registerCommand("llama", { + description: "Manage llama.cpp router models", + handler: async (_args, ctx) => { + if (ctx.mode !== "tui") { + ctx.ui.notify("/llama is available in interactive mode", "warning"); + return; + } + const client = await configuredClient(ctx); + if (!client) return; + await showLlamaUi(ctx, async (ui) => { + const readCatalog = async (): Promise => { + while (true) { + try { + return await syncCatalog(ctx, client); + } catch (error) { + if ((await ui.connectionError(client.serverUrl, connectionErrorMessage(error))) === "close") { + return undefined; + } + } + } + }; + + let catalog = await readCatalog(); + if (!catalog) return; + while (true) { + const action = await ui.showModels(client.serverUrl, catalog); + if (action.type === "close") return; + let actionError: unknown; + try { + if (action.type === "download") await downloadModel(ctx, ui, client); + else if (modelIsLoaded(action.model)) await unloadModel(ctx, ui, client, action.model); + else if (action.model.status.value === "unloaded") + await loadModel(ctx, ui, client, catalog, action.model); + else ctx.ui.notify(`${action.model.id} is ${action.model.status.value}`, "warning"); + } catch (error) { + actionError = error; + } + const refreshed = await readCatalog(); + if (!refreshed) return; + catalog = refreshed; + if (actionError && !isConnectionError(actionError)) { + ctx.ui.notify(actionError instanceof Error ? actionError.message : String(actionError), "error"); + } + } + }); + }, + }); +} diff --git a/packages/coding-agent/src/extensions/llama/provider.ts b/packages/coding-agent/src/extensions/llama/provider.ts new file mode 100644 index 00000000..8ab552df --- /dev/null +++ b/packages/coding-agent/src/extensions/llama/provider.ts @@ -0,0 +1,127 @@ +import type { + ApiKeyCredential, + AuthContext, + AuthResult, + Model, + Provider, + ProviderStreamOptions, + RefreshModelsContext, +} from "@earendil-works/pi-ai"; +import { stream, streamSimple } from "@earendil-works/pi-ai/compat"; +import { LlamaClient, type LlamaModelInfo, llamaInferenceUrl, normalizeLlamaServerUrl } from "./client.ts"; + +export const LLAMA_PROVIDER_ID = "llama.cpp"; +export const DEFAULT_LLAMA_SERVER_URL = "http://127.0.0.1:8080"; +const DEFAULT_MAX_TOKENS = 16384; + +function credentialServerUrl(credential: ApiKeyCredential | undefined): string | undefined { + const value = credential?.env?.LLAMA_BASE_URL; + return typeof value === "string" && value.trim() ? normalizeLlamaServerUrl(value) : undefined; +} + +async function resolveServerUrl( + ctx: AuthContext, + credential: ApiKeyCredential | undefined, +): Promise { + const configured = credentialServerUrl(credential) ?? (await ctx.env("LLAMA_BASE_URL"))?.trim(); + return configured ? normalizeLlamaServerUrl(configured) : undefined; +} + +function toPiModel(model: LlamaModelInfo, serverUrl: string): Model<"openai-completions"> { + const reportedContextWindow = model.meta?.n_ctx ?? model.meta?.n_ctx_train; + const contextWindow = reportedContextWindow && reportedContextWindow > 0 ? reportedContextWindow : 128000; + return { + id: model.id, + name: model.id, + api: "openai-completions", + provider: LLAMA_PROVIDER_ID, + baseUrl: llamaInferenceUrl(serverUrl), + reasoning: false, + input: model.architecture?.input_modalities?.includes("image") ? ["text", "image"] : ["text"], + cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 }, + contextWindow, + maxTokens: Math.min(DEFAULT_MAX_TOKENS, contextWindow), + compat: { + supportsStore: false, + supportsDeveloperRole: false, + supportsReasoningEffort: false, + supportsUsageInStreaming: false, + supportsStrictMode: false, + maxTokensField: "max_tokens", + }, + }; +} + +export interface LlamaProviderController { + provider: Provider<"openai-completions">; + setCatalog(models: readonly LlamaModelInfo[], serverUrl: string): void; +} + +export function createLlamaProvider(): LlamaProviderController { + let models: readonly Model<"openai-completions">[] = []; + + const setCatalog = (catalog: readonly LlamaModelInfo[], serverUrl: string): void => { + models = catalog.filter((model) => model.status.value === "loaded").map((model) => toPiModel(model, serverUrl)); + }; + + const provider: Provider<"openai-completions"> = { + id: LLAMA_PROVIDER_ID, + name: "llama.cpp", + baseUrl: llamaInferenceUrl(DEFAULT_LLAMA_SERVER_URL), + auth: { + apiKey: { + name: "llama.cpp server", + login: async (interaction): Promise => { + const enteredUrl = await interaction.prompt({ + type: "text", + message: "llama.cpp server URL", + placeholder: process.env.LLAMA_BASE_URL ?? DEFAULT_LLAMA_SERVER_URL, + }); + const serverUrl = normalizeLlamaServerUrl( + enteredUrl.trim() || process.env.LLAMA_BASE_URL || DEFAULT_LLAMA_SERVER_URL, + ); + const apiKey = ( + await interaction.prompt({ + type: "secret", + message: "API key (optional)", + }) + ).trim(); + await new LlamaClient(serverUrl, apiKey || undefined).list({ signal: interaction.signal }); + return { + type: "api_key", + key: apiKey || undefined, + env: { LLAMA_BASE_URL: serverUrl }, + }; + }, + check: async ({ ctx, credential }) => { + const serverUrl = await resolveServerUrl(ctx, credential); + return serverUrl + ? { type: "api_key", source: credential ? "stored credential" : "LLAMA_BASE_URL" } + : undefined; + }, + resolve: async ({ ctx, credential }): Promise => { + const serverUrl = await resolveServerUrl(ctx, credential); + if (!serverUrl) return undefined; + const apiKey = credential?.key ?? (await ctx.env("LLAMA_API_KEY")) ?? "local"; + return { + auth: { apiKey, baseUrl: llamaInferenceUrl(serverUrl) }, + env: { ...credential?.env, LLAMA_BASE_URL: serverUrl }, + source: credential ? "stored credential" : "LLAMA_BASE_URL", + }; + }, + }, + }, + getModels: () => models, + refreshModels: async (context: RefreshModelsContext): Promise => { + if (!context.allowNetwork || context.signal?.aborted || context.credential?.type !== "api_key") return; + const serverUrl = credentialServerUrl(context.credential); + if (!serverUrl) return; + const catalog = await new LlamaClient(serverUrl, context.credential.key).list({ signal: context.signal }); + setCatalog(catalog, serverUrl); + }, + stream: (model, context, options) => stream(model, context, options as ProviderStreamOptions | undefined), + streamSimple: (model, context, options) => streamSimple(model, context, options), + }; + + return { provider, setCatalog }; +} diff --git a/packages/coding-agent/src/extensions/llama/ui.ts b/packages/coding-agent/src/extensions/llama/ui.ts new file mode 100644 index 00000000..2c46381c --- /dev/null +++ b/packages/coding-agent/src/extensions/llama/ui.ts @@ -0,0 +1,343 @@ +import { + Container, + type Focusable, + Input, + type SelectItem, + SelectList, + Spacer, + Text, + type TUI, + truncateToWidth, + visibleWidth, +} from "@earendil-works/pi-tui"; +import type { ExtensionCommandContext } from "../../core/extensions/types.ts"; +import type { KeybindingsManager } from "../../core/keybindings.ts"; +import { DynamicBorder } from "../../modes/interactive/components/dynamic-border.ts"; +import { keyHint } from "../../modes/interactive/components/keybinding-hints.ts"; +import type { Theme } from "../../modes/interactive/theme/theme.ts"; +import type { LlamaModelInfo, LlamaProgress } from "./client.ts"; + +const DOWNLOAD_VALUE = "\0download"; + +export type LlamaManagerAction = { type: "model"; model: LlamaModelInfo } | { type: "download" } | { type: "close" }; + +interface ProgressState extends LlamaProgress { + title: string; + model: string; +} + +function contextLabel(model: LlamaModelInfo): string | undefined { + const context = model.meta?.n_ctx ?? model.meta?.n_ctx_train; + if (context) return context >= 1000 ? `${Math.round(context / 1000)}k` : String(context); + const args = model.status.args ?? []; + for (let index = 0; index < args.length - 1; index++) { + if (args[index] !== "--ctx-size" && args[index] !== "-c" && args[index] !== "-ctx") continue; + const value = Number(args[index + 1]); + if (Number.isFinite(value) && value > 0) return value >= 1000 ? `${Math.round(value / 1000)}k` : String(value); + } + return undefined; +} + +function modelDescription(model: LlamaModelInfo): string { + const details: string[] = []; + const loaded = model.status.value === "loaded" || model.status.value === "sleeping"; + if (loaded) details.push("loaded"); + else if (model.status.value !== "unloaded") details.push(model.status.value); + const context = loaded ? contextLabel(model) : undefined; + if (context) details.push(`${context} context`); + return details.join(" · "); +} + +function selectTheme(theme: Theme) { + return { + selectedPrefix: (text: string) => theme.fg("accent", text), + selectedText: (text: string) => theme.fg("accent", text), + description: (text: string) => theme.fg("muted", text), + scrollInfo: (text: string) => theme.fg("dim", text), + noMatch: (text: string) => theme.fg("warning", text), + }; +} + +function frame( + theme: Theme, + title: string, + body: Array, + footer?: string, +): Container { + const container = new Container(); + container.addChild(new DynamicBorder((text: string) => theme.fg("accent", text))); + container.addChild(new Text(theme.fg("accent", theme.bold(title)), 1, 0)); + for (const child of body) container.addChild(child); + if (footer) { + container.addChild(new Spacer(1)); + container.addChild(new Text(theme.fg("dim", footer), 1, 0)); + } + container.addChild(new DynamicBorder((text: string) => theme.fg("accent", text))); + return container; +} + +export interface LlamaUi { + showModels(serverUrl: string, models: LlamaModelInfo[]): Promise; + select(title: string, options: string[]): Promise; + confirm(title: string, message: string): Promise; + connectionError(serverUrl: string, message: string): Promise<"retry" | "close">; + input(title: string, placeholder: string): Promise; + progress(state: ProgressState): Promise; + updateProgress(state: ProgressState): void; +} + +class LlamaView implements LlamaUi, Focusable { + private readonly tui: TUI; + private readonly theme: Theme; + private readonly keybindings: KeybindingsManager; + private content: Container; + private inputHandler: { handleInput?(data: string): void } | undefined; + private inputTarget: Focusable | undefined; + private progressPromise: Promise | undefined; + private progressResolver: (() => void) | undefined; + private showingProgress = false; + private _focused = false; + + constructor(tui: TUI, theme: Theme, keybindings: KeybindingsManager) { + this.tui = tui; + this.theme = theme; + this.keybindings = keybindings; + this.content = frame(theme, "llama.cpp models", [new Text(theme.fg("muted", "Loading…"), 1, 1)]); + } + + get focused(): boolean { + return this._focused; + } + + set focused(value: boolean) { + this._focused = value; + if (this.inputTarget) this.inputTarget.focused = value; + } + + private setContent( + content: Container, + inputHandler?: { handleInput?(data: string): void }, + inputTarget?: Focusable, + ): void { + if (this.inputTarget) this.inputTarget.focused = false; + this.progressPromise = undefined; + this.progressResolver = undefined; + this.showingProgress = false; + this.content = content; + this.inputHandler = inputHandler; + this.inputTarget = inputTarget; + if (this.inputTarget) this.inputTarget.focused = this._focused; + this.tui.requestRender(); + } + + showModels(serverUrl: string, models: LlamaModelInfo[]): Promise { + const sorted = [...models].sort((left, right) => { + const loaded = Number(right.status.value === "loaded") - Number(left.status.value === "loaded"); + return loaded || left.id.localeCompare(right.id); + }); + const byId = new Map(sorted.map((model) => [model.id, model])); + const items: SelectItem[] = [ + ...sorted.map((model) => ({ + value: model.id, + label: model.id, + description: modelDescription(model), + })), + { value: DOWNLOAD_VALUE, label: "Download model…", description: "Hugging Face owner/repository[:quant]" }, + ]; + return new Promise((resolve) => { + const list = new SelectList(items, Math.min(items.length, 12), selectTheme(this.theme), { + minPrimaryColumnWidth: 36, + maxPrimaryColumnWidth: 56, + }); + list.onSelect = (item) => { + if (item.value === DOWNLOAD_VALUE) resolve({ type: "download" }); + else { + const model = byId.get(item.value); + if (model) resolve({ type: "model", model }); + } + }; + list.onCancel = () => resolve({ type: "close" }); + this.setContent( + frame( + this.theme, + "llama.cpp models", + [new Text(this.theme.fg("dim", serverUrl), 1, 0), new Spacer(1), list], + `${keyHint("tui.select.confirm", "load/unload/download")} • ${keyHint("tui.select.cancel", "close")}`, + ), + list, + ); + }); + } + + select(title: string, options: string[]): Promise { + return new Promise((resolve) => { + const list = new SelectList( + options.map((option) => ({ value: option, label: option })), + options.length, + selectTheme(this.theme), + ); + list.onSelect = (item) => resolve(item.value); + list.onCancel = () => resolve(undefined); + this.setContent( + frame( + this.theme, + title, + [new Spacer(1), list], + `${keyHint("tui.select.confirm", "select")} • ${keyHint("tui.select.cancel", "cancel")}`, + ), + list, + ); + }); + } + + async confirm(title: string, message: string): Promise { + return (await this.select(`${title}\n${message}`, ["Yes", "No"])) === "Yes"; + } + + async connectionError(serverUrl: string, message: string): Promise<"retry" | "close"> { + const choice = await this.select(`llama.cpp unavailable\n${serverUrl}\n\n${message}`, ["Retry", "Close"]); + return choice === "Retry" ? "retry" : "close"; + } + + input(title: string, placeholder: string): Promise { + return new Promise((resolve) => { + const input = new Input(); + input.onSubmit = (value) => resolve(value); + input.onEscape = () => resolve(undefined); + this.setContent( + frame( + this.theme, + title, + [new Spacer(1), new Text(this.theme.fg("dim", placeholder), 1, 0), input], + `${keyHint("tui.input.submit", "submit")} • ${keyHint("tui.select.cancel", "cancel")}`, + ), + input, + input, + ); + }); + } + + progress(state: ProgressState): Promise { + if (!this.progressPromise) { + this.progressPromise = new Promise((resolve) => { + this.progressResolver = resolve; + }); + } + this.showingProgress = true; + this.updateProgress(state); + return this.progressPromise; + } + + updateProgress(state: ProgressState): void { + if (!this.showingProgress) return; + const body = [ + new Text(this.theme.fg("text", state.model), 1, 0), + new Spacer(1), + new Text(this.theme.fg("muted", state.message), 1, 0), + ]; + if (state.ratio !== undefined) { + const available = 40; + const filled = Math.round(Math.max(0, Math.min(1, state.ratio)) * available); + body.push( + new Text( + this.theme.fg( + "accent", + `${"█".repeat(filled)}${"─".repeat(available - filled)} ${Math.round(state.ratio * 100)}%`, + ), + 1, + 0, + ), + ); + } + if (state.detail) body.push(new Text(this.theme.fg("dim", state.detail), 1, 0)); + this.content = frame(this.theme, state.title, body, keyHint("tui.select.cancel", "stop")); + this.inputHandler = undefined; + this.tui.requestRender(); + } + + handleInput(data: string): void { + if (this.progressResolver && this.keybindings.matches(data, "tui.select.cancel")) { + const resolve = this.progressResolver; + this.progressPromise = undefined; + this.progressResolver = undefined; + resolve(); + return; + } + this.inputHandler?.handleInput?.(data); + this.tui.requestRender(); + } + + render(width: number): string[] { + return this.content + .render(width) + .map((line) => (visibleWidth(line) > width ? truncateToWidth(line, width, "") : line)); + } + + invalidate(): void { + this.content.invalidate(); + } +} + +export async function showLlamaUi(ctx: ExtensionCommandContext, run: (ui: LlamaUi) => Promise): Promise { + await ctx.ui.custom((tui, theme, keybindings, done) => { + const view = new LlamaView(tui, theme, keybindings); + void run(view).then( + () => done(), + (error: unknown) => { + ctx.ui.notify(error instanceof Error ? error.message : String(error), "error"); + done(); + }, + ); + return view; + }); +} + +export async function runWithProgress( + ui: LlamaUi, + options: { + title: string; + model: string; + initialMessage: string; + cancelTitle: string; + cancelMessage: string; + run(signal: AbortSignal, update: (progress: LlamaProgress) => void): Promise; + cancel(): Promise; + }, +): Promise<{ cancelled: true } | { cancelled: false; value: T }> { + const controller = new AbortController(); + const state: ProgressState = { title: options.title, model: options.model, message: options.initialMessage }; + const settled = options + .run(controller.signal, (progress) => { + Object.assign(state, progress); + ui.updateProgress(state); + }) + .then( + (value) => ({ ok: true as const, value }), + (error: unknown) => ({ ok: false as const, error }), + ); + let completed = false; + settled.finally(() => { + completed = true; + }); + + while (!completed) { + const outcome = await Promise.race([ + settled.then(() => "settled" as const), + ui.progress(state).then(() => "stop" as const), + ]); + if (outcome === "settled") break; + const stop = await ui.confirm(options.cancelTitle, options.cancelMessage); + if (!stop || completed) continue; + try { + await options.cancel(); + } finally { + controller.abort(new Error("Cancelled")); + } + await settled; + return { cancelled: true }; + } + + const result = await settled; + if (!result.ok) throw result.error; + return { cancelled: false, value: result.value }; +} diff --git a/packages/coding-agent/src/main.ts b/packages/coding-agent/src/main.ts index 5eaa3e5c..7d9c5525 100644 --- a/packages/coding-agent/src/main.ts +++ b/packages/coding-agent/src/main.ts @@ -41,6 +41,7 @@ import { assertValidSessionId, SessionManager } from "./core/session-manager.ts" import { SettingsManager } from "./core/settings-manager.ts"; import { printTimings, resetTimings, time } from "./core/timings.ts"; import { hasTrustRequiringProjectResources, ProjectTrustStore } from "./core/trust-manager.ts"; +import { builtInExtensions } from "./extensions/index.ts"; import { runMigrations, showDeprecationWarnings } from "./migrations.ts"; import { InteractiveMode, runPrintMode, runRpcMode } from "./modes/index.ts"; import { initTheme, stopThemeWatcher } from "./modes/interactive/theme/theme.ts"; @@ -471,6 +472,7 @@ export interface MainOptions { export async function main(args: string[], options?: MainOptions) { resetTimings(); + const extensionFactories = [...builtInExtensions, ...(options?.extensionFactories ?? [])]; const offlineMode = args.includes("--offline") || isTruthyEnvFlag(process.env.PI_OFFLINE); if (offlineMode) { process.env.PI_OFFLINE = "1"; @@ -487,7 +489,7 @@ export async function main(args: string[], options?: MainOptions) { applyHttpProxySettings(bootstrapSettingsManager.getGlobalSettings().httpProxy); configureHttpDispatcher(); - if (await handlePackageCommand(args, { extensionFactories: options?.extensionFactories })) { + if (await handlePackageCommand(args, { extensionFactories })) { const exitCode = process.exitCode ?? 0; if (process.platform === "win32" && exitCode === 0 && args[0] === "update") { // We normally prefer process.exit(0) for package commands so bad extensions cannot keep @@ -500,7 +502,7 @@ export async function main(args: string[], options?: MainOptions) { return; } - if (await handleConfigCommand(args, { extensionFactories: options?.extensionFactories })) { + if (await handleConfigCommand(args, { extensionFactories })) { return; } @@ -670,7 +672,7 @@ export async function main(args: string[], options?: MainOptions) { noContextFiles: parsed.noContextFiles, systemPrompt: parsed.systemPrompt, appendSystemPrompt: parsed.appendSystemPrompt, - extensionFactories: options?.extensionFactories, + extensionFactories, }, }); const { settingsManager, modelRuntime, resourceLoader } = services; diff --git a/packages/coding-agent/src/modes/interactive/interactive-mode.ts b/packages/coding-agent/src/modes/interactive/interactive-mode.ts index 483d0b14..b7dc08fc 100644 --- a/packages/coding-agent/src/modes/interactive/interactive-mode.ts +++ b/packages/coding-agent/src/modes/interactive/interactive-mode.ts @@ -1440,10 +1440,13 @@ export class InteractiveMode { const themesResult = this.session.resourceLoader.getThemes(); const extensions = options?.extensions ?? - this.session.resourceLoader.getExtensions().extensions.map((extension) => ({ - path: extension.path, - sourceInfo: extension.sourceInfo, - })); + this.session.resourceLoader + .getExtensions() + .extensions.filter((extension) => !extension.hidden) + .map((extension) => ({ + path: extension.path, + sourceInfo: extension.sourceInfo, + })); const sourceInfos = new Map(); for (const extension of extensions) { if (extension.sourceInfo) { diff --git a/packages/coding-agent/test/llama-extension.test.ts b/packages/coding-agent/test/llama-extension.test.ts new file mode 100644 index 00000000..3be915c6 --- /dev/null +++ b/packages/coding-agent/test/llama-extension.test.ts @@ -0,0 +1,207 @@ +import { once } from "node:events"; +import { createServer, type RequestListener, type Server, type ServerResponse } from "node:http"; +import type { AddressInfo } from "node:net"; +import type { AuthContext, AuthPrompt } from "@earendil-works/pi-ai"; +import { afterEach, describe, expect, it } from "vitest"; +import { createEventBus } from "../src/core/event-bus.ts"; +import { createExtensionRuntime, loadExtensionFromFactory } from "../src/core/extensions/loader.ts"; +import { LlamaClient, type LlamaProgress, normalizeLlamaServerUrl } from "../src/extensions/llama/client.ts"; +import llamaExtension from "../src/extensions/llama/index.ts"; +import { createLlamaProvider, LLAMA_PROVIDER_ID } from "../src/extensions/llama/provider.ts"; + +const servers: Server[] = []; + +async function listen(handler: RequestListener): Promise<{ server: Server; url: string }> { + const server = createServer(handler); + servers.push(server); + server.listen(0, "127.0.0.1"); + await once(server, "listening"); + const address = server.address() as AddressInfo; + return { server, url: `http://127.0.0.1:${address.port}` }; +} + +function json(response: ServerResponse, value: unknown): void { + response.writeHead(200, { "Content-Type": "application/json" }); + response.end(JSON.stringify(value)); +} + +afterEach(async () => { + await Promise.all( + servers.splice(0).map( + (server) => + new Promise((resolve) => { + server.close(() => resolve()); + server.closeAllConnections(); + }), + ), + ); +}); + +describe("llama.cpp extension", () => { + it("registers a native provider and /llama command", async () => { + const runtime = createExtensionRuntime(); + const extension = await loadExtensionFromFactory( + llamaExtension, + process.cwd(), + createEventBus(), + runtime, + "", + ); + + expect(extension.commands.get("llama")?.description).toBe("Manage llama.cpp router models"); + expect(runtime.pendingNativeProviderRegistrations.map((entry) => entry.provider.id)).toEqual([LLAMA_PROVIDER_ID]); + }); + + it("normalizes management and inference URLs", () => { + expect(normalizeLlamaServerUrl("http://127.0.0.1:8080/v1/")).toBe("http://127.0.0.1:8080"); + expect(normalizeLlamaServerUrl("https://example.com/prefix/v1")).toBe("https://example.com/prefix"); + expect(() => normalizeLlamaServerUrl("file:///tmp/llama")).toThrow("http or https"); + }); + + it("exposes only loaded models with router metadata", () => { + const controller = createLlamaProvider(); + controller.setCatalog( + [ + { + id: "loaded", + status: { value: "loaded", args: ["llama-server", "--n-gpu-layers", "999"] }, + architecture: { input_modalities: ["text", "image"] }, + meta: { n_ctx: 16384, n_ctx_train: 131072 }, + }, + { id: "unloaded", status: { value: "unloaded" } }, + { id: "loading", status: { value: "loading" } }, + ], + "http://localhost:8080", + ); + + expect(controller.provider.getModels()).toEqual([ + expect.objectContaining({ + id: "loaded", + baseUrl: "http://localhost:8080/v1", + contextWindow: 16384, + maxTokens: 16384, + input: ["text", "image"], + }), + ]); + }); + + it("stays dormant until configured and stores URL plus optional key", async () => { + const { provider } = createLlamaProvider(); + const auth = provider.auth.apiKey!; + const emptyContext: AuthContext = { + env: async () => undefined, + fileExists: async () => false, + }; + expect(await auth.check?.({ ctx: emptyContext })).toBeUndefined(); + expect(await auth.resolve({ ctx: emptyContext })).toBeUndefined(); + + const { url } = await listen((request, response) => { + expect(request.headers.authorization).toBe("Bearer secret"); + json(response, { data: [] }); + }); + const answers = [url, "secret"]; + const credential = await auth.login!({ + prompt: async (_prompt: AuthPrompt) => answers.shift()!, + notify: () => {}, + }); + expect(credential).toEqual({ + type: "api_key", + key: "secret", + env: { LLAMA_BASE_URL: url }, + }); + expect(await auth.resolve({ ctx: emptyContext, credential })).toEqual({ + auth: { apiKey: "secret", baseUrl: `${url}/v1` }, + env: { LLAMA_BASE_URL: url }, + source: "stored credential", + }); + }); + + it("loads with SSE progress and waits for the loaded catalog state", async () => { + let status: "unloaded" | "loading" | "loaded" = "unloaded"; + const streams = new Set(); + const send = (event: unknown) => { + for (const response of streams) response.write(`data: ${JSON.stringify(event)}\n\n`); + }; + const { url } = await listen((request, response) => { + if (request.url === "/models/sse") { + response.writeHead(200, { "Content-Type": "text/event-stream" }); + streams.add(response); + request.on("close", () => streams.delete(response)); + return; + } + if (request.url === "/models/load" && request.method === "POST") { + status = "loading"; + json(response, { success: true }); + setTimeout(() => { + send({ + model: "test-model", + event: "status_change", + data: { + status: "loading", + progress: { stages: ["text_model", "mmproj_model"], current: "text_model", value: 0.5 }, + }, + }); + status = "loaded"; + send({ model: "test-model", event: "status_change", data: { status: "loaded" } }); + }, 20); + return; + } + if (request.url === "/models") { + json(response, { data: [{ id: "test-model", status: { value: status } }] }); + return; + } + response.writeHead(404).end(); + }); + + const progress: string[] = []; + const model = await new LlamaClient(url).loadAndWait("test-model", (entry) => progress.push(entry.message)); + expect(model.status.value).toBe("loaded"); + expect(progress).toContain("Loading text model"); + }); + + it("downloads with byte progress and returns the refreshed catalog", async () => { + let status: "missing" | "downloading" | "unloaded" = "missing"; + const streams = new Set(); + const send = (event: unknown) => { + for (const response of streams) response.write(`data: ${JSON.stringify(event)}\n\n`); + }; + const { url } = await listen((request, response) => { + if (request.url === "/models/sse") { + response.writeHead(200, { "Content-Type": "text/event-stream" }); + streams.add(response); + request.on("close", () => streams.delete(response)); + return; + } + if (request.url === "/models" && request.method === "POST") { + status = "downloading"; + json(response, { success: true }); + setTimeout(() => { + send({ + model: "owner/repo:Q4_K_M", + event: "download_progress", + data: { "https://example/model.gguf": { done: 512, total: 1024 } }, + }); + status = "unloaded"; + send({ model: "owner/repo:Q4_K_M", event: "download_finished", data: {} }); + }, 20); + return; + } + if (request.url?.startsWith("/models")) { + json(response, { + data: status === "missing" ? [] : [{ id: "owner/repo:Q4_K_M", status: { value: status } }], + }); + return; + } + response.writeHead(404).end(); + }); + + const progress: LlamaProgress[] = []; + const models = await new LlamaClient(url).downloadAndWait("owner/repo:Q4_K_M", (entry) => progress.push(entry)); + expect(models).toEqual([{ id: "owner/repo:Q4_K_M", status: { value: "unloaded" } }]); + expect(progress).toContainEqual({ + message: "Downloading model", + ratio: 0.5, + detail: "512 B / 1.00 KiB", + }); + }); +}); diff --git a/packages/coding-agent/test/suite/regressions/6260-inline-extension-naming.test.ts b/packages/coding-agent/test/suite/regressions/6260-inline-extension-naming.test.ts index 522e3fa2..186d422e 100644 --- a/packages/coding-agent/test/suite/regressions/6260-inline-extension-naming.test.ts +++ b/packages/coding-agent/test/suite/regressions/6260-inline-extension-naming.test.ts @@ -76,6 +76,26 @@ describe("inline extension naming", () => { expect(result.extensions[1].path).toBe(""); }); + it("preserves hidden state for named factories", async () => { + const { cwd, agentDir } = fixture("hidden"); + const loader = new DefaultResourceLoader({ + cwd, + agentDir, + noSkills: true, + noPromptTemplates: true, + noThemes: true, + extensionFactories: [{ name: "built-in", factory: noop, hidden: true }], + }); + + await loader.reload(); + + const result = loader.getExtensions(); + + expect(result.extensions).toHaveLength(1); + expect(result.extensions[0].path).toBe(""); + expect(result.extensions[0].hidden).toBe(true); + }); + it("supports mixed bare and named factories", async () => { const { cwd, agentDir } = fixture("mixed"); const loader = new DefaultResourceLoader({