Providers and transports: OpenAI-compatible endpoints, the AI SDK bridge, a model in the browser, the scripted scenario provider, local models, writing your own provider.
# Put a chat on a model on your machine
> Put a chat on Ollama, LM Studio or llama.cpp through their OpenAI-compatible endpoint — the browser calls the server directly, with no key and no backend.
## Start the server and wire the chat [Section titled “Start the server and wire the chat”](#start-the-server-and-wire-the-chat)
```bash
npm install @aparte/core @aparte/provider-openai-compat
```
* Ollama
```bash
OLLAMA_ORIGINS=* ollama serve # PowerShell: $env:OLLAMA_ORIGINS = "*"; ollama serve
ollama pull llama3.2 # in a second terminal
```
```ts
import { aparteGlobalConfig, AparteClient } from '@aparte/core';
import { createOpenAICompatProvider, presets } from '@aparte/provider-openai-compat';
import '@aparte/core/styles.css';
aparteGlobalConfig.registerAIProvider(createOpenAICompatProvider(presets.OLLAMA)); // http://localhost:11434/v1
aparteGlobalConfig.setModelConfig({ defaultProvider: 'ollama', defaultModel: 'llama3.2:latest' });
new AparteClient().start();
```
* LM Studio Load a model, start the server in the Developer tab, and turn on **Enable CORS** there.
```ts
import { aparteGlobalConfig, AparteClient } from '@aparte/core';
import { createOpenAICompatProvider, presets } from '@aparte/provider-openai-compat';
import '@aparte/core/styles.css';
aparteGlobalConfig.registerAIProvider(createOpenAICompatProvider(presets.LMSTUDIO)); // http://localhost:1234/v1
aparteGlobalConfig.setModelConfig({ defaultProvider: 'lmstudio', defaultModel: 'qwen2.5-7b-instruct' }); // the id LM Studio shows
new AparteClient().start();
```
* llama.cpp
```bash
llama-server -m ./models/qwen2.5-7b-instruct-q4_k_m.gguf
```
```ts
import { aparteGlobalConfig, AparteClient } from '@aparte/core';
import { createOpenAICompatProvider } from '@aparte/provider-openai-compat';
import '@aparte/core/styles.css';
aparteGlobalConfig.registerAIProvider(createOpenAICompatProvider({
id: 'llamacpp',
name: 'llama.cpp',
baseURL: 'http://localhost:8080/v1', // llama-server's default port
isLocal: true,
}));
aparteGlobalConfig.setModelConfig({ defaultProvider: 'llamacpp', defaultModel: 'default' }); // it answers with the model it loaded
new AparteClient().start();
```
```html
```
The three servers speak OpenAI’s chat completions at `/v1`, so [`@aparte/provider-openai-compat`](/providers/ai/openai-compat/) reaches them all, and the browser calls the server itself — the default transport — with no key to hold and no backend to write. `presets.OLLAMA` and `presets.LMSTUDIO` are marked local, so no key is asked for; llama.cpp has no preset, and an `id`, a `baseURL` and `isLocal: true` make one, `name` labelling it. A reply the server logs but the page never shows is the browser refusing it: the server has to allow the page’s origin, which is what `OLLAMA_ORIGINS` and **Enable CORS** do ([CORS on a local server](/guides/troubleshooting/#cors-on-local-byok-lm-studio--ollama)). ## Let the person pick the model [Section titled “Let the person pick the model”](#let-the-person-pick-the-model)
```ts
import { aparteGlobalConfig } from '@aparte/core';
import '@aparte/plugin-model-selector';
aparteGlobalConfig.setModelPreferenceProvider({
save: (provider, model) => localStorage.setItem('model', JSON.stringify({ provider, model })),
load: () => JSON.parse(localStorage.getItem('model') ?? 'null'),
});
aparteGlobalConfig.restoreModelPreference(); // before the chat mounts
```
```html
```
A local server lists the models it holds at `/v1/models`, and a local provider fetches that list with no key; `llama-server` lists the one it was started with. The [model selector](/components/model-selector/) shows the list, `auto-select` picks the first model when none is selected, and `persist` makes the pick the model the next message goes to — without it, the selector shows a choice nothing sends. The preference provider above keeps the pick across reloads. A toolbar needs the chat written out, as here: an `` with no `` inside replaces its children with the default chat. ## Show how full the context is [Section titled “Show how full the context is”](#show-how-full-the-context-is) The `` in the toolbar above is the [context gauge](/components/context-gauge/). The model list gives it a window only when the server returns a `context_length` for each model, which a local server usually does not, so `window` gives it yourself, in tokens: the context size the server runs the model with. It fills in after the first reply, from the usage the server reports. ## Tools [Section titled “Tools”](#tools) The tools you register go with every request: every model the server lists is marked able to call them, and so is a model you list without `capabilities`; whether it uses them is the model’s call. [Tools and approval](/guides/tools/) work the same, the approval panel included, with a model that supports function calling. ## Next [Section titled “Next”](#next) * [Run a model in the browser No server at all: the model runs inside the page.](/providers/ai/transformers/) * [Keep the key on your server A model behind your own /api/chat, local or not.](/guides/backend-transport/) * [Own the history A prefix cache — llama.cpp slots, vLLM — that wants each turn to extend the last.](/guides/engine/#own-the-history)
# Connect the chat to a model or an agent
> Reach your model — OpenAI or any compatible API, Ollama, the Vercel AI SDK, a model in the browser — or your agent, and decide where the key lives.
## Pick a package [Section titled “Pick a package”](#pick-a-package) | You want to reach | Package | | ---------------------------------------------------------------------------------------------- | ---------------------------------------------------------------- | | OpenAI, Mistral, OpenRouter, Z.ai, LM Studio, Ollama — or any other OpenAI-compatible endpoint | [`@aparte/provider-openai-compat`](/providers/ai/openai-compat/) | | Anthropic, Google, Amazon Bedrock, or another `@ai-sdk/*` vendor | [`@aparte/provider-ai-sdk`](/providers/ai/ai-sdk/) | | A model running in the browser, with no server and no key | [`@aparte/provider-transformers`](/providers/ai/transformers/) | | Scripted replies, for a demo or a test | [`@aparte/provider-scenario`](/providers/ai/scenario/) | | An agent on its own server, over AG-UI — Agno, LangGraph, Mastra, CrewAI, Pydantic AI | [`@aparte/runner-ag-ui`](/providers/agent/ag-ui/) | | A TanStack AI chat route | [`@aparte/runner-ag-ui`](/providers/agent/tanstack-ai/) | A provider is the wire format of a family of models: how a request is shaped and how its answer is read. The transport is where the request goes — from the browser to the model, or to [your server](/guides/backend-transport/), which keeps the key. ## Wire it [Section titled “Wire it”](#wire-it)
```ts
import { aparteGlobalConfig, AparteClient, AparteDirectTransport } from '@aparte/core';
import { createOpenAICompatProvider, presets } from '@aparte/provider-openai-compat';
aparteGlobalConfig.registerAIProvider(createOpenAICompatProvider(presets.OPENROUTER));
aparteGlobalConfig.setModelConfig({ defaultProvider: 'openrouter', defaultModel: 'openai/gpt-4o-mini' });
aparteGlobalConfig.setTransport(new AparteDirectTransport({ byok: true }));
new AparteClient({
keyResolver: (providerId) => localStorage.getItem(`${providerId}.key`) ?? undefined,
}).start();
```
Register a provider, name the model — or let the person pick it in the [model selector](/components/model-selector/) — and start a client: the `` on the page streams from there ([Installation and first chat](/guides/getting-started/)). A provider that lists its models over the network selects none by itself. The browser transport is the default, so `setTransport` only says `byok: true` — the key is the person’s own — which keeps the console from warning, once, that a key went out from the page. ## The key [Section titled “The key”](#the-key) `keyResolver` is asked for a provider’s key on each request, and may be async — a vault, an input of yours:
```ts
declare const keyInput: HTMLInputElement; // a field in your settings
keyInput.addEventListener('change', () => localStorage.setItem('openrouter.key', keyInput.value.trim()));
```
It may also answer `{ apiKey, endpoint }`, which points the provider, and its list of models, at a host of yours. `aparteGlobalConfig.setKeyProvider(fn)` does the same for every client, and `registerKeyProvider(fn)` adds a source and returns the function that removes it; a client’s own resolver is asked first. A local server — the Ollama and LM Studio presets — needs no key. A cloud provider without one lists no models, and the vendor refuses its requests. To keep the key off the browser altogether, send the requests through [your server](/guides/backend-transport/). ## Agents are not providers [Section titled “Agents are not providers”](#agents-are-not-providers) An agent that runs its own loop and its own tools does not plug in as a provider: it plugs in one level up, as the client’s `streamRunner`. The page registers no provider and picks no model; it draws what the agent streams. [AG-UI](/providers/agent/ag-ui/) and [TanStack AI](/providers/agent/tanstack-ai/) work this way, and so does [aparté’s own loop on your server](/guides/server-loop/). ## Write your own provider [Section titled “Write your own provider”](#write-your-own-provider) A provider is an `AparteAIProvider`, in one of two shapes: * **a format adapter** — `defaultEndpoint`, `buildRequest(request)`, `parseStream(body)`, one of `authHeaders(key)` or `authQuery(key)`, and `parseText` for an answer that does not stream. The transport sends the request and holds the key. This is [`openai-compat`](/providers/ai/openai-compat/). * **its own I/O** — `chat(request, config, { providerId, signal })`, which makes the call itself, through an SDK or a runtime in the page. This is [`ai-sdk`](/providers/ai/ai-sdk/) and [`transformers`](/providers/ai/transformers/). A transport of your own tells the two apart with `isFormatAdapter(provider)`. Both answer `id`, `getMetadata()` — an `AparteAIProviderMetadata`, whose `name` the model selector shows as a group label, with an `icon` and a `color` for a picker of your own — and `getModels()`. `fetchModels` fetches the list at run time, and is what the [model selector](/components/model-selector/) shows; `getModelStatus`, `prepareModel` and `deleteModel` are for a model that has to be downloaded first. A model’s `capabilities` say what it can do; one that declares them without `function_calling` gets no tools. A tool turn reaches the provider as an `assistant` message carrying the model’s `toolCalls`, then one `tool` message per result, which the provider turns into its vendor’s shape:
```ts
import type { AparteChatMessage } from '@aparte/core';
const history: AparteChatMessage[] = [
{ role: 'user', content: 'Any news on the beta?' },
{ role: 'assistant', content: 'Let me look.', toolCalls: [{ id: 'call_1', name: 'search', input: { query: 'beta' } }] },
{ role: 'tool', content: '3 results', toolCallId: 'call_1', toolName: 'search' },
];
```
## Next [Section titled “Next”](#next) * [OpenAI-compatible OpenAI, Mistral, OpenRouter, Ollama and every compatible endpoint.](/providers/ai/openai-compat/) * [Local models A model on your machine, from Ollama, LM Studio or llama.cpp.](/guides/local-models/) * [Keep the key on your server The same chat, with each request going through your server.](/guides/backend-transport/)
# Put a chat on an AG-UI agent
> Connect an agent served over AG-UI — Agno, LangGraph, Mastra, CrewAI, Pydantic AI — and the chat draws its replies, its reasoning, its tool calls and its approvals while the agent runs on its own server.
## Connect the agent [Section titled “Connect the agent”](#connect-the-agent)
```bash
npm install @aparte/runner-ag-ui @aparte/core
```
```ts
import { AparteClient } from '@aparte/core';
import { createAgUiRunner } from '@aparte/runner-ag-ui';
new AparteClient({
streamRunner: createAgUiRunner({ endpoint: 'https://agents.example.com/agui' }),
}).start();
```
[AG-UI](https://docs.ag-ui.com) is the protocol many agent frameworks serve their agents over: Agno, LangGraph, Mastra, CrewAI, Pydantic AI, and [TanStack AI](/providers/agent/tanstack-ai/). The agent runs its loop and its tools on its server, so the page registers no provider and picks no model. The runner is the client’s `streamRunner`: the chat, Stop, the conversation manager and the approval panel work as they do with aparté’s own loop. ## What the agent’s stream becomes [Section titled “What the agent’s stream becomes”](#what-the-agents-stream-becomes) | The agent sends | The chat shows | | ------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------- | | `TEXT_MESSAGE_*` | the reply’s text | | `REASONING_*` | a [reasoning block](/components/reasoning/) | | `TOOL_CALL_START`, `TOOL_CALL_ARGS` or `TOOL_CALL_CHUNK`, `TOOL_CALL_END` | a [tool row](/components/tool/), drawn with its whole arguments once the call ends | | `TOOL_CALL_RESULT` | the row’s result | | `RUN_FINISHED` | the end of the reply, with its usage — or, with an interrupt, a decision to make or a tool the page runs | | `RUN_ERROR` | the reply’s error, with the agent’s message | | an HTTP error | the reply’s error, with the start of the response body — keep secrets out of it | Each model step of the run is a turn of its own in the reply, and the usage of every step is added up and reported with the reply. A stream that ends without `RUN_FINISHED` ends the reply with an error. `STEP_*`, `STATE_*`, `MESSAGES_SNAPSHOT`, `ACTIVITY_*`, `RAW`, `CUSTOM` and sub-agent events are not drawn. The runner reads the answer as server-sent events, or as NDJSON when its content type says so. ## Tools the page runs [Section titled “Tools the page runs”](#tools-the-page-runs) A tool registered in the page — `ask_user` through `setupAskUser()`, your own through `registerTool` — is sent to the agent as an AG-UI frontend tool. When the agent calls it, the run pauses with the call open; the page runs the tool, shows its result in the row, and calls the agent again with it. For an agent server, the pause is an interrupt whose reason contains `client_tool`, or a call the agent leaves without a result. The page runs the tool as soon as the agent calls it: its `needsApproval` and your approval policy are not consulted, so a tool that needs a person asks in its own handler, as `ask_user` does. ## Approvals [Section titled “Approvals”](#approvals) When the agent pauses on a call that needs a person — an interrupt whose reason is `tool_call` or `confirmation` — the page asks with the same [approval panel](/guides/tools/#require-approval-human-in-the-loop) as for its own tools, and your [approval policy](/components/approval-mode/) can answer first. Every answer goes back to the agent in one `resume`, and the run goes on on the agent’s server, in the same reply. The pause is part of the conversation, so a [saved conversation](/guides/conversation-persistence/) brings the question back after a reload. ## Options [Section titled “Options”](#options)
```ts
import { uuid } from '@aparte/core';
import { createAgUiRunner } from '@aparte/runner-ag-ui';
let conversationId = uuid(); // set it when your app opens a conversation
const runner = createAgUiRunner({
endpoint: '/agui',
headers: async () => ({ authorization: `Bearer ${sessionStorage.getItem('token') ?? ''}` }),
threadId: () => conversationId, // read on every turn
});
```
| Option | What it does | | ---------------- | ----------------------------------------------------------------------------- | | `endpoint` | the agent’s AG-UI URL; required | | `headers` | sent with every request; a function is called for each, for a fresh token | | `fetch` | the `fetch` the runner calls; the global one by default | | `credentials` | for a cross-origin endpoint that authenticates by cookie | | `threadId` | the AG-UI thread a turn runs in: a string, or a function of the run’s options | | `forwardedProps` | `RunAgentInput.forwardedProps`: settings your agent reads from the request | | `context` | `RunAgentInput.context`: `{ description, value }` entries for the agent | These are the `AgUiRunnerOptions`. By default every turn runs in a new thread with the whole history, which suits a stateless agent; give `threadId` when the agent keeps state per thread, as Agno does with its session — the conversation’s id is the usual choice. A paused run resumes in its own thread, whatever `threadId` answers. An agent on another origin than the page has to allow it: answer the preflight (`OPTIONS`) with `Access-Control-Allow-Origin` (the page’s origin) and `Access-Control-Allow-Headers: content-type` plus the headers you send, and with `Access-Control-Allow-Credentials: true` when `credentials` is `'include'`. ## Agno [Section titled “Agno”](#agno)
```py
from agno.agent.agent import Agent
from agno.models.openai import OpenAIResponses
from agno.os import AgentOS
from agno.os.interfaces.agui import AGUI
chat_agent = Agent(model=OpenAIResponses(id="gpt-5.4"))
agent_os = AgentOS(agents=[chat_agent], interfaces=[AGUI(agent=chat_agent)])
app = agent_os.get_app() # POST /agui
```
Agno serves an agent or a team over AG-UI with the `AGUI` interface of its `AgentOS`, at `POST /agui`; give that URL to `createAgUiRunner`. This is [Agno’s AG-UI example](https://docs.agno.com/agent-os/interfaces/ag-ui/introduction). ## What is not supported [Section titled “What is not supported”](#what-is-not-supported) * A person’s message goes as text: images and files are left out. * An interrupt the page cannot answer ends the reply with an error that names its reason: one such as `input_required`, or one that names no `toolCallId`. * A refusal reaches the agent as `{ approved: false }`, without the words the person wrote. ## Next [Section titled “Next”](#next) * [TanStack AI A TanStack AI chat route, through the same runner.](/providers/agent/tanstack-ai/) * [Run the loop on your server aparté's own loop next to your tools, instead of an agent framework.](/guides/server-loop/)
# Put a chat on a TanStack AI route
> Give a TanStack AI chat route a UI — the chat draws its stream, its tool calls and its approvals, and a tool can run in the page — in React, Vue, Svelte, Angular or plain HTML.
## Pass the run’s fields in your route [Section titled “Pass the run’s fields in your route”](#pass-the-runs-fields-in-your-route)
```ts
// app/api/chat/route.ts
import { chat, chatParamsFromRequest, mergeAgentTools, toServerSentEventsResponse } from '@tanstack/ai';
import { openaiText } from '@tanstack/ai-openai';
import { serverTools } from './tools';
export async function POST(request: Request) {
let params;
try { params = await chatParamsFromRequest(request); }
catch (error) { if (error instanceof Response) return error; throw error; } // a malformed body
return toServerSentEventsResponse(chat({
adapter: openaiText('gpt-5.5'), // your adapter: any TanStack AI text adapter
messages: params.messages,
threadId: params.threadId,
runId: params.runId,
parentRunId: params.parentRunId,
resume: params.resume,
tools: mergeAgentTools(serverTools, params.tools),
}));
}
```
[TanStack AI](https://tanstack.com/ai) runs the model, the tools and the approvals, and leaves the interface to you. Read the request with `chatParamsFromRequest` and hand `chat()` the run’s fields with the messages: `resume`, `threadId`, `runId` and `parentRunId` are what let a paused run continue. A route that passes only `messages` streams, but an approval or a tool the page ran never reaches the run. `mergeAgentTools` adds the tools the page declares to yours; a page tool must not share a name with a server tool, since the server’s wins. ## Connect the page [Section titled “Connect the page”](#connect-the-page)
```bash
npm install @aparte/runner-ag-ui @aparte/core
```
```ts
import { AparteClient } from '@aparte/core';
import { createAgUiRunner } from '@aparte/runner-ag-ui';
new AparteClient({
streamRunner: createAgUiRunner({ endpoint: '/api/chat' }),
}).start();
```
The route streams AG-UI, so [`@aparte/runner-ag-ui`](/providers/agent/ag-ui/) draws it as it is: text, reasoning, tool rows, the approval panel. Your route stays a TanStack AI route. Under a framework, hand the same runner to the wrapper’s client: `useAparteClient({ streamRunner })` in React and Vue, `createAparteClient({ streamRunner })` in Svelte, `provideAparte({ clientOptions: { streamRunner } })` in Angular. ## Approvals and tools in the page [Section titled “Approvals and tools in the page”](#approvals-and-tools-in-the-page) * **`needsApproval: true`** on a server tool pauses the run. The page asks with its [approval panel](/guides/tools/#require-approval-human-in-the-loop), and its [approval policy](/components/approval-mode/) can answer first. Approve runs the tool on your server; Reject sends `{ approved: false }`, without the words the person wrote, and the row stays refused. * **A tool that needs the page** — `ask_user` through `setupAskUser()`, your own through `registerTool` — reaches your route as a declared tool, which `mergeAgentTools` adds. TanStack AI pauses on it, the page runs it, and its result resumes the run: TanStack’s `.client()` implementation is not needed, the page’s handler is the implementation. As with any [AG-UI agent](/providers/agent/ag-ui/#tools-the-page-runs), the page asks nobody before running it. It works with `@tanstack/ai` `^0.68.0`. TanStack AI is on 0.x, where a minor can change its wire: check that range when you upgrade it. ## Next [Section titled “Next”](#next) * [AG-UI agents What the stream becomes, the options, and what is not supported.](/providers/agent/ag-ui/) * [Tools and approval The tool rows and the approval panel the route's calls use.](/guides/tools/)
# Connect Claude, Gemini and other vendors through the Vercel AI SDK
> Bring an @ai-sdk/* package — Anthropic, Google, Amazon Bedrock and the others the SDK supports — and the chat streams its replies, reasoning and tool calls.
## Install and register [Section titled “Install and register”](#install-and-register)
```bash
npm install @aparte/core @aparte/provider-ai-sdk ai @ai-sdk/anthropic
```
```ts
import { createAnthropic } from '@ai-sdk/anthropic';
import { createAiSdkProvider } from '@aparte/provider-ai-sdk';
import { aparteGlobalConfig, AparteClient, AparteDirectTransport } from '@aparte/core';
aparteGlobalConfig.registerAIProvider(createAiSdkProvider({
id: 'anthropic',
name: 'Anthropic',
languageModel: (modelId, auth) =>
createAnthropic({
apiKey: typeof auth === 'string' ? auth : auth?.apiKey,
headers: { 'anthropic-dangerous-direct-browser-access': 'true' }, // a key used from the browser
})(modelId),
}));
aparteGlobalConfig.setTransport(new AparteDirectTransport({ byok: true }));
aparteGlobalConfig.setModelConfig({ defaultProvider: 'anthropic', defaultModel: 'claude-sonnet-4-5' });
new AparteClient({
keyResolver: (providerId) => localStorage.getItem(`${providerId}.key`) ?? undefined, // the person's key
}).start();
```
The provider needs `ai` in the range `^7.0.0` beside it, and the `@ai-sdk/*` package of your vendor; its options are an `AiSdkProviderOptions`. `languageModel` is called for each request with the model’s id and what the [key resolver](/providers/#the-key) answered, so a key the person types reaches the vendor’s SDK; a factory that already holds its key ignores `auth`. Stop cancels the vendor’s call. ## Choose the model [Section titled “Choose the model”](#choose-the-model) The SDK has no list of models to fetch, so the [model selector](/components/model-selector/) shows none for this provider: choose the model with `setModelConfig`, as above, or from a control of your own. A `models` option gives the provider a list of its own, which the client reads for a model’s capabilities; a model without `capabilities` is sent the tools. ## What maps to what [Section titled “What maps to what”](#what-maps-to-what) | From the SDK | In the chat | | -------------------------- | ------------------------------------------------------------------ | | text | the reply | | reasoning | a [reasoning block](/components/reasoning/) | | a tool call | a [tool](/guides/tools/) the chat runs, then calls the model again | | the finish, with its usage | the end of the reply, with its tokens, cached ones included | | an error | the message’s error | The provider calls `streamText` for one step at a time: the tools go as declarations without an `execute`, and the chat’s loop runs them. An image in the person’s message goes as an image part. ## In a server route [Section titled “In a server route”](#in-a-server-route) `toModelMessages`, `toToolSet`, `toToolChoice` and `fullStreamToAparteEvents` are exported on their own, to answer the chat from a route that already uses the SDK ([Keep the key on your server](/guides/backend-transport/#if-your-server-already-uses-the-ai-sdk)). `fullStreamToAparteEvents` passes the vendor’s error text along, and a vendor’s refusal can quote parts of the key: replace an `error` event with a message of your own before it leaves the server. ## Next [Section titled “Next”](#next) * [Keep the key on your server The same vendors, with the key on your server.](/guides/backend-transport/) * [OpenAI-compatible The vendors that speak OpenAI's format, with no SDK.](/providers/ai/openai-compat/)
# Connect OpenAI and every OpenAI-compatible endpoint
> One adapter for every endpoint that speaks OpenAI's chat completions — OpenAI, Mistral, OpenRouter, Z.ai, Ollama, LM Studio, and any other compatible URL such as Groq or vLLM.
## Register a preset, or any endpoint [Section titled “Register a preset, or any endpoint”](#register-a-preset-or-any-endpoint)
```bash
npm install @aparte/core @aparte/provider-openai-compat
```
```ts
import { aparteGlobalConfig } from '@aparte/core';
import { createOpenAICompatProvider, presets } from '@aparte/provider-openai-compat';
aparteGlobalConfig.registerAIProvider(createOpenAICompatProvider(presets.MISTRAL));
aparteGlobalConfig.registerAIProvider(createOpenAICompatProvider({
id: 'groq',
name: 'Groq',
baseURL: 'https://api.groq.com/openai/v1',
}));
```
`createOpenAICompatProvider` takes an `OpenAICompatProviderOptions`. A preset is one filled in — a vendor’s base URL and branding — and any other endpoint takes an `id` and its `baseURL`. Spread a preset to change a field: `{ ...presets.OPENAI, baseURL: 'https://my-proxy.example/v1' }`. The key goes through the [key resolver](/providers/#the-key). Nothing is selected until you name a model — `aparteGlobalConfig.setModelConfig({ defaultProvider: 'mistral', defaultModel: 'mistral-small-latest' })` — or the person picks one in the [model selector](/components/model-selector/). | Preset | Base URL | Key | | --------------------------------- | -------------------------------------- | ------------------------------ | | `presets.OPENAI` — OpenAI | `https://api.openai.com/v1` | yours | | `presets.MISTRAL` — Mistral AI | `https://api.mistral.ai/v1` | yours | | `presets.ZAI` — Z.ai (Zhipu) | `https://open.bigmodel.cn/api/paas/v4` | yours | | `presets.OPENROUTER` — OpenRouter | `https://openrouter.ai/api/v1` | yours | | `presets.LMSTUDIO` — LM Studio | `http://localhost:1234/v1` | none, a server on your machine | | `presets.OLLAMA` — Ollama | `http://localhost:11434/v1` | none, a server on your machine | `presets.OPENROUTER` sends OpenRouter’s attribution headers, `HTTP-Referer` with the page’s origin and `X-Title: aparté`; name your app with `{ ...presets.OPENROUTER, extraHeaders: { 'HTTP-Referer': location.origin, 'X-Title': 'My app' } }`. ## The list of models [Section titled “The list of models”](#the-list-of-models) The provider fetches `GET {baseURL}/models`, and the [model selector](/components/model-selector/) lists each model by its `name`, or its `id` when the server sends none; a model’s `context_length` becomes the `contextWindow` the [context gauge](/components/context-gauge/) reads. Each fetched model is marked as streaming and calling tools. A cloud provider fetches nothing without a key; a local one — `isLocal: true`, which the Ollama and LM Studio presets set — lists its models with none. A `models` option gives the provider a list of its own, which the client reads for a model the fetched list does not have: a fetched entry with the same id wins, and the selector shows only the fetched list. ## Fields the OpenAI format does not have [Section titled “Fields the OpenAI format does not have”](#fields-the-openai-format-does-not-have)
```ts
import { aparteGlobalConfig } from '@aparte/core';
import { createOpenAICompatProvider, presets } from '@aparte/provider-openai-compat';
aparteGlobalConfig.registerAIProvider(createOpenAICompatProvider({
...presets.ZAI,
extraBody: { thinking: { type: 'disabled' } },
}));
```
`extraBody` adds fields to every request body, for a vendor’s own switch — here, Z.ai’s `thinking` field — and `extraHeaders` adds headers to every request and to the list of models; spreading a preset with `extraHeaders` replaces its own. The fields the adapter writes itself stay its own: `model`, `messages`, `temperature`, `max_tokens`, `stream`, `stream_options`, `seed`, `tools` and `tool_choice`. ## What it sends and reads [Section titled “What it sends and reads”](#what-it-sends-and-reads) | In the chat | On the wire | | ---------------------------------- | --------------------------------------------------------------------------------------- | | registered tools | `tools`, with `tool_choice: 'auto'`, or the tool a `toolChoice` forces | | an image part in a message | an `image_url` part; an assistant turn that calls tools goes as text only | | `temperature`, `maxTokens`, `seed` | `temperature`, `max_tokens`, `seed` | | the usage of a reply | `stream_options.include_usage`, read back as input, output, total and cached tokens | | reasoning | `reasoning_content` in the stream, drawn as a [reasoning block](/components/reasoning/) | The key goes as `Authorization: Bearer …`, and a key pasted with its `Bearer `is not doubled. ## Without aparté’s client [Section titled “Without aparté’s client”](#without-apartés-client)
```ts
import { readableToAsyncIterable } from '@aparte/core';
import type { AparteChatRequest } from '@aparte/core';
import { createOpenAICompatProvider, presets } from '@aparte/provider-openai-compat';
const provider = createOpenAICompatProvider(presets.MISTRAL);
const request: AparteChatRequest = { messages: [{ role: 'user', content: 'hi' }], modelId: 'mistral-small-latest', stream: true };
const { path, body, headers } = provider.buildRequest(request);
const response = await fetch(`${provider.defaultEndpoint}${path}`, {
method: 'POST',
headers: { 'Content-Type': 'application/json', ...headers, ...provider.authHeaders(process.env['MISTRAL_API_KEY'] ?? '') },
body: JSON.stringify(body),
});
if (!response.ok) throw new Error(`${response.status}: ${await response.text()}`); // an error body has no events
const stop = new AbortController();
for await (const event of readableToAsyncIterable(provider.parseStream(response.body!), stop.signal)) {
// text, thinking, tool_use, error, then done with its usage
}
```
The provider is the wire format alone: build the request, send it yourself, read the stream back as events — in a server route, a worker, another process. It is an `OpenAICompatProvider`, whose `buildRequest`, `authHeaders` and `parseText` are always there; `parseOpenAICompatStream(body)` is the same stream reader on its own. `readableToAsyncIterable` makes the stream something `for await` reads in every browser. ## A model on your machine [Section titled “A model on your machine”](#a-model-on-your-machine) Ollama and LM Studio speak this format at their `/v1` endpoint — for Ollama, rather than its own `/api/chat`. [Local models](/guides/local-models/) covers the browser’s permission, llama.cpp, and picking a model. ## Next [Section titled “Next”](#next) * [Local models Ollama, LM Studio or llama.cpp, from the browser.](/guides/local-models/) * [Vercel AI SDK Anthropic, Google and the vendors this format does not reach.](/providers/ai/ai-sdk/) * [Keep the key on your server The same provider, called from your server.](/guides/backend-transport/)
# Script a model for demos and tests
> Replay turns you wrote — text at a typing pace, reasoning, tool calls, errors — with no key and no network, the same way every time, for a demo page or your own tests.
## Script the replies [Section titled “Script the replies”](#script-the-replies)
```bash
npm install @aparte/core @aparte/provider-scenario @aparte/plugin-marked marked
```
```html
```
```ts
import { aparteGlobalConfig, AparteClient, AparteDirectTransport } from '@aparte/core';
import { createScenarioProvider } from '@aparte/provider-scenario';
import { setupMarkedProvider } from '@aparte/plugin-marked';
setupMarkedProvider(); // draws the **markdown** below; without it, text streams plain
aparteGlobalConfig.registerAIProvider(createScenarioProvider({
turns: [
'Hello! Ask me anything.',
[{ thinking: 'Let me see…' }, { text: 'Here is my **answer**, with *markdown*.' }],
],
}));
aparteGlobalConfig.setTransport(new AparteDirectTransport({ byok: true }));
new AparteClient().start();
```
The provider never calls a model: it replays the turns you wrote, and everything around it — the loop, your tools, the transcript — runs for real. `turns` answers the calls in order and repeats the last one; every call advances, a retry and the second half of a tool round-trip included. Given both `turns` and `scenarios`, `turns` answers. The client adds the person’s message itself. A turn, a `ScenarioTurn`, is a string — one text step — or a list of `ScenarioStep`s: | Step | What it does | | ----------------------- | ------------------------------------------------------------------------------------------------------------------ | | `{ text }` | streams the text, drawn as [Markdown](/guides/markdown/) once a Markdown plugin is set up | | `{ thinking }` | streams reasoning | | `{ tool, input?, id? }` | calls the tool of that name: the loop runs the handler you registered and calls the provider again with its result | | `{ error }` | fails the turn with that message; the steps after it do not run | | `{ wait }` | pauses for that many milliseconds | | `{ usage }` | replaces fields of the usage reported at the end | ## Answer by what was asked [Section titled “Answer by what was asked”](#answer-by-what-was-asked)
```ts
import { createScenarioProvider } from '@aparte/provider-scenario';
createScenarioProvider({
scenarios: {
default: 'Ask me for a haiku, or the weather.',
haiku: { when: /haiku/i, turn: 'Web components hum — \nno framework in the wind, \njust the page, alive.' },
weather: { when: 'weather', turn: [{ text: 'Let me check.' }, { tool: 'get_weather', input: { city: 'Lille' } }] },
forecast: { after: 'get_weather', turn: 'Cloudy, 14 °C. Bring a jacket.' },
},
});
```
A `Scenario` is `{ when?, after?, turn }`, or a bare turn. Each call picks one: after a tool, the one whose `after` names it; otherwise the first whose `when` matches the last message — the person’s, or a tool’s result that no `after` claims — a string as a substring, any case, a RegExp as it is; otherwise `default`, or the first one declared. A tool called here only runs if the app registered one of that name; one that is not fails the call, as it would with a real model. A tool with no `after` route hands its result to whatever that rule picks next. When that is the scenario that called the tool, the call repeats until the client’s `maxTurns` stops it, so the provider warns when you create it. ## Branch on the person’s answer [Section titled “Branch on the person’s answer”](#branch-on-the-persons-answer)
```ts
import { aparteGlobalConfig } from '@aparte/core';
import { createScenarioProvider } from '@aparte/provider-scenario';
import { setupAskUser } from '@aparte/plugin-ask-user';
setupAskUser(); // registers ask_user, and draws its question
aparteGlobalConfig.registerAIProvider(createScenarioProvider({
scenarios: {
start: { turn: [{ tool: 'ask_user', input: { question: 'Where do you run?', options: [{ title: 'browser' }, { title: 'server' }] } }] },
browser: { turn: 'In the browser, then: the direct transport, your key stays local.' },
server: { turn: 'On a server: the backend transport, the key never leaves it.' },
other: { turn: 'Either works: start with the direct transport.' },
},
match: (request, scenarios) => {
const last = request.messages[request.messages.length - 1];
if (last?.role !== 'tool') return 'start'; // a message from the person
const picked = String(last.content).trim(); // the tool's result: "browser" or "server"
return Object.hasOwn(scenarios, picked) ? picked : 'other'; // a decline, an answer of their own
},
}));
```
`after` routes on which tool ran, not on what it answered. To pick the next scenario from the answer — the option chosen in an `ask_user` question, a step of a wizard — read the tool’s result in a `match` of your own, from the last message: earlier results stay in the history of every later call. It returns a scenario’s key, or `undefined` to leave the call to the rule above, which ships as `defaultMatch`; a key it does not know streams an empty turn and says so in the console. Here the creation warning is expected: your `match` routes the result. ## Pace, usage and models [Section titled “Pace, usage and models”](#pace-usage-and-models) `pacing`, a `ScenarioPacing`, is `{ chunk: 12, delay: 24 }` by default — twelve characters every 24 ms — and `pacing: 'instant'` gives a test the whole reply at once. A turn ends with its usage, estimated at four characters a token with its duration, so a [context gauge](/components/context-gauge/) given a `window` moves; a failed turn reports none. The provider is local and needs no key. It offers one model, `scripted`, which streams and calls tools, and the [model selector](/components/model-selector/) lists it. `models` replaces that list; with more than one, pick one with the selector or `setModelConfig`. Its `id` is `scenario` and its `name` `Scripted model` unless its `ScenarioProviderOptions` say otherwise. The `turns` cursor belongs to the provider, so two chats on one provider take turns from the same script: create one per chat, or use `scenarios`, which answers from the request and keeps nothing. ## The ready-made showcase [Section titled “The ready-made showcase”](#the-ready-made-showcase)
```ts
import { aparteGlobalConfig } from '@aparte/core';
import { createScenarioProvider } from '@aparte/provider-scenario';
import { showcase } from '@aparte/provider-scenario/showcase';
aparteGlobalConfig.registerAIProvider(createScenarioProvider({ scenarios: showcase }));
```
`showcase` is the set this site’s live examples run on: Markdown, a table, code, reasoning, tool calls, questions, an artifact, a slow turn and a failure. It ships on its own entry point, so your own scenarios never bring it into your bundle. It answers what the person types: | Scenario | Starts when the message matches | | ---------- | ------------------------------------------- | | `haiku` | `/haiku/i` | | `table` | `/table/i` | | `code` | `/code\|example\|snippet/i` | | `weather` | `/weather\|forecast/i` | | `release` | `/ship it\|release\|checklist/i` | | `survey` | `/survey\|two questions\|a few questions/i` | | `question` | `/question\|ask me\|colou?r scheme/i` | | `artifact` | `/artifact\|document\|write me/i` | | `slow` | `/slow\|think hard\|take your time/i` | | `failure` | `/fail\|error\|crash/i` | It calls `get_weather`, `search_docs`, `read_file`, `write_file`, `run_command` and `ask_user`: register those tools with `aparteGlobalConfig.registerTool` for its tool turns to run, `ask_user` through `setupAskUser()` from [`@aparte/plugin-ask-user`](/guides/elicitation/#let-the-model-ask), and `setupArtifacts()` from the [artifacts plugin](/components/artifact/) for its artifact to be drawn as a card. ## In your tests [Section titled “In your tests”](#in-your-tests)
```ts
import type { AparteChatRequest } from '@aparte/core';
import { playTurn } from '@aparte/provider-scenario';
const request: AparteChatRequest = { messages: [{ role: 'user', content: 'Hi' }], modelId: 'scripted' };
const reader = playTurn([{ text: 'Hello.' }, { tool: 'search', input: { q: 'docs' } }], request, 'instant').getReader();
const events = [];
for (let next = await reader.read(); !next.done; next = await reader.read()) events.push(next.value);
// { type: 'text', delta: 'Hello.' }, then { type: 'tool_use', name: 'search', … }, then { type: 'done', usage }
```
A chat on this provider behaves the same at every run, which is what a test of your app around the chat needs; `playTurn` gives the events of one turn with no provider at all. It stands in for the whole provider: what sits above it runs for real, and what sits below — a wire format, the network, a provider of yours — does not run at all, so test that layer with a fake of its own. ## Next [Section titled “Next”](#next) * [Tools and approval The tools a scripted turn calls, and the approval asked before they run.](/guides/tools/) * [Connect a model The providers that reach a real model, when the script is done.](/providers/)
# Run a model in the browser
> Run a language model inside the page with Transformers.js, on the graphics card through WebGPU or on the processor through WASM — no server, no key — with its download shown, its cache kept and its memory released.
## Register a model [Section titled “Register a model”](#register-a-model)
```bash
npm install @aparte/core @aparte/provider-transformers @huggingface/transformers
```
```ts
import { aparteGlobalConfig, AparteClient } from '@aparte/core';
import { TransformersProvider, registerModel } from '@aparte/provider-transformers';
registerModel({
id: 'onnx-community/Qwen2.5-0.5B-Instruct',
name: 'Qwen2.5 0.5B',
capabilities: ['streaming'],
dtype: 'q4',
});
aparteGlobalConfig.registerAIProvider(TransformersProvider);
new AparteClient({ requestInterceptor: (request) => ({ ...request, maxTokens: request.maxTokens ?? 1024 }) }).start();
```
The model runs in a Web Worker, off the page’s main thread, through the [Transformers.js](https://huggingface.co/docs/transformers.js) you install beside the provider; its weights download once into the browser’s cache. `capabilities` is required: `['streaming']` for a chat model. With one model registered, the chat uses it; with several, name one with `aparteGlobalConfig.setModelConfig({ defaultProvider: 'transformers', defaultModel })`, or let the person pick in the [model selector](/components/model-selector/). A reply stops at 512 new tokens unless the request asks for more, which the `requestInterceptor` above does. This provider sends a model no tools, and a conversation that already holds tool calls runs with those calls and their results left out of the prompt; the console says so once. ## Bundler, CDN and security policy [Section titled “Bundler, CDN and security policy”](#bundler-cdn-and-security-policy) The package starts its worker with `new Worker(new URL('./worker.js', import.meta.url))`, the form Vite and webpack recognise. From a CDN, map `@huggingface/transformers` in the page’s import map beside the provider. When the package is served from another origin than the page — a CDN, an asset host — the provider starts its worker from a `blob:` URL. A Content-Security-Policy then allows `blob:` in `worker-src` (or `script-src`), and, for the model itself, the hosts it comes from in `connect-src` — huggingface.co, and cdn.jsdelivr.net, where Transformers.js fetches its WASM runtime by default — with `'wasm-unsafe-eval'` in `script-src`. ## Show the download [Section titled “Show the download”](#show-the-download)
```ts
import { TransformersProvider } from '@aparte/provider-transformers';
declare const bar: HTMLProgressElement;
bar.max = 100;
await TransformersProvider.prepareModel('onnx-community/Qwen2.5-0.5B-Instruct', (p) => {
if (p.progress !== undefined) bar.value = p.progress; // this file's percentage: each file starts at 0
bar.title = p.file ?? p.status; // 'downloading' (from the network or the cache), 'loading', 'ready'
});
```
A first load downloads the whole model. `prepareModel` downloads and loads it before the first message, reporting each file as a `ModelLoadProgress`, and rejects if the load fails; without it, the first message loads the model and the reply waits with no progress to show. `TransformersProvider.getModelStatus(id)` answers `'ready'` once the model is loaded, `'cached'` when its files are in the cache or while it loads, and `'not-downloaded'` otherwise. The [model selector](/components/model-selector/) lists the registered models and the ones already in the cache. ## The cache and the memory [Section titled “The cache and the memory”](#the-cache-and-the-memory) | Function | What it does | | ----------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `listCachedModels()` | the models in the browser’s cache, each a `CachedModelEntry`: `modelId`, `name`, `sizeBytes`, `loaded` | | `deleteCachedModel(id)` | deletes the cached files whose address holds that id — those of a model whose id extends it too — ending the worker first if the model is loaded; `TransformersProvider.deleteModel` does the same | | `setMaxCachedModels(n)` | how many models the cache keeps: `1` by default, `0` for no limit | | `getLoadedModelId()` | the model in the worker’s memory, or `null` | | `terminateWorker()` | ends the worker and the model in memory; a reply still streaming ends with an error | The worker holds one model in memory at a time, for the whole page: two chats on the same model share it, and two chats on different models take turns, each reply loading its own model again — the console warns once. Another model replaces the one in memory; closing a chat does not release it. `terminateWorker()` does, on leaving the route or behind an “unload” button, and the next message starts a new worker. When a model finishes loading and the cache holds more models than `setMaxCachedModels` allows, the others are deleted, the earliest downloaded first. The count takes in every Hugging Face model in the site’s browser cache, ones your app downloaded with Transformers.js for something else included. At the default of one, switching to another model deletes the previous one’s weights, so switching back downloads it again; `setMaxCachedModels(2)` keeps both, and `0` keeps everything. ## Choose the device [Section titled “Choose the device”](#choose-the-device)
```ts
import { aparteGlobalConfig } from '@aparte/core';
import { detectHardware, registerModel, setHardwareTierModels } from '@aparte/provider-transformers';
registerModel({ id: 'onnx-community/Qwen2.5-0.5B-Instruct', name: 'Qwen2.5 0.5B', capabilities: ['streaming'], dtype: 'q4' });
registerModel({ id: 'onnx-community/Qwen3-0.6B-ONNX', name: 'Qwen3 0.6B', capabilities: ['streaming'], dtype: 'q4' });
setHardwareTierModels({ low: 'onnx-community/Qwen2.5-0.5B-Instruct', high: 'onnx-community/Qwen3-0.6B-ONNX' });
const { recommendedModelId } = await detectHardware();
aparteGlobalConfig.setModelConfig({ defaultProvider: 'transformers', defaultModel: recommendedModelId });
```
A model runs on the graphics card when the browser has a WebGPU adapter, and on the processor through WASM otherwise; a model that fails to load on the graphics card is loaded again on WASM, and the console says so. `setComputeDevice` forces one, a `ComputeDevice`: `'webgpu'`, `'wasm'`, or `'auto'`, the default; `getComputeDevice()` reads it back. It is one setting for the page — a `device` on a registered model is not read — and it applies when a model loads: to change it afterwards, call `terminateWorker()`, and the next message loads the model on the new device. `detectHardware()` answers a `HardwareProfile`: it looks for a WebGPU adapter and sorts the machine into a `low`, `mid` or `high` tier — no WebGPU adapter, whatever the hardware, or under 4 GB of memory is low, under 8 GB is mid — then answers the model you gave that tier, or the `high` one for a tier you left out. Only Chromium reports the memory, and at most 8 GB; in other browsers the machine counts as 4 GB. ## A model that reads images [Section titled “A model that reads images”](#a-model-that-reads-images)
```ts
import { registerModel } from '@aparte/provider-transformers';
registerModel({
id: 'HuggingFaceTB/SmolVLM-256M-Instruct',
name: 'SmolVLM 256M',
task: 'image-text-to-text',
capabilities: ['streaming', 'vision'],
dtype: { embed_tokens: 'fp16', vision_encoder: 'q4', decoder_model_merged: 'q4' },
});
```
`task` picks the runner that loads the model: `'text-generation'`, the default, for a chat model, and `'image-text-to-text'` for one that reads the images the person [attaches](/guides/attachments/) — SmolVLM, Qwen2-VL, Gemma 3 and the other models Transformers.js loads as `AutoModelForImageTextToText`. `dtype` is one precision for the whole model, or one per part, as here. A text model sent an image answers without it, and the console says the image was left out. Stop interrupts the model itself, not only the reply on screen. ## A runner of your own [Section titled “A runner of your own”](#a-runner-of-your-own)
```ts
import { registerModel } from '@aparte/provider-transformers';
registerModel({ id: 'my-org/my-model', name: 'Mine', capabilities: ['streaming'], runner: '/runners/mine.js' });
```
```js
// /runners/mine.js — served as it is (from public/ under a bundler); the worker imports it by URL
export async function createRunner(ctx) {
const { AutoTokenizer, AutoModelForCausalLM, TextStreamer, InterruptableStoppingCriteria } = ctx.transformers;
const tokenizer = await AutoTokenizer.from_pretrained(ctx.modelId);
const model = await AutoModelForCausalLM.from_pretrained(ctx.modelId, {
dtype: ctx.dtype,
device: ctx.device,
});
return {
async generate({ messages, options, emit, signal }) {
if (signal.aborted) return;
const turns = messages
.filter((m) => m.role === 'user' || m.role === 'assistant' || m.role === 'system')
.map((m) => ({ role: m.role, content: typeof m.content === 'string' ? m.content : m.content.filter((p) => p.type === 'text').map((p) => p.text).join('') }));
const inputs = tokenizer.apply_chat_template(turns, { add_generation_prompt: true, return_dict: true });
const stopping = new InterruptableStoppingCriteria();
signal.addEventListener('abort', () => stopping.interrupt(), { once: true });
await model.generate({
...inputs,
max_new_tokens: options.maxTokens ?? 512,
streamer: new TextStreamer(tokenizer, {
skip_prompt: true, skip_special_tokens: true,
callback_function: (text) => emit({ type: 'text', delta: text }),
}),
stopping_criteria: stopping,
});
},
async command(name, payload) { /* what is not a reply: swap an adapter, warm a cache */ },
dispose() { return model.dispose(); },
};
}
```
`runner` names an ES module that exports `createRunner`, and it wins over `task`. Give it an absolute path: a relative one is resolved against the current page’s URL. The module runs in the worker, where neither your bundler nor your import map applies, so it imports nothing by package name: it gets the same Transformers.js as the built-in runners as `ctx.transformers`, and a model they cannot load keeps the worker, the queue, the progress, Stop and the cache. `messages` arrive with their content parts, and the runner decides what the model reads. `emit` sends `text`, `thinking`, `tool_use`, `done` or `error`; `signal` fires on Stop; `ctx.progress` and `ctx.warn` reach the page; `ctx.device` is `'webgpu'` or `'wasm'`, `'auto'` already resolved. `runnerCommand(modelId, name, payload)` calls a runner’s `command` from the page, after the replies already waiting; when that model is not the one in memory, it is loaded first, in its place, with no progress to show. The contract’s types are exported — `TransformersRunner`, `RunnerContext`, `RunnerGenerateInput`, `RunnerProgress`, `CreateRunner`, `RunnerModule`, `BuiltInRunner`, `TransformersModule` — and a model’s options are a `TransformersModelConfig`. ## Next [Section titled “Next”](#next) * [Local models A model on a server of your own machine instead: Ollama, LM Studio, llama.cpp.](/guides/local-models/) * [Model selector Let the person pick the model, cached ones included.](/components/model-selector/)