Sybil-2/docs/api/streaming-chat.md

Streaming Chat API Contract

This document defines the server-sent events (SSE) contract for chat completions.

Endpoint:

• POST /v1/chat-completions/stream
• POST /v1/chats/:chatId/stream/attach

Transport:

• HTTP response uses Content-Type: text/event-stream; charset=utf-8
• Events are emitted in SSE format (event: ..., data: ...)
• Request body is JSON
• Request body supports the same inline attachment schema and limits documented in docs/api/rest.md.

Authentication:

• Same as REST endpoints (Authorization: Bearer <token> when token mode is enabled)

Request Body

{
  "chatId": "optional-chat-id",
  "persist": true,
  "clientRequestId": "optional-client-generated-id",
  "provider": "openai|anthropic|xai|gemini|hermes-agent",
  "model": "string",
  "messages": [
    {
      "role": "system|user|assistant|tool",
      "content": "string",
      "name": "optional",
      "attachments": [
        {
          "kind": "image",
          "id": "attachment-id",
          "filename": "photo.jpg",
          "mimeType": "image/jpeg",
          "sizeBytes": 12345,
          "dataUrl": "data:image/jpeg;base64,..."
        },
        {
          "kind": "text",
          "id": "attachment-id",
          "filename": "notes.md",
          "mimeType": "text/markdown",
          "sizeBytes": 4567,
          "text": "# Notes\\n...",
          "truncated": false
        }
      ]
    }
  ],
  "additionalSystemPrompt": "optional one-off system prompt",
  "enabledTools": ["web_search", "fetch_url"],
  "temperature": 0.2,
  "maxTokens": 256
}

Notes:

• persist defaults to true.
• If persist is true and chatId is omitted, backend creates a new chat.
• If chatId is provided, backend validates it exists.
• If persist is false, chatId must be omitted. Backend does not create a chat and does not persist input messages, tool-call messages, assistant output, or LlmCall metadata.
• clientRequestId is optional and is only valid for a persisted stream with a chatId. Clients should generate one stable, unique value per user submission and reuse it when retrying a disconnected request.
• A retry with the same chatId and clientRequestId attaches to and replays the matching active stream. If that submission already completed, the endpoint replays meta and done without invoking the provider again. This makes retrying the initial streaming POST idempotent.
• clientRequestId values may be up to 128 characters. The server stores the value in metadata.clientRequestId on the submitted user message and completed assistant message.
• For persisted streams, backend stores only new non-assistant input history rows to avoid duplicates.
• additionalSystemPrompt, when present directly or loaded from stored chat settings, is prepended to the provider request as a system message and is not inserted into the persisted chat transcript by this endpoint.
• enabledTools limits Sybil-managed tools for this request. When omitted for a saved chat, the stored chat setting is used; otherwise all available tools are enabled by default. An empty array disables Sybil-managed tools.
• maxTokens is optional. For anthropic, when omitted the backend requests the selected model's maximum output token limit from Anthropic's Models API and uses that as max_tokens; if the model limit cannot be loaded, the fallback is 128000. For other providers, omitted maxTokens is not sent as an explicit cap.
• Attachments are optional and are persisted under message.metadata.attachments on stored user messages when persist is true.

Persisted chat streams with a chatId are backend-owned active runs:

• Once started, the backend keeps the stream running even if the HTTP client disconnects or refreshes.
• While running, GET /v1/active-runs includes the chatId.
• Starting a second persisted stream for the same active chatId returns 409, unless its clientRequestId matches the active submission, in which case the existing stream is replayed.
• Clients can reattach with POST /v1/chats/:chatId/stream/attach.

Attach Endpoint

POST /v1/chats/:chatId/stream/attach

• Body: none.
• Response uses the same text/event-stream transport and event names as POST /v1/chat-completions/stream.
• Replays buffered events for the active in-memory stream, then emits new events until done or error.
• Returns 404 { "message": "active chat stream not found" } if no stream is currently active for that chat.
• Authentication is the same as all other API endpoints.

This endpoint is intended for clients that restored an active chatId from GET /v1/active-runs, especially after browser refresh. Replayed delta events may include text that was originally emitted before the client attached.

Event Stream Contract

Event order:

1. Exactly one meta
2. Zero or more tool_call
3. Zero or more delta
4. Exactly one terminal event: done or error

Each tool invocation can emit multiple tool_call events with the same toolCallId. The backend emits status: "initiated" before the tool starts executing, then emits status: "completed" or status: "failed" when execution finishes. Clients should upsert by toolCallId instead of appending each event.

meta

{
  "type": "meta",
  "chatId": "chat-id-or-null",
  "callId": "llm-call-id-or-null",
  "provider": "openai",
  "model": "gpt-4.1-mini"
}

For persist: false streams, chatId and callId are null.

delta

{ "type": "delta", "text": "next chunk" }

text may contain partial words, punctuation, or whitespace.

tool_call

{
  "toolCallId": "call_123",
  "name": "web_search",
  "status": "initiated",
  "summary": "Searching web for 'latest CPI release'.",
  "args": { "query": "latest CPI release" },
  "startedAt": "2026-03-02T10:00:00.000Z"
}

Terminal tool-call event:

{
  "toolCallId": "call_123",
  "name": "web_search",
  "status": "completed",
  "summary": "Performed web search for 'latest CPI release'.",
  "args": { "query": "latest CPI release" },
  "startedAt": "2026-03-02T10:00:00.000Z",
  "completedAt": "2026-03-02T10:00:00.820Z",
  "durationMs": 820,
  "resultPreview": "{\"ok\":true,...}"
}

status is one of initiated, completed, or failed. completedAt and durationMs are only present on terminal events. error is present on failed terminal events; resultPreview is present on terminal events when available.

done

{
  "type": "done",
  "text": "full assistant response",
  "usage": {
    "inputTokens": 123,
    "outputTokens": 456,
    "totalTokens": 579
  }
}

usage may be omitted when provider does not expose final token accounting for stream mode.

error

{ "type": "error", "message": "provider timeout" }

Provider Streaming Behavior

• openai: backend uses OpenAI's Responses API and may execute internal function tool calls (web_search, fetch_url, optional codex_exec, and optional shell_exec) before producing final text.
• anthropic: backend uses Anthropic's Messages API and may execute the same internal tools with tool_use/tool_result content blocks before producing final text.
• xai: backend uses xAI's OpenAI-compatible Chat Completions API and may execute the same internal tool calls before producing final text.
• gemini: backend uses Google's native Gemini streamGenerateContent API and may execute the same internal tool calls before producing final text.
• fetch_url sends browser-like navigation headers for outbound URL requests to reduce false 403s from sites that reject generic server clients.
• hermes-agent: backend uses the configured Hermes Agent OpenAI-compatible Chat Completions API. Sybil does not add its own tool definitions for this provider; Hermes Agent handles its own tools server-side. Custom Hermes stream events are normalized away unless they produce text deltas in this SSE contract.
• openai: image attachments are sent as Responses input_image items; text attachments are sent as input_text items.
• gemini: image attachments are sent as native Gemini inlineData parts; text attachments are inlined as text parts.
• xai and hermes-agent: image attachments are sent as Chat Completions content parts; text attachments are inlined as text parts.
• openai: Responses calls that can enter the server-managed tool loop use store: true so reasoning and function-call items can be passed between tool rounds.
• anthropic: streamed via event stream; emits delta from content_block_delta with text_delta, and emits normalized tool_call SSE events when Anthropic tool_use blocks are executed. Image attachments are sent as base64 image blocks and text attachments are appended as text blocks.
• web_search uses CHAT_WEB_SEARCH_ENGINE: exa (default), brave (requires BRAVE_SEARCH_API_KEY), or searxng (requires SEARXNG_BASE_URL; the instance must allow format=json). This only affects chat-mode tool calls, not search-mode endpoints.
• Brave searches are queued and evenly paced according to the shortest window in Brave's X-RateLimit-Policy response header. The backend also honors X-RateLimit-Remaining/X-RateLimit-Reset and retries 429 responses up to three times with reset-aware exponential backoff; quota resets beyond the bounded retry window fail immediately.
• codex_exec is available only when CHAT_CODEX_TOOL_ENABLED=true. It SSHes to CHAT_CODEX_REMOTE_HOST, creates/uses CHAT_CODEX_REMOTE_WORKDIR, and runs codex exec --dangerously-bypass-approvals-and-sandbox --skip-git-repo-check <non-interactive wrapped prompt> there with SSH stdin closed. Prefer CHAT_CODEX_SSH_KEY_PATH with a read-only mounted private key; CHAT_CODEX_SSH_PRIVATE_KEY_B64 is also supported.
• shell_exec is available only when CHAT_SHELL_TOOL_ENABLED=true. It uses the same devbox SSH configuration, starts in CHAT_CODEX_REMOTE_WORKDIR, and runs non-interactive shell commands there with SSH stdin closed, not inside the Sybil server container.
• CHAT_MAX_TOOL_ROUNDS controls how many model/tool result cycles may occur before the backend returns a tool-call limit message; default is 100.

Tool-enabled streaming notes (openai/anthropic/xai/gemini):

• Stream still emits standard meta, delta, done|error events.
• Stream may emit tool_call events while tool calls are executed.
• delta events carry assistant text and are emitted incrementally for normal text rounds. The backend may buffer model-native text briefly while determining whether a provider round contains tool calls.
• OpenAI Responses stream events are normalized by the backend into this SSE contract; clients do not consume OpenAI's raw Responses stream event names.

Persistence + Consistency Model

Backend database remains source of truth.

For persisted streams:

• Client may optimistically render accumulated delta text.
• Backend emits initiated tool-call events without persisting them.
• Backend persists each completed or failed tool call as a tool message before emitting its terminal tool_call SSE event, so chat detail refreshes can show completed tool calls while the assistant response is still running.

On successful persisted completion:

• Backend persists assistant Message and updates LlmCall usage/latency in a transaction.
• Backend then emits done.

On persisted failure:

• Backend records call error and emits error.

For persist: false streams:

• Client may render the same meta, tool_call, delta, and terminal events.
• Backend does not write any chat, message, tool-call log, assistant output, or call metadata rows.
• done.text is the canonical assistant text if the client later imports the result into a saved chat.

Client recommendation (for iOS/web):

1. Render deltas in real time for UX.
2. On done, refresh chat detail from REST (GET /v1/chats/:chatId) and use DB-backed data as canonical.
3. On error, preserve user input and show retry affordance.

SSE Parsing Rules

• Concatenate multiple data: lines with newline before JSON parse.
• Event completes on blank line.
• Ignore unknown event names for forward compatibility.

Example Stream

event: meta
data: {"type":"meta","chatId":"c1","callId":"k1","provider":"openai","model":"gpt-4.1-mini"}

event: delta
data: {"type":"delta","text":"Hello"}

event: delta
data: {"type":"delta","text":" world"}

event: done
data: {"type":"done","text":"Hello world"}