# `LlamaCppEx.Server`
[🔗](https://github.com/nyo16/llama_cpp_ex/blob/main/lib/llama_cpp_ex/server.ex#L1)

GenServer for continuous batched multi-sequence inference.

Manages a shared model/context and serves multiple concurrent callers
using a slot pool with continuous batching — one forward pass per tick
with decode tokens and prefill chunks mixed in a single batch.

## Example

    {:ok, server} = LlamaCppEx.Server.start_link(
      model_path: "model.gguf",
      n_gpu_layers: -1,
      n_parallel: 4,
      n_ctx: 8192
    )

    # Sync generation
    {:ok, text} = LlamaCppEx.Server.generate(server, "Once upon a time", max_tokens: 100)

    # Streaming
    LlamaCppEx.Server.stream(server, "Tell me a story", max_tokens: 200)
    |> Enum.each(&IO.write/1)

## Telemetry

The server emits the following telemetry events:

### `[:llama_cpp_ex, :server, :tick]`

Emitted after each batch forward pass.

Measurements:

  * `:batch_size` - Total tokens in the batch.
  * `:decode_tokens` - Number of decode (generation) tokens.
  * `:prefill_tokens` - Number of prefill (prompt) tokens.
  * `:active_slots` - Slots currently prefilling or generating.
  * `:queue_depth` - Requests waiting for a slot.
  * `:eval_ms` - Forward pass wall time in milliseconds.

Metadata:

  * `:server` - PID of the server process.

### `[:llama_cpp_ex, :server, :request, :start]`

Emitted when a slot is assigned to a request and prefill begins.

Measurements:

  * `:prompt_tokens` - Number of prompt tokens.
  * `:prefix_cache_tokens` - Number of prompt tokens reused from the KV
    prefix cache (`0` when `cache_prompt: false`).

Metadata:

  * `:server` - PID of the server process.
  * `:seq_id` - Slot sequence ID.
  * `:mode` - `:generate` or `:stream`.

### `[:llama_cpp_ex, :server, :request, :done]`

Emitted when a request (generate or stream) completes.

Measurements:

  * `:prompt_tokens` - Number of prompt tokens.
  * `:generated_tokens` - Number of tokens generated.
  * `:duration_ms` - Total request duration in milliseconds.
  * `:ttft_ms` - Time to first token in milliseconds.
  * `:prompt_eval_rate` - Prompt evaluation speed (tokens/sec).
  * `:generation_rate` - Generation speed (tokens/sec).
  * `:prefix_cache_tokens` - Number of prompt tokens skipped via prefix cache.
  * `:prefix_cache_ratio` - Ratio of cached to total prompt tokens (0.0–1.0).

Metadata:

  * `:server` - PID of the server process.
  * `:seq_id` - Slot sequence ID (integer).
  * `:mode` - `:generate` or `:stream`.
  * `:stop_reason` - `:eog` (end-of-generation token sampled),
    `:max_tokens` (request `max_tokens` reached), or `:cancelled` (consumer
    died or cancelled the request).

### `[:llama_cpp_ex, :server, :kv_pressure]`

Emitted when a forward pass hit KV-cache pressure (`llama_decode == 1`) and
the server recovered by purging idle slots' cached prefixes and/or splitting
the batch.

Measurements:

  * `:purged_slots` - Number of idle slots whose cached KV was dropped.
  * `:batch_splits` - Number of times the batch was halved to fit.

Metadata:

  * `:server` - PID of the server process.
  * `:purged_seq_ids` - Sequence IDs whose caches were purged.

### `[:llama_cpp_ex, :server, :prefix_instability]`

Emitted when a cache-eligible request matches only 10–50% of a slot's
cached history — the signature of a chat template that rewrites earlier
turns (e.g. stripping thinking blocks), silently defeating prefix caching.

Measurements:

  * `:matched_tokens` - Length of the common prefix actually reusable.
  * `:cached_tokens` - Length of the cached history that was expected to match.

Metadata:

  * `:server` - PID of the server process.
  * `:seq_id` - Slot sequence ID.

### `[:llama_cpp_ex, :server, :ram_cache]`

Emitted on level-2 RAM prompt cache activity (see `:prompt_cache_ram_mb`).

Measurements:

  * `:bytes` - Size of the entry involved.
  * `:tokens` - Cached prefix length of the entry involved.
  * `:total_bytes` - Cache size after the operation.
  * `:entries` - Entry count after the operation.

Metadata:

  * `:server` - PID of the server process.
  * `:op` - `:save`, `:restore`, or `:evict`.

### `[:llama_cpp_ex, :server, :request, :exception]`

Emitted when an inference error aborts an active request (e.g. the
underlying `batch_eval` returns an error). Measurement shape matches
`:done` so handlers can aggregate them together; `:stop_reason` is
`:error` and the failure reason is in `:reason`.

Metadata:

  * `:server` - PID of the server process.
  * `:seq_id` - Slot sequence ID.
  * `:mode` - `:generate` or `:stream`.
  * `:stop_reason` - `:error`.
  * `:reason` - The underlying failure term from the NIF.

# `cancel`

```elixir
@spec cancel(GenServer.server(), reference()) :: :ok
```

Cancels an in-flight or queued stream request by its subscription reference.

The slot stops being scheduled immediately and is freed for other requests
(its prefix cache is retained per the request's `:cache_prompt`). Consumer
death is detected automatically via monitors — explicit cancel is for
consumers that stop reading without exiting. `Server.stream/3` and
`stream_tokens/3` call this from their cleanup, so halting those streams
early (e.g. `Enum.take/2`) cancels generation instead of burning batch
budget to `max_tokens`.

# `child_spec`

Returns a specification to start this module under a supervisor.

See `Supervisor`.

# `complete_tokens`

```elixir
@spec complete_tokens(GenServer.server(), [integer()], keyword()) ::
  {:ok,
   %{
     text: String.t(),
     completion_tokens: non_neg_integer(),
     finish_reason: atom()
   }}
  | {:error, term()}
```

Like `generate_tokens/3`, but returns completion metadata alongside the text.

Returns `{:ok, %{text: text, completion_tokens: n, finish_reason: reason}}`
where `reason` is `:eog` or `:max_tokens`. Used by the OpenAI-shaped
`LlamaCppEx.chat_completion/3` when routed through a server.

# `fetch_model`

```elixir
@spec fetch_model(GenServer.server()) ::
  {:ok, LlamaCppEx.Model.t()} | {:error, term()}
```

Returns the model struct for external tokenization, or an error.

The model resource is reference-counted and thread-safe for read-only
operations like tokenization. Served from `LlamaCppEx.Registry` — an ETS read,
no round-trip through the server's mailbox.

Returns `{:error, :noproc}` when `server` does not resolve to a live process or
died on its way down, and `{:error, :not_ready}` when the server is still
loading its model (the window between `start_link/1` returning and
`handle_continue/2` finishing).

A server whose model load *failed* returns `{:error, {:load_failed, reason}}`:
`handle_continue/2` stops with that reason, so the `:get_model` call exits with
it. That escaped the previous three `catch` clauses — from inside both
`Stream.resource` start-functions, where nothing can catch it — even though the
`@spec` promised a total function.

# `generate`

```elixir
@spec generate(GenServer.server(), String.t(), keyword()) ::
  {:ok, String.t()} | {:error, term()}
```

Generates text synchronously. Blocks until generation is complete.

## Options

  * `:max_tokens` - Maximum tokens to generate. Defaults to `256`.
  * `:timeout` - Call timeout in ms. Defaults to
    `60000`.
  * `:cache_prompt` - Reuse/retain this request's KV prefix on the slot.
    Defaults to the server-level `:cache_prompt` setting.
  * `:session` - Any term identifying a conversation. Requests with the same
    session are routed to the same slot whenever it is free, keeping their
    cached prefix intact under concurrency.
  * `:cache_scope` - Trust boundary for KV prefix reuse. A request only reuses
    a cached prefix — from its own slot, from another slot, or from the RAM
    prompt cache — when the cached content was produced under the *same*
    scope. Defaults to `nil`, a single shared pool, which is only safe when
    every caller of this server is in one trust domain. In a multi-tenant
    deployment set it to the tenant id: prefix reuse is a KV read, so two
    tenants whose prompts share a system prompt would otherwise be able to
    inherit each other's cache. Unlike `:session`, this does not affect slot
    routing, only what may be reused.
  * Sampling options (`:temp`, `:top_k`, `:top_p`, `:min_p`, `:seed`,
    `:penalty_repeat`, `:penalty_freq`, `:penalty_present`, `:grammar`,
    `:grammar_root`) - override the server-level defaults for this request.

# `generate_tokens`

```elixir
@spec generate_tokens(GenServer.server(), [integer()], keyword()) ::
  {:ok, String.t()} | {:error, term()}
```

Generates text from pre-tokenized input. Blocks until generation is complete.

Use `get_model/1` to obtain the model for tokenization outside the server.

## Options

  * `:max_tokens` - Maximum tokens to generate. Defaults to `256`.
  * `:timeout` - Call timeout in ms. Defaults to
    `60000`.

# `get_model`

```elixir
@spec get_model(GenServer.server()) :: LlamaCppEx.Model.t()
```

Returns the model struct for external tokenization.

Raises when the server is not running or has not finished loading. Use
`fetch_model/1` when either is a possibility.

The previous `@spec` claimed a total function while the implementation exited
with `{:noproc, ...}` on a dead server, so callers had no documented way to
handle it.

# `get_stats`

```elixir
@spec get_stats(GenServer.server()) :: map()
```

Returns a snapshot of the server's current state.

# `request_option_keys`

```elixir
@spec request_option_keys() :: [atom()]
```

The per-request options this server accepts, beyond `:max_tokens`/`:timeout`.

Follows the same ownership rule as `LlamaCppEx.Sampler.option_keys/0`: callers
that forward user options into a server — `LlamaCppEx.chat_completion/3` and
`LlamaCppEx.stream_chat_completion/3` — select them with this function instead
of keeping their own copy. Their copy had already drifted once, silently
rejecting `:cache_scope`.

# `start_link`

```elixir
@spec start_link(keyword()) :: GenServer.on_start()
```

Starts the server.

## Options

  * `:model_path` (required) - Path to the GGUF model file.
  * `:n_gpu_layers` - GPU layers. Defaults to `99`.
  * `:n_ctx` - Total context size (shared across slots). Defaults to `8192`.
  * `:n_parallel` - Number of concurrent slots. Defaults to `4`.
  * `:n_batch` - Max tokens per forward pass. Defaults to `min(n_ctx, 2048)`.
    Bounds worst-case tick latency: one huge prompt can occupy at most
    `n_batch` tokens of a tick, so decode tokens of other slots are never
    delayed by more than one `n_batch`-sized pass. Raise it for pure batch
    throughput (fewer, larger passes); lower it (or lower `:chunk_size`) for
    smoother streaming latency under mixed load.
  * `:chunk_size` - Max prefill tokens per slot per tick. Defaults to `512`.
  * `:max_queue` - Max queued requests waiting for a slot. When the bound is
    hit, calls return `{:error, :queue_full}` immediately and streams emit a
    single `{:error, :queue_full}` element — no silent queueing until the
    call timeout. `0` means unlimited, which is **not** the default: at `0`
    the reject branch is dead code, the documented `:queue_full` error can
    never fire, and each queued entry holds a full token list, so a burst is
    bounded only by memory. Defaults to `64`.
  * `:cache_prompt` - Retain KV cache between requests on the same slot for
    prefix reuse. Defaults to `true` (matching llama-server). Overridable
    per request via the `:cache_prompt` option on `generate/3` and friends.
  * `:kv_unified` - Share one KV buffer across all slots instead of splitting
    `n_ctx` evenly (`n_ctx/n_parallel` each). Enables cross-slot prefix
    sharing: a system prompt cached by any slot is adopted by every other
    slot via a metadata-only copy. Slots then compete for the shared `n_ctx`
    budget, and idle slots' caches are purged under KV pressure. Defaults to
    `true`. Set `false` for strictly isolated per-slot budgets.
  * `:prompt_cache_ram_mb` - Byte budget (in MB) for the level-2 RAM prompt
    cache: when a slot's cached prefix is about to be destroyed it is
    serialized to RAM and can be restored later instead of re-prefilling.
    State blobs are KV-sized (up to hundreds of MB for long contexts) —
    entries larger than the budget are never stored, so a small budget
    degrades to "disabled" rather than OOM. Defaults to `0` (off).
  * `:batch_strategy` - Batch building strategy module. Defaults to
    `LlamaCppEx.Server.Strategy.DecodeMaximal`. See `LlamaCppEx.Server.BatchStrategy`.
  * Sampling options: `:temp`, `:top_k`, `:top_p`, `:min_p`, `:seed`, `:penalty_repeat`,
    `:penalty_freq`, `:penalty_present`, `:grammar`, `:grammar_root`.
  * Context tuning options are forwarded to `LlamaCppEx.Context.create/2` —
    `:n_threads`, `:n_threads_batch`, `:n_ubatch`, `:type_k`, `:type_v`,
    `:flash_attn`, `:offload_kqv`, `:op_offload`, the RoPE/YaRN options,
    `:attention_type`, `:no_perf` and `:swa_full`. See
    `LlamaCppEx.Context.tuning_option_keys/0` for the authoritative list.
    `:n_ctx`, `:n_batch`, `:n_seq_max` and `:kv_unified` are set by the server
    from the options above and cannot be overridden here.
  * Model loading options are forwarded to `LlamaCppEx.Model.load/2` —
    `:main_gpu`, `:split_mode`, `:tensor_split`, `:use_mmap`, `:use_mlock`,
    `:use_direct_io`, `:check_tensors` and `:rpc_servers`. The three load flags
    collapse into llama.cpp's single `load_mode`: `:use_direct_io` wins
    outright, otherwise `:use_mlock` and `:use_mmap` combine; see
    `LlamaCppEx.Model.load/2`. `:rpc_servers` registers remote endpoints before
    the load so their devices can hold part of the model — see
    `LlamaCppEx.RPC`, including the caveat that a peer failure aborts the VM.
  * GenServer options like `:name`.

# `start_option_keys`

```elixir
@spec start_option_keys() :: [atom()]
```

The options `start_link/1` accepts.

Exposed for the same reason as `request_option_keys/0`: callers that forward
user options into a server must select them rather than guess. The one caller
that guessed — `LlamaCppEx.ModelManager.ModelIO` — used a `Keyword.drop/2`
denylist, so `:vocab_only` reached `init/1` and raised there.

# `stream`

```elixir
@spec stream(GenServer.server(), String.t(), keyword()) :: Enumerable.t()
```

Returns a stream of generated text chunks.

If the request is rejected (`:queue_full`), fails mid-generation, or a chunk
does not arrive within `:timeout`, the stream emits a single
`{:error, reason}` element and halts — consumers that need to distinguish
errors from text should match on it. A per-token timeout emits
`{:error, :timeout}` and cancels the request server-side; it used to truncate
the stream silently, which is indistinguishable from a completed generation.

## Options

  * `:max_tokens` - Maximum tokens to generate. Defaults to `256`.
  * `:timeout` - Per-token timeout, and the budget for being admitted to a slot
    or the queue. Defaults to `30000`.

Also accepts the per-request options documented on `generate/3`.

# `stream_tokens`

```elixir
@spec stream_tokens(GenServer.server(), [integer()], keyword()) :: Enumerable.t()
```

Returns a stream of generated text chunks from pre-tokenized input.

Emits a single `{:error, reason}` element and halts on rejection, mid-generation
failure, or a per-token timeout — see `stream/3`.

## Options

  * `:max_tokens` - Maximum tokens to generate. Defaults to `256`.
  * `:timeout` - Per-token timeout, and the budget for being admitted to a slot
    or the queue. Defaults to `30000`.

---

*Consult [api-reference.md](api-reference.md) for complete listing*