ADR 009: Multi-Model Manager

Copy Markdown View Source

Status

Accepted

Context

The library could load and serve one model at a time: Model.load/2 returns a %Model{} handle, and LlamaCppEx.Server (ADR 006) wraps a single model + context for batched inference. Running two or more models simultaneously — e.g. a chat model and an embedding model, or hot-swapping models on a fixed-memory box — required the caller to track handles, pids, and memory by hand.

We wanted a layer that keeps several models resident, routes requests to them by id, and exposes a clean load/unload abstraction, while reusing what already exists: the Hub HuggingFace downloader/cache and the batching Server.

Key constraints from the existing design:

  • A %Model{} is a refcounted, read-only, thread-shareable NIF resource. Inference NIFs run on dirty schedulers and do not block the BEAM. Concurrency does not require one process per model — many callers can hit one model at once.
  • Model cleanup is GC-only: the RAII destructor (ADR 002) runs when the last reference is dropped. There is no explicit free NIF.
  • Server loads its own model + context internally and has no embedding path (Embedding.embed/3 builds its own embeddings: true context).
  • The library has no Application/supervision tree, and shouldn't impose one on dependents that don't use this feature.

Decision

Add an opt-in LlamaCppEx.ModelManager (a singleton GenServer) plus a LlamaCppEx.ModelSupervisor that starts a Registry, a DynamicSupervisor, and the manager.

ETS ownership — writes serialize, reads bypass

The manager owns a named, :protected, read_concurrency: true ETS table of model entries. Load/unload writes serialize through the GenServer; inference-time lookups (generate/stream/chat/embed) read the ETS table directly from the caller process. This keeps the manager off the inference hot path — a generation request never enters the manager's mailbox, so the singleton is not a throughput bottleneck.

The slow part of a load (Hub download + native model load) runs in a supervised Task, not on the manager process, so a long load does not head-of-line-block other lifecycle calls (unload, set_default, concurrent loads). Only the budget reservation and the ETS commit happen on the GenServer; a load in flight is recorded with status: :loading so a duplicate load of the same id is refused while it runs.

last_used (for diagnostics/LRU) is kept in a separate :public table so callers can bump it with :ets.insert/2 without writing to the protected entries table.

Two backing modes

  • :server (default for generation/chat) — backs the model with a supervised LlamaCppEx.Server under the DynamicSupervisor, named in the Registry by id. Gets continuous batching, streaming, prefix caching, and telemetry for free.
  • :direct — the manager holds the %Model{} and callers run stateless LlamaCppEx.generate/3 / Embedding.embed/2. No standing KV-cache memory. Auto-selected when :embed is in :capabilities, because the server has no embedding path; this is a structural constraint, not a preference.

Routing

By explicit id, or :default (set at load with default: true or via set_default/1). Capability gating is enforced for embeddings (embed/3 requires an :embed-capable, :direct model).

Memory budget — placement-aware, refuse, don't evict

Loads are checked against an advisory, placement-aware budget. Footprint is estimated from GGUF file size (plus a coarse KV-cache estimate for :server mode) and distributed across RAM and GPUs from the load's :n_gpu_layers/:split_mode/:tensor_split/:main_gpu. Three budget shapes: :infinity; an integer (a single combined RAM+VRAM pool, backward-compatible); and :auto/explicit map (per-device — RAM pool plus per-GPU VRAM pools checked independently). :auto reads each GPU's free VRAM via a new backend-agnostic device_list NIF (ggml_backend_dev_*, exposed as LlamaCppEx.devices/0). Over-budget loads are refused, naming the device: {:error, {:insufficient_memory, device: :total | :ram | {:gpu, i}, required:, available:}}. There is no automatic eviction — yanking a model another caller is mid-stream on is worse than a clear refusal.

Estimation is coarse and advisory: partial offload (0 < n_gpu_layers < n_layers) is treated as fully offloaded, and compute buffers/fragmentation aren't modeled.

GC-based unload

unload/1 stops the backing server (dropping its context and model refs), deletes the ETS entry, and forces a GC. Because reclamation is by garbage collection, a caller still holding a %Model{} (e.g. from fetch_model/1) keeps the model alive until that reference is dropped. This is documented; list/0/info/1 return sanitized, ref-free views to avoid accidentally leaking handles.

Opt-in supervision

mix.exs application/0 is unchanged — there is no auto-started Application. Users add {LlamaCppEx.ModelSupervisor, opts} to their own tree (or call start_link/1 for scripts/IEx). The supervisor starts the Registry and DynamicSupervisor before the manager.

Consequences

  • Zero impact on existing single-model usage — the manager is additive and opt-in, with no new dependencies.
  • The singleton GenServer is not a bottleneck: inference bypasses it via direct ETS reads.
  • A backing-server crash is isolated by the DynamicSupervisor; the manager monitors each server and marks the entry :error on :DOWN (it does not auto-reload, to avoid crash loops without backoff).
  • Unload is eventual, not deterministic — adequate for swapping a few models, but a fixed-VRAM box needing hard, immediate reclamation would want an explicit model_free NIF (deferred; see below).
  • Model I/O is behind an injectable Backend behaviour, so lifecycle tests run without real GGUF files.

Alternatives Considered

One process per model

Rejected — models are read-only and shareable and inference runs on dirty schedulers, so a process per model buys no concurrency. It would only add supervision overhead. The :server mode still uses a process per model, but for batching, not for safe concurrent access.

Auto-started Application

Rejected — this is a library. Auto-starting a process tree would impose it on every dependent whether or not they use multi-model. Opt-in via ModelSupervisor keeps users in control.

Automatic LRU eviction on over-budget

Rejected as the default — eviction can terminate a model another caller is actively streaming from (the busy-check is advisory and racy). Refusing is safer and predictable; the caller decides what to unload.

Deterministic model_free NIF

Deferred. A NIF that nulls the llama_model* would make unload's memory reclamation immediate, but any live llama_context still references that model — freeing it out from under a context is a use-after-free that crashes the BEAM. It would need refcount gating in every accessor. GC-based reclamation plus the documented caveat is sufficient for now.