# `LlamaCppEx.Decision` [🔗](https://github.com/nyo16/llama_cpp_ex/blob/main/lib/llama_cpp_ex/decision.ex#L1) Typed decisions with a decision model: llama.cpp's `/v1/systemone` API, in-process. A decision model answers typed questions about a `state` in one forward pass per prompt; no token is generated. Each answer is a probability distribution, so it comes with a confidence instead of free text. This is llama-server's [TypeSafe-compatible System One API](https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md#post-v1systemone-typesafe-compatible-system-one-api), ported into the NIF: same request shape, same prompts, same answers. ## Models The model file says what kind of decision model it is (`".decision.type"`); `model_type/1` reads it. Supported types: * `:openjev`, `:lev`, `:nimble`, `:pplx_decider`, `:vllm_d3`, `:lfm2_d1`, `:lfm3_d1`, `:lfm3_d1_omni`, `:granite_switch`, `:h2o_lightning` - read the logits of one label token per option (`:nimble` lists every question of the request in each prompt, `:lev` asks each choice twice with the options reversed, `:pplx_decider`, `:vllm_d3` and `:granite_switch` use one- or two-letter label codes like `:lev`, `:lfm2_d1` picks label codes per question and accepts a `nil` state). `:vllm_d3` is non-causal, so each prompt is evaluated in one batch; it accepts a `nil` state and a question without `:instructions` (its prompt has a default question). `:lfm3_d1` and `:lfm3_d1_omni` pick label codes like `:lfm2_d1` (`A`..`Z`, then `AA`, `AB`.. past 26 options), accept a `nil` state, have no memory, so every prompt is evaluated in one batch, and return the median level as a `:score` and the probability of the answer as its confidence. `:granite_switch` is IBM's Granite Switch, whose decision adapter is built into the model; the same model also generates text through the rest of the API. `:h2o_lightning` reads the labels its model file lists, after an answer prefill: `:instructions` and `:type` may be left out (`:choice` by default), a `:score` takes 1 to 255 levels, `:noul` descriptions are also read under `"yes"` and `"no"`, and a state longer than 32000 tokens keeps its first and last 16000. * `:kev` - scores each option by a dot product of hidden states. * `:laya` - a ModernBERT encoder; one score per `[MASK]` marker. * `:lfm2_d1_omni` - like `:laya` with another prompt layout: a non-causal LFM2 trunk with a decision head, one score per `<|mask|>` marker. It has no memory, so every prompt is evaluated in one batch. Accepts a `nil` state, and also reads `:noul` descriptions under `"yes"` and `"no"`. * `:clef` - reads every question in one prompt and decides them jointly. * `:vllm_decision2` - clef's joint head, one question per prompt: a Qwen3 trunk scores each option span against the question span. Strict input: the state, instructions and descriptions must be text, a map or a list, a choice needs at least 2 options, and `:noul` criteria may only describe `"true"` and `"false"`. Decision GGUFs are published in ggml-org's "Decision models" collection on Hugging Face, for example `ggml-org/Laya-GGUF`, `ggml-org/Clef-Flash-GGUF` and `ggml-org/OpenJev-GGUF`. ## Questions `questions` maps a question id to a question. Each question has: * `:type` - `:choice`, `:score` or `:noul` (strings work too). * `:instructions` - the question. A string, or any JSON-encodable term, which the model is given as JSON text. * `:criteria` - depends on `:type`: * `:choice` - the options, mapping each option key to its description (`nil` for none). The number of options is capped by the model: 52 for openjev, 255 for laya, clef, pplx-decider, vllm-decision2, vllm-d3, lfm2-d1, lfm2-d1-omni, lfm3-d1, lfm3-d1-omni, granite-switch and h2o-lightning. * `:score` - a list of 2 to 10 level descriptions, lowest first (1 to 255 for h2o-lightning). * `:noul` - optional; `%{"true" => ..., "false" => ...}` descriptions. The questions of a request are answered independently, except with clef, which decides them jointly. Order matters to the model: the options of a choice are shown in the order given, and with clef and nimble so are the questions. A map is taken in Elixir's map order (sorted keys for up to 32 entries); pass a keyword list or a list of `{key, value}` tuples to fix the order yourself. Ids and option keys come back exactly as given, atoms included. ## State A string, or any JSON-encodable term (a map, a list), which the model is given as JSON text. A list of chat messages, or a map with a `"messages"` list, works the same way. ## Answers * `:choice` - `%{type: :choice, choice: key, probabilities: %{key => p}, confidence: c}` * `:score` - `%{type: :score, score: s, legend: %{0 => desc, ...}, probabilities: %{0 => p, ...}, confidence: c}`, where `score` is the probability-weighted level index and can fall between two levels (lfm3-d1 and lfm3-d1-omni: the median level, a whole number). * `:noul` - `%{type: :noul, noul: p}`, the probability that the answer is true. `confidence` runs from 0 (every option equally likely) to 1; vllm-decision2 and vllm-d3 compute it from the normalized entropy of the probabilities, and lfm3-d1 and lfm3-d1-omni return the probability of the answer. Probabilities are scaled with the temperatures (and, for scores, the per-level biases) stored in the model file; they are not guaranteed to be calibrated for your data. vllm-decision2 and vllm-d3 return a `legend` entry that is not a string as its JSON text, with sorted keys. ## Example {:ok, model} = LlamaCppEx.load_model("Laya-Q8_0.gguf", n_gpu_layers: -1) {:ok, decision} = LlamaCppEx.Decision.new(model) {:ok, %{answers: answers}} = LlamaCppEx.Decision.decide( decision, "Customer message: I was charged twice for my order last week.", route: [ type: :choice, instructions: "Which team should handle this?", criteria: [billing: "payments and refunds", shipping: nil, technical: nil] ], angry: [type: :noul, instructions: "Is the customer angry?"], urgency: [ type: :score, instructions: "How urgent is this?", criteria: ["can wait", "this week", "today", "right now"] ] ) answers.route.choice #=> :billing (0.992 with Laya-Q8_0) ## Limits * Text only. Upstream also takes images (openjev, clef, pplx-decider, vllm-d3, lfm2-d1, lfm2-d1-omni, lfm3-d1, lfm3-d1-omni), videos (lfm3-d1, lfm3-d1-omni) and audio (lfm2-d1-omni, lfm3-d1-omni spoken questions) through libmtmd, which this build does not link; a request with any of them is rejected. For the same reason upstream's `openbmb-minicpm-ocr-jev` type, which scores an OCR text against two images and nothing else, is not supported (`model_type/1` returns `:unknown`). * laya, clef, vllm-d3, lfm2-d1-omni, lfm3-d1 and lfm3-d1-omni evaluate the whole prompt in one batch, so it has to fit in `:n_batch` (2048 tokens by default, see `new/2`); vllm-decision2 needs the question, its options and the end of the prompt to fit in one. * A `%Decision{}` drives one context and is not safe to use from two processes at once, like `LlamaCppEx.Context`. Give each process its own, or serialize calls through one process. # `answer` ```elixir @type answer() :: %{ type: :choice, choice: id(), probabilities: %{optional(id()) => float()}, confidence: float() } | %{ type: :score, score: float(), legend: %{optional(non_neg_integer()) => term()}, probabilities: %{optional(non_neg_integer()) => float()}, confidence: float() } | %{type: :noul, noul: float()} ``` # `decision_type` ```elixir @type decision_type() :: :openjev | :lev | :kev | :nimble | :laya | :clef | :pplx_decider | :lfm2_d1 | :lfm2_d1_omni | :lfm3_d1 | :lfm3_d1_omni | :vllm_decision2 | :vllm_d3 | :granite_switch | :h2o_lightning ``` # `id` ```elixir @type id() :: atom() | String.t() ``` # `question` ```elixir @type question() :: %{optional(atom() | String.t()) => term()} | [{atom() | String.t(), term()}] ``` # `questions` ```elixir @type questions() :: %{optional(id()) => question()} | [{id(), question()}] ``` # `result` ```elixir @type result() :: %{ answers: %{optional(id()) => answer()}, usage: %{input_tokens: non_neg_integer(), output_tokens: 0} } ``` # `t` ```elixir @type t() :: %LlamaCppEx.Decision{ context: LlamaCppEx.Context.t(), ref: reference(), type: decision_type() } ``` # `decide` ```elixir @spec decide(t(), term(), questions()) :: {:ok, result()} | {:error, String.t()} ``` Answers `questions` about `state`. See the module doc for the shape of `state`, `questions` and the answers. Returns `{:error, reason}` for a request the model rejects (a missing `:instructions`, an unknown `:type`, too many options, a prompt that does not fit the context) with the same messages as llama-server. Raises `ArgumentError` if two question ids, or two option keys of one question, turn into the same string. # `model_type` ```elixir @spec model_type(LlamaCppEx.Model.t()) :: decision_type() | :unknown | nil ``` The decision type of a model. Returns `nil` for a model that is not a decision model and `:unknown` for a decision type this build does not support. # `new` ```elixir @spec new( LlamaCppEx.Model.t(), keyword() ) :: {:ok, t()} | {:error, String.t()} ``` Creates a decision engine on a new context for `model`. The context is shaped for the model's decision type: laya, kev, clef, lfm2-d1-omni and vllm-decision2 read the embeddings output, so their context has embeddings on and no pooling; the types that share a prompt prefix across questions (openjev, granite-switch, lev, kev, nimble, pplx-decider, vllm-decision2, lfm2-d1, h2o-lightning) get a second sequence to evaluate that prefix once per request. ## Options * `:n_ctx` - Context size. Defaults to `4096`. A prompt holds the state, one question and its options (all questions for clef and nimble), so a long state needs a larger context. * `:n_batch` - Max tokens per batch. For laya, kev, clef, lfm2-d1-omni, vllm-decision2, vllm-d3, lfm3-d1 and lfm3-d1-omni it is also the micro-batch size and caps the prompt that can be evaluated at once; defaults to `min(n_ctx, 2048)` for them and to `LlamaCppEx.Context`'s default otherwise. Any `LlamaCppEx.Context.tuning_option_keys/0` option (`:n_threads`, `:flash_attn`, ...) is passed through to `LlamaCppEx.Context.create/2`. --- *Consult [api-reference.md](api-reference.md) for complete listing*