# `LlamaCppEx.MTP`
[🔗](https://github.com/nyo16/llama_cpp_ex/blob/main/lib/llama_cpp_ex/mtp.ex#L1)

Multi-Token Prediction (MTP) speculative decoding.

Drives a target/draft speculative loop where the draft model is an MTP head —
either embedded in the target GGUF, or shipped beside it as a sidecar file. On
Qwen 3.6 with `n_draft: 3` this typically yields ~2x token-generation
throughput at ~75% draft acceptance.

## Usage

    :ok = LlamaCppEx.init()
    {:ok, model} = LlamaCppEx.load_model("Qwen3.6-35B-A3B-MTP-Q4_K_M.gguf",
                                          n_gpu_layers: 999, load_mtp: true)

    {:ok, mtp} = LlamaCppEx.MTP.init(model, n_draft: 3, n_ctx: 8192)

    mtp
    |> LlamaCppEx.MTP.stream("Write a haiku about the sea:", max_tokens: 200)
    |> Stream.each(&IO.write/1)
    |> Stream.run()

    stats = LlamaCppEx.MTP.stats(mtp)
    IO.puts("acceptance: #{Float.round(stats.acceptance_rate * 100, 1)}%")

## Where the MTP head lives

The head is a set of `*.nextn_predict_layers` and comes in two shapes, and
either way the file carrying it must be loaded with `load_mtp: true`. Upstream
defaults that flag to `false` so non-speculative callers do not pay for the
head's tensors, and the layers cannot be attached afterwards, so `init/2`
refuses a model loaded without it.

**In the target GGUF** (e.g. `ggml-org/Qwen3.6-35B-A3B-MTP-GGUF`) — pass just
the model, as above.

**In a sidecar GGUF** — pass it as `:draft_model`. This is how Qwen 3.8 ships:
`Qwen3.8-27B-Q4_K_M.gguf` carries no head at all (`n_layer_nextn == 0`) and
`mtp-Qwen3.8-27B-Q4_0.gguf` carries nothing else. It is the binding's
equivalent of upstream's `-hf <target> -hfd <draft> --spec-type draft-mtp`.

    {:ok, target} = LlamaCppEx.load_model("Qwen3.8-27B-Q4_K_M.gguf",
                                          n_gpu_layers: 999, load_mtp: true)
    {:ok, head}   = LlamaCppEx.load_model("mtp-Qwen3.8-27B-Q4_0.gguf",
                                          n_gpu_layers: 999, load_mtp: true)

    {:ok, mtp} = LlamaCppEx.MTP.init(target, draft_model: head, n_draft: 1)

> #### Speculation is not always a win on hybrid models {: .warning}
>
> A model that mixes recurrent (SSM) layers with attention ones — Qwen 3.8 is
> 48 SSM layers to 16 attention layers — cannot roll back part of a sequence
> natively, so every speculative iteration snapshots and restores the whole
> recurrent state. That state is over 100 MiB at Qwen 3.8's sizes, and the
> cost lands in `stats/1`'s `timing_us.ckpt`. Measured on an M1 Max (Metal,
> Q4_K_M target + Q4_0 head), MTP was a net *slowdown* at every draft length:
> 0.89x at `n_draft: 1` (75% acceptance) falling to 0.56x at `n_draft: 5`
> (30%). Check `timing_us.ckpt` against `timing_us.total` before assuming
> speculation is helping, and prefer small `n_draft` when it is not.

Upstream currently requires `n_parallel = 1` for MTP. This module reflects
that — a single MTP session decodes one sequence at a time. Reuse the same
`%MTP{}` value across calls to `stream/3` / `generate/3` to avoid rebuilding
the contexts; KV caches are cleared on each call.

> #### Do not reuse a session straight after abandoning a stream {: .warning}
>
> Cancellation is asynchronous and unacknowledged: abandoning a `stream/3`
> early (`Enum.take/2`, `break`, an exception) sets a flag that the draft loop
> notices and then exits without reporting that it has done so. A session's two
> contexts are long-lived and shared by every call on it, so starting the next
> `generate/3` or `stream/3` immediately can put a second writer on a KV cache
> the cancelled loop has not finished with — which aborts the VM rather than
> returning an error. Let an abandoned stream run to a terminal event, or build
> a fresh session with `init/2`, before using the session again.

MTP is the only speculative type this binding exposes. Upstream llama.cpp
also implements EAGLE-3, DFlash (block-diffusion drafting), n-gram
self-speculation and combinations of them behind the same
`common_speculative` API; the NIF pins the MTP type, so `--spec-default`-style
stacking of n-gram speculation on top of MTP is not reachable from here. See
the "Speculative decoding" section of the README for the current status of
DFlash on Apple Silicon.

# `t`

```elixir
@type t() :: %LlamaCppEx.MTP{
  main_ctx: LlamaCppEx.Context.t(),
  mtp_ctx: LlamaCppEx.Context.t(),
  n_draft: pos_integer(),
  spec_ref: reference()
}
```

# `generate`

```elixir
@spec generate(t(), String.t(), keyword()) :: {:ok, String.t()} | {:error, term()}
```

Synchronously generates text. Equivalent to running `stream/3` and joining
the pieces into a single binary.

Accepts the same options as `stream/3`.

# `init`

```elixir
@spec init(
  LlamaCppEx.Model.t(),
  keyword()
) :: {:ok, t()} | {:error, term()}
```

Initializes an MTP speculative session: builds the target context, the MTP
draft context (`ctx_type: :mtp`), and the underlying `common_speculative`
state.

## Options

  * `:draft_model` - A separate `LlamaCppEx.Model` holding the MTP head, for
    checkpoints that ship it as a sidecar GGUF rather than inside the target
    file (Qwen 3.8 is the current example: `Qwen3.8-27B-Q4_K_M.gguf` plus
    `mtp-Qwen3.8-27B-Q4_0.gguf`). It must be loaded with `load_mtp: true`.
    Defaults to `nil`, meaning the head is expected inside the target model
    and the draft context is built against it.
  * `:n_draft` - Max draft tokens generated per iteration. Defaults to `3`.
    Larger values mean fewer model forward passes but lower per-iteration
    acceptance; 2–4 is the sweet spot in practice.
  * `:n_ctx` - Context size for both contexts. Defaults to `2048`.
  * Any `LlamaCppEx.Context` option (e.g. `:n_threads`, `:flash_attn`,
    `:type_k`/`:type_v`, `:offload_kqv`). The same options are applied to
    both the target and draft contexts.

Returns `{:ok, %MTP{}}` or `{:error, reason}`.

# `print_stats`

```elixir
@spec print_stats(t()) :: :ok
```

Writes upstream's own speculative stats summary to stdout (via llama.cpp
logging). Useful when cross-checking acceptance rates against the upstream
llama-server benchmark output.

# `stats`

```elixir
@spec stats(t()) :: map()
```

Returns the current MTP statistics snapshot (lock-free read of atomic
counters). Safe to call at any time — including from another process while
a stream is in flight.

Returns a map with keys:

  * `:iters` - speculative loop iterations completed
  * `:drafts_generated` - draft tokens proposed by the MTP head
  * `:drafts_accepted` - draft tokens accepted by the target model
  * `:acceptance_rate` - `drafts_accepted / drafts_generated` (0.0–1.0)
  * `:tokens_emitted` - tokens streamed back to the caller
  * `:tokens_per_sec` - throughput over the active generation window
  * `:timing_us` - `%{draft: μs, verify: μs, sample: μs, ckpt: μs, other: μs,
    total: μs}`. `:ckpt` is the recurrent-state save/restore that only hybrid
    models pay and is zero elsewhere; on Qwen 3.8 it is large enough to decide
    whether speculation helps at all. `:other` is whatever falls outside the
    named buckets, dominated on Metal by GPU-sync waits from the previous
    iteration's async verify decode.
  * `:n_draft` - max draft length configured at init

Counters are cumulative across all `stream/3` / `generate/3` calls on this
MTP value.

# `stream`

```elixir
@spec stream(t(), String.t(), keyword()) :: Enumerable.t()
```

Returns a lazy stream of generated text pieces.

## Options

  * `:max_tokens` - Maximum tokens to generate (default `256`).
  * `:emit_stats_every` - When > 0, also emits `{:stats, snapshot_map}`
    events every Nth token via the underlying message stream. Note: these
    events are filtered out of this `String.t()` stream — to consume them
    use `stream_events/3` instead. Default `0` (off).
  * `:timeout` - Receive timeout in milliseconds (default `60_000`).
  * Any sampler option from `LlamaCppEx.Sampler.create/2` (`:temp`, `:top_k`,
    `:top_p`, `:min_p`, `:seed`, `:penalty_*`, `:grammar`, etc.).

Each emitted element is the text piece for one accepted token. The stream
ends on end-of-generation, max-tokens, or error.

# `stream_events`

```elixir
@spec stream_events(t(), String.t(), keyword()) :: Enumerable.t()
```

Like `stream/3`, but yields the raw event tuples emitted by the NIF:

  * `{:token, token_id, text_piece}` - one accepted token
  * `{:stats, snapshot_map}` - periodic stats (only when `:emit_stats_every > 0`)
  * `{:done, final_stats_map}` - generation completed normally
  * `{:eog, nil}` - model emitted an end-of-generation token

The stream halts after `:done` / `:eog` / `:error`. The final stats map is
available via `stats/1` on the MTP struct even after the stream ends.

---

*Consult [api-reference.md](api-reference.md) for complete listing*