Speculative decoding with Qwen 32B + Qwen 1.5B

Example:

llama-server \
    -m qwen2.5-coder-32b-instruct-q4_k_m.gguf \
    -md qwen2.5-coder-1.5b-instruct-q4_k_m.gguf \
    --ctx-size 65536

GGUF

Model size

1.78B params

Architecture

qwen2

4-bit

Inference Examples

Inference API (serverless) does not yet support gguf models for this pipeline type.

Model tree for ggml-org/Qwen2.5-Coder-1.5B-32B-speculative-GGUF

Base model

Qwen/Qwen2.5-32B

Finetuned

Finetuned

Quantized

(72)

this model