NOTE: You will need a recent build of llama.cpp to run these quants (i.e. at least commit 494c870).

The importance matrix was trained for ~50K tokens (105 batches of 512 tokens) using a general purpose imatrix calibration dataset.
The imatrix is being used on the K-quants as well.

Layers	Context	Template
60	32768	<\|startoftext\|>[INST] <<SYS>> {instructions} <</SYS>> {prompt} [INST]

GGUF

Model size

34.4B params

Architecture

llama

4-bit

5-bit

6-bit

Inference Examples

Inference API (serverless) does not yet support gguf models for this pipeline type.

Model tree for dranger003/UNA-SimpleSmaug-34b-v1beta-iMat.GGUF

Base model

Finetuned

Finetuned

Quantized

(3)

this model