ibm-granite

/

granite-34b-code-instruct-8k-GGUF

like 2

Follow

IBM Granite 868

I am using latest llama-cpp-python and I keep getting this error

ValueError: Failed to load model from file:
I dabbled with adding n_gqa param but in vain.
Code:

llm = Llama.from_pretrained(
    repo_id="ibm-granite/granite-34b-code-instruct-GGUF",
    filename="granite-34b-code-instruct.Q4_K_M.gguf",
    verbose=False,
    n_gpu_layers=-1,
    # n_ctx=30000,
    n_gqa=4,
)