TheBloke

/

LMCocktail-phi-2-v1-GGUF

like 4

Thought I'd give it a try. Didn't expect much. But I'm finding that ctransformers is not able to load the 8-bit GGUF of this model. I don't recall what is the family for phi. LLama? It's been a while since I last looked at it.

While in google colab:

import time
from ctransformers import AutoModelForCausalLM

load_tm = time.time()

llm = AutoModelForCausalLM.from_pretrained(

    "TheBloke/LMCocktail-phi-2-v1-GGUF",
    model_file="lmcocktail-phi-2-v1.Q8_0.gguf",

    model_type="llama", # ?
    )

load_rt_sec = time.time() - load_tm
if load_rt_sec > 60:
  print(f"\nDownload and/or Loading runtime (min): {load_rt_sec / 60}")
else:
  print(f"\nDownload and/or Loading runtime (sec): {load_rt_sec}")

error:
```

RuntimeError Traceback (most recent call last)
in <cell line: 3>()
1 load_tm = time.time()
2
----> 3 llm = AutoModelForCausalLM.from_pretrained(
4
5 "TheBloke/LMCocktail-phi-2-v1-GGUF",

1 frames
/usr/local/lib/python3.10/dist-packages/ctransformers/llm.py in init(self, model_path, model_type, config, lib)
251 )
252 if self._llm is None:
--> 253 raise RuntimeError(
254 f"Failed to create LLM '{model_type}' from '{model_path}'."
255 )

RuntimeError: Failed to create LLM 'llama' from '/root/.cache/huggingface/hub/models--TheBloke--LMCocktail-phi-2-v1-GGUF/blobs/05d6680da2235732940781679c7925e140fad0ff087cbf10961942a644dca7b6'.

Maybe a conversation for marella on github.