ruliad
/

deepthought-8b-llama-v0.01-alpha

Text Generation

Model card Files Files and versions Community

alpindale commited on 8 days ago

Commit

de747da

•

1 Parent(s): efee38e

default->eager

Files changed (1) hide show

deepthought_inference.py +1 -1

deepthought_inference.py CHANGED Viewed

@@ -38,7 +38,7 @@ class DeepthoughtModel:
             self.model_name,
             torch_dtype=torch.bfloat16,
             device_map="auto",
-            attn_implementation=("flash_attention_2" if flash_attn_exists else "default"),
             use_cache=True,
             trust_remote_code=True,
         )

             self.model_name,
             torch_dtype=torch.bfloat16,
             device_map="auto",
+            attn_implementation=("flash_attention_2" if flash_attn_exists else "eager"),
             use_cache=True,
             trust_remote_code=True,
         )