Intel
/

neural-chat-7b-v3

Text Generation

text-generation-inference

Inference Endpoints

Model card Files Files and versions Community

Haihao commited on Feb 20

Commit

a03a0fe

•

1 Parent(s): 959607f

Update README.md

Use bf16 compute type

Files changed (1) hide show

README.md +1 -1

README.md CHANGED Viewed

@@ -186,7 +186,7 @@ So, the sum of 100, 520, and 60 is 680.
 from transformers import AutoTokenizer, TextStreamer
 from intel_extension_for_transformers.transformers import AutoModelForCausalLM, WeightOnlyQuantConfig
 model_name = "Intel/neural-chat-7b-v3"
-config = WeightOnlyQuantConfig(compute_dtype="int8", weight_dtype="int4")
 prompt = "Once upon a time, there was a horse in the forest,"
 tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)

 from transformers import AutoTokenizer, TextStreamer
 from intel_extension_for_transformers.transformers import AutoModelForCausalLM, WeightOnlyQuantConfig
 model_name = "Intel/neural-chat-7b-v3"
+config = WeightOnlyQuantConfig(compute_dtype="bf16", weight_dtype="int4")
 prompt = "Once upon a time, there was a horse in the forest,"
 tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)