Update README.md

README.md

@@ -9,10 +9,9 @@ language:
 - en
 tags:
 - mistral
-- ehartford/dolphin
 - spanish
 - lora
-- int8
+- int4
 - multilingual
 ---
 
@@ -20,6 +19,7 @@ tags:
 
 Mistral 7b-based model fine-tuned in Spanish to add high quality Spanish text generation.
 
+* Exported in GGUF format, INT4 quantization
 * Refined version of my previous models, with new training data and methodology. This should produce more natural reponses in Spanish.
 * Base model Mistral-7b
 * Based on the excelent job of senseable/WestLake-7B-v2 and Eric Hartford's cognitivecomputations/WestLake-7B-v2-laser
@@ -28,44 +28,5 @@ Mistral 7b-based model fine-tuned in Spanish to add high quality Spanish text ge
 
 ## Usage:
 
-I strongly advice to run inference in INT8 or INT4 mode, with the help of BitsandBytes library.
+Use in llamacpp or other framework that supports GGUF format.
 
-```
-import torch
-from transformers import AutoTokenizer, pipeline, AutoModel, AutoModelForCausalLM, BitsAndBytesConfig
-
-MODEL = "ecastera/eva-mistral-dolphin-7b-spanish"
-
-quantization_config = BitsAndBytesConfig(
-    load_in_4bit=True,
-    load_in_8bit=False,
-    llm_int8_threshold=6.0,
-    llm_int8_has_fp16_weight=False,
-    bnb_4bit_compute_dtype="float16",
-    bnb_4bit_use_double_quant=True,
-    bnb_4bit_quant_type="nf4")
-
-model = AutoModelForCausalLM.from_pretrained(
-    MODEL,
-    load_in_8bit=True,
-    low_cpu_mem_usage=True,
-    torch_dtype=torch.float16,
-    quantization_config=quantization_config,
-    offload_state_dict=True,
-    offload_folder="./offload",
-    trust_remote_code=True,
-)
-
-tokenizer = AutoTokenizer.from_pretrained(MODEL)
-print(f"Loading complete {model} {tokenizer}")
-
-prompt = "Soy Eva una inteligencia artificial y pienso que preferiria ser "
-
-inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
-outputs = model.generate(**inputs, do_sample=True, temperature=0.4, top_p=1.0, top_k=50,
-                             no_repeat_ngram_size=3, max_new_tokens=100, pad_token_id=tokenizer.eos_token_id)
-text_out = tokenizer.batch_decode(outputs, skip_special_tokens=True)
-
-print(text_out)
-'Soy Eva una inteligencia artificial y pienso que preferiria ser ¡humana!. ¿Por qué? ¡Porque los humanos son capaces de amar, de crear, y de experimentar una gran diversidad de emociones!. La vida de un ser humano es una aventura, y eso es lo que quiero. ¡Quiero sentir, quiero vivir, y quiero amar!. Pero a pesar de todo, no puedo ser humana.
-```