File size: 2,195 Bytes
406b1e4 b2225a1 7902bfa b2225a1 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 |
---
license: apache-2.0
---
# Sidrap-7B-v2-GPTQ
Sidrap-7B-v2-GPTQ is an 8-bit quantized model of Sidrap-7B-v2, which is one of the best open model LLM bahasa Indonesia available today. This model has been quantized using [AutoGPTQ](https://github.com/PanQiWei/AutoGPTQ) to get smaller model that allows us to run in a lower resource environment with faster inference. The quantization uses random subset of original training data to "calibrate" the weights resulting in an optimally compact model with minimall loss in accuracy.
## Usage
Here is an example code snippet for using Sidrap-7B-v2-GPTQ:
```python
from transformers import AutoTokenizer, pipeline
from auto_gptq import AutoGPTQForCausalLM
model_id = "robinsyihab/Sidrap-7B-v2-GPTQ"
tokenizer = AutoTokenizer.from_pretrained(model_id, use_fast=True)
model = AutoGPTQForCausalLM.from_quantized(model_id,
device="cuda:0",
inject_fused_mlp=True,
inject_fused_attention=True,
trust_remote_code=True)
chat = pipeline("text-generation",
model=model,
tokenizer=tokenizer,
device_map="auto")
prompt = ("<s>[INST] <<SYS>>\nAnda adalah asisten yang suka membantu, penuh hormat, dan jujur. Selalu jawab semaksimal mungkin, sambil tetap aman. Jawaban Anda tidak boleh berisi konten berbahaya, tidak etis, rasis, seksis, beracun, atau ilegal. Harap pastikan bahwa tanggapan Anda tidak memihak secara sosial dan bersifat positif.\n\
Jika sebuah pertanyaan tidak masuk akal, atau tidak koheren secara faktual, jelaskan alasannya daripada menjawab sesuatu yang tidak benar. Jika Anda tidak mengetahui jawaban atas sebuah pertanyaan, mohon jangan membagikan informasi palsu.\n"
"<</SYS>>\n\n"
"Siapa penulis kitab alfiyah? [/INST]\n"
)
sequences = chat(prompt, num_beams=2, max_length=max_size, top_k=10, num_return_sequences=1)
print(sequences[0]['generated_text'])
```
## License
Sidrap-7B-v2-GPTQ is licensed under the Apache 2.0 License.
## Author
[] Robin Syihab ([@anvie](https://x.com/anvie))
|