---
library_name: peft
---

# WIP

## 1. 사용절차

* Install model and PEFT parameters

```
import torch
from peft import PeftModel, PeftConfig
from transformers import AutoTokenizer, AutoModelForCausalLM, GPTQConfig

model_id = "TheBloke/WizardLM-13B-V1.2-GPTQ"

config = PeftConfig.from_pretrained("a2ran/GPTeacher_ko_llama2_13B")
tokenizer = AutoTokenizer.from_pretrained(model_id, use_fast=True)
quantization_config_loading = GPTQConfig(bits=4, disable_exllama=True)

model = AutoModelForCausalLM.from_pretrained(model_id, quantization_config=quantization_config_loading,
                                             torch_dtype=torch.float16, device_map="auto")
model = PeftModel.from_pretrained(model, "a2ran/GPTeacher_ko_llama2_13B")
```

* How to Generate Tokens

```
from transformers import TextStreamer

streamer = TextStreamer(tokenizer)

# your input sentence가 들어갈 곳
input = """
### input @ 미국의 행정시스템에 대해 설명해줘.\n\n### response @"""

output = tokenizer.decode(model.cuda().generate(
    **tokenizer(
        input,
        return_tensors='pt',
    ).to(0),
    max_new_tokens = 2048,
    temperature = 1.2,
    top_p = 0.7,
    early_stopping = True,
    eos_token_id = 2,
    do_sample = True,
    repetition_penalty = 1.1,
    streamer = streamer
)[0]).replace(input+" ", "")
```

## 2. Training procedure


The following `bitsandbytes` quantization config was used during training:
- quant_method: gptq
- bits: 4
- tokenizer: None
- dataset: None
- group_size: 128
- damp_percent: 0.1
- desc_act: False
- sym: True
- true_sequential: True
- use_cuda_fp16: False
- model_seqlen: None
- block_name_to_quantize: None
- module_name_preceding_first_block: None
- batch_size: 1
- pad_token_id: None
- disable_exllama: True
- max_input_length: None
### Framework versions


- PEFT 0.6.0.dev0