PomeranIAn

Adapter Description

This adapter was created with the PEFT library and allowed the base model Falcon-7b to be fine-tuned on the Spanish instructions dataset by using the method QLoRA.

Model description

Falcon 7B

Intended uses & limitations

TBA

Training and evaluation data

TBA

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0002
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• gradient_accumulation_steps: 8
• total_train_batch_size: 64
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: constant
• lr_scheduler_warmup_ratio: 0.03
• num_epochs: 8

Training results

Training Loss	Epoch	Step	Validation Loss
1.6302	1.54	10	1.6542
1.5742	3.08	20	1.6157
1.3779	4.62	30	1.5896
1.2988	6.15	40	1.5753

How to use

import torch
from peft import PeftModel, PeftConfig
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig, AutoTokenizer, GenerationConfig

peft_model_id = "mrm8488/pomeranian"

config = PeftConfig.from_pretrained(peft_model_id)
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.float16,
)

tokenizer = AutoTokenizer.from_pretrained(peft_model_id)

model = AutoModelForCausalLM.from_pretrained(config.base_model_name_or_path,
                                             return_dict=True,
                                             quantization_config=bnb_config, 
                                             trust_remote_code=True, 
                                             device_map={"":0})


prompt_input = "A continuación, se muestra una instrucción que describe una tarea, junto con una entrada que proporciona más contexto. Escribe una respuesta que complete adecuadamente la solicitud.\n\n### Instrucción:\n{instruction}\n\n### Entrada:\n{input}\n\n### Respuesta:\n"
prompt_no_input = "A continuación, se muestra una instrucción que describe una tarea. Escribe una respuesta que complete adecuadamente la solicitud.\n\n### Instrucción:\n{instruction}\n\n### Respuesta:\n"

def create_prompt(instruction, input=None):
  if input:
    return  prompt_input.format(instruction=instruction, input=input)
  else:
    return prompt_no_input.format(instruction=instruction)

def generate(
        instruction,
        input=None,
        max_new_tokens=128,
        temperature=0.1,
        top_p=0.75,
        top_k=40,
        num_beams=4,
        **kwargs,
):
    prompt = create_prompt(instruction, input)
    inputs = tokenizer(prompt, return_tensors="pt")
    input_ids = inputs["input_ids"].to("cuda")
    attention_mask = inputs["attention_mask"].to("cuda")
    generation_config = GenerationConfig(
        temperature=temperature,
        top_p=top_p,
        top_k=top_k,
        num_beams=num_beams,
        **kwargs,
    )
    with torch.no_grad():
        generation_output = model.generate(
            input_ids=input_ids,
            attention_mask=attention_mask,
            generation_config=generation_config,
            return_dict_in_generate=True,
            output_scores=True,
            max_new_tokens=max_new_tokens
        )
    s = generation_output.sequences[0]
    output = tokenizer.decode(s)
    return output.split("### Respuesta:")[1]

instruction = "Dime algo sobre los halcones"

print("Instrucción:", instruction)
print("Respuesta:", generate(instruction))

Framework versions

• Transformers 4.30.0.dev0
• Pytorch 2.0.1+cu118
• Datasets 2.12.0
• Tokenizers 0.13.3