File size: 3,250 Bytes
3710c26 936181c 3710c26 936181c 30bb517 936181c cbf40e1 3f5cf16 2adf457 936181c 2adf457 936181c dd31142 936181c 4935bbe 936181c b3087e8 936181c 1ed261d 936181c 6f37e36 936181c 4935bbe 936181c 4935bbe 936181c 4935bbe 936181c 4935bbe 936181c cd36331 936181c |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 |
---
datasets:
- tatsu-lab/alpaca
language:
- en
pipeline_tag: text2text-generation
library_name: transformers
license: other
---
# Model Details
- **Model name:** Flan-UL2-Alpaca-LoRA
- **Model type:** - Text2Text Generation
- **Parent Model:** [google/flan-ul2](https://huggingface.co/google/flan-ul2)
- **Training dataset:** [Alpaca](https://huggingface.co/datasets/tatsu-lab/alpaca)
- **Language:** English
- **Framework:** PyTorch
- **Model version:** 1.0
We take the instruction-tuned Flan models (trained on Academic datasets) and perform style transfer using the Alpaca dataset.
We released the code for LORA fine-tuning Seq2Seq models along with code walkthrough medium article here:
- https://github.com/vmware-labs/research-and-development-artificial-intelligence-lab/tree/main/instruction-tuning/peft-seq2seq
- https://medium.com/vmware-data-ml-blog/lora-finetunning-of-ul-2-and-t5-models-35a08863593d
We fine-tuned the `google/flan-ul2` model on the Alpaca dataset using [PEFT-LORA](https://huggingface.co/docs/diffusers/main/en/training/lora).
# License
- Parent model ([google/flan-ul2](https://huggingface.co/google/flan-ul2)): Apache 2.0
- Dataset ([Alpaca](https://huggingface.co/datasets/tatsu-lab/alpaca)) : cc-by-4.0
- text-davinci-003 (Used to generate Alpaca): [OpenAI Terms of Use](https://openai.com/policies/terms-of-use)
# How to Use
```
import torch
from transformers import pipeline
# Chose the model inference precision
dtype = torch.float16 # options are torch.float16, torch.bfloat16, torch.float32
model = pipeline(
model = "VMware/flan-ul2-alpaca-lora",
device_map = 'auto',
torch_dtype=dtype
)
prompt = "YOUR PROMPT HERE"
output = model(prompt, max_length=2048, do_sample=True)
```
Using Alpaca prompt template might generate better outputs for certain prompts as the model was trained using the bellow template.
```
# Chose the model inference precision
import torch
from transformers import pipeline
dtype = torch.float16 # options are torch.float16, torch.bfloat16, torch.float32
model = pipeline(model="VMware/flan-ul2-alpaca-lora",device_map = 'auto',torch_dtype=dtype )
prompt_template = """
Below is an instruction that describes a task. Write a response that appropriately completes the request.
### Instruction:
{instruction}
### Response:"""
prompt = "YOUR INSTRUCTION HERE"
output = model(prompt_template.format(instruction=prompt), max_length=2048)
```
# Training Details
The model was trained on 3xV100 GPUs using PEFT-LORA and Deepspeed
* Hyperparameters:
* learning_rate = 3e-4
* per_device_batch_size = 2
* gradient_accumulation_steps = 21
* epochs = 3
# Limitations and Bias
The model is based on a large and diverse dataset, but it may still have limitations and biases in certain areas. Some limitations include:
- Language: The model is designed to work with English text only and may not perform as well in other languages.
In addition, the model may have some bias in terms of the data it was trained on. The dataset includes questions from a variety of sources, but it may not be representative of all populations or perspectives. As a result, the model may perform better or worse for certain types of questions or on certain types of texts.
|