---
license: apache-2.0
datasets:
- tatsu-lab/alpaca
---

## 🍮 🦙 Flan-Alpaca: Instruction Tuning from Humans and Machines

Thanks to [declare-lab](https://huggingface.co/declare-lab) for the training [repository](https://github.com/declare-lab/flan-alpaca), contains code for extending the [Stanford Alpaca](https://github.com/tatsu-lab/stanford_alpaca)
synthetic instruction tuning to existing instruction-tuned models such as [Flan-T5](https://arxiv.org/abs/2210.11416).
The pretrained models and demos are available on HuggingFace 🤗 :

| Model                                                                     | Parameters | Training GPUs   |
|---------------------------------------------------------------------------|------------|-----------------|
| [Flan-Alpaca-Base](https://huggingface.co/declare-lab/flan-alpaca-base)   | 220M       | 1x A6000        |
| [Flan-Alpaca-Large](https://huggingface.co/declare-lab/flan-alpaca-large) | 770M       | 1x A6000        |
| [Flan-Alpaca-XL](https://huggingface.co/declare-lab/flan-alpaca-xl)       | 3B         | 1x A6000        |
| [Flan-Alpaca-XXL](https://huggingface.co/declare-lab/flan-alpaca-xxl)     | 11B        | 4x A6000 (FSDP) |
| [Flan-Alpaca-UL2](https://huggingface.co/0-hero/flan-alpaca-ul2)          | 20B        | 4x A100 (80G) (FSDP)  |

### Why?

[Alpaca](https://crfm.stanford.edu/2023/03/13/alpaca.html) represents an exciting new direction
to approximate the performance of large language models (LLMs) like ChatGPT cheaply and easily.
Concretely, they leverage an LLM such as GPT-3 to generate instructions as synthetic training data.
The synthetic data which covers more than 50k tasks can then be used to finetune a smaller model.
However, the original implementation is less accessible due to licensing constraints of the
underlying [LLaMA](https://ai.facebook.com/blog/large-language-model-llama-meta-ai/) model.
Furthermore, users have noted [potential noise](https://github.com/tloen/alpaca-lora/issues/65) in the synthetic
dataset. Hence, it may be better to explore a fully accessible model that is already trained on high-quality (but
less diverse) instructions such as [Flan-T5](https://arxiv.org/abs/2210.11416).

### Usage

```
from transformers import pipeline

prompt = "Write an email about an alpaca that likes flan"
model = pipeline(model="0-hero/flan-alpaca-ul2")
model(prompt, max_length=128, do_sample=True)

```

Readme forked from declare-lab/flan-alpaca-xxl