|
--- |
|
language: |
|
- en |
|
datasets: |
|
- natural_instructions |
|
- the_pile |
|
- cot |
|
- Muennighoff/P3 |
|
tags: |
|
- gpt |
|
pipeline_tag: text-generation |
|
inference: |
|
parameters: |
|
temperature: 0.0 |
|
widget: |
|
- text: "Where is Zurich? Ans:" |
|
- text: "What is the highest mountain? Answer:" |
|
--- |
|
|
|
<div id="image" style="display:inline;"> |
|
<img src="https://toma.together.xyz/logo.svg" width="110"/> |
|
<h1>TOGETHER<h1/> |
|
</div> |
|
|
|
***!!! Be careful, this repo is still under construction. The content might change recently. !!!*** |
|
|
|
# Model Summary |
|
|
|
We present Together-GPT-J-6B-ProxAdam-50x, capable of following human instructions and conduct zero/few-shot inference. |
|
The model trained in a decentralized fashion with ProxAdam optimizer, requiring only 2% cross-machine communication compared to vanilla data parallel training. |
|
|
|
# Quick Start |
|
|
|
```python |
|
from transformers import pipeline |
|
|
|
pipe = pipeline(model='togethercomputer/Together-gpt-J-6B-ProxAdam-50x') |
|
|
|
pipe("Where is Zurich? Ans:") |
|
``` |
|
|
|
# Training Data |
|
|
|
We fine-tune [GPT-J-6B](https://huggingface.co/EleutherAI/gpt-j-6B) on NI, P3, COT, the pile data. |
|
- [Natural-Instructions](https://github.com/allenai/natural-instructions) |
|
- [P3](https://huggingface.co/datasets/Muennighoff/P3) |
|
- [MMLU-COT](https://github.com/jasonwei20/flan-2/blob/main/mmlu-cot.json) |
|
- [the pile](https://huggingface.co/datasets/the_pile) |
|
|
|
The pile is used to keep the general ability of GPT-J. |
|
Others are instruction-tuning datasets. |
|
|
|
# Hyperparameters |
|
|
|
We used AdamW with a learning rate of 1e-5 and global batch size of 64, and train for 5k steps. |
|
We used mix-precision training where the activation is in FP16 while the optimizer states are kept in FP32. |
|
We truncate the input sequence to 2048 tokens, and for input sequence that contains less than 2048 tokens, we concatenate multiple sequences into one long sequence to improve the data efficiency. |
|
|
|
# Infrastructure |
|
|
|
We used [the Together Research Computer](https://together.xyz/) to conduct training. |
|
Specifically, we used 4 data parallel workers, each containing 2 \* A100 80GB GPUs. |
|
Together Research Computer connects clusters at Stanford University, ETH Zurich, Open Science Grid, and University of Wisconsin-Madison. |
