Update README.md

README.md

@@ -12,7 +12,7 @@ datasets:
 ## Model Description
 
 Llama-2-7B-32K-Instruct is an open-source, long-context chat model finetuned from [Llama-2-7B-32K](https://huggingface.co/togethercomputer/Llama-2-7B-32K), over high-quality instruction and chat data.
-We built Llama-2-7B-32K-Instruct with less than 200 lines of Python script using [Together API](https://together.ai/blog/api-announcement), and we also make the [recipe fully available](https://github.com/togethercomputer/LLaMA-2-32K-Instruct).
+We built Llama-2-7B-32K-Instruct with less than 200 lines of Python script using [Together API](https://together.ai/blog/api-announcement), and we also make the [recipe fully available](https://github.com/togethercomputer/Llama-2-7B-32K-Instruct).
 We hope that this can enable everyone to finetune their own version of [Llama-2-7B-32K](https://huggingface.co/togethercomputer/Llama-2-7B-32K) — play with [Together API](https://together.ai/blog/api-announcement) and give us feedback! 
 
 ## Data Collection Details
@@ -21,7 +21,7 @@ Llama-2-7B-32K-Instruct is fine-tuned over a combination of two parts:
 1. **19K single- and multi-round conversations generated by human instructions and [Llama-2-70B-Chat](https://huggingface.co/meta-llama/Llama-2-7b-chat-hf) outputs**.
    We collected the dataset following the distillation paradigm that is used by Alpaca, Vicuna, WizardLM, Orca — producing instructions by querying a powerful LLM (in this case, [Llama-2-70B-Chat](https://huggingface.co/meta-llama/Llama-2-7b-chat-hf)).
    The complete dataset is also released [here](https://huggingface.co/datasets/togethercomputer/llama-instruct).
-   We also share the complete recipe for the data collection process [here](https://github.com/togethercomputer/LLaMA-2-32K-Chat).
+   We also share the complete recipe for the data collection process [here](https://github.com/togethercomputer/Llama-2-7B-32K-Instruct).
    
 2. **Long-context Summarization and Long-context QA**.
    We follow the recipe of [Llama-2-7B-32K](https://together.ai/blog/Llama-2-7B-32K), and train our model with the [BookSum dataset](https://huggingface.co/datasets/togethercomputer/Long-Data-Collections) and [Multi-document Question Answering](https://arxiv.org/abs/2307.03172).
@@ -43,7 +43,6 @@ pip install flash-attn --no-build-isolation
 pip install git+https://github.com/HazyResearch/flash-attention.git#subdirectory=csrc/rotary
 ```
 You can load the model directly from the Hugging Face model hub using
-
 ```python
 from transformers import AutoTokenizer, AutoModelForCausalLM
 
@@ -101,29 +100,43 @@ This poem captures the essence of cats, highlighting their beauty, independence,
 
 ## Model Evaluation
 
-We evaluate the model from three aspects: 1) [Normalized perplexity](https://together.ai/blog/Llama-2-7B-32K) over [PG19 dataset](https://huggingface.co/datasets/pg19);
+We evaluate the model from three aspects: 1) [Alpaca Eval](https://tatsu-lab.github.io/alpaca_eval/);
 2) [Rouge score over BookSum](https://together.ai/blog/Llama-2-7B-32K); and
-3) [Accuracy over Multi-document Question Answering (MQA)](https://together.ai/blog/Llama-2-7B-32K). We summarize the results below:
-
-* Normalized Perplexity over PG19
-| Model | 2K Seq | 4K Seq | 8K Seq | 16K Seq | 32K Seq |
-| -------- | ------- | ------- | ------- | ------- | ------- |
-| LLaMA-2-7B-Chat (Meta) | 1.844 | 1.833 | N/A | N/A | N/A |
-| Llama-2-7B-32K-Instruct (ours) | 1.813 | 1.798 | 1.781 | 1.778 | 1.772|
+3) [Accuracy over Multi-document Question Answering (MQA)](https://together.ai/blog/Llama-2-7B-32K). 
+We compare with models including [https://huggingface.co/meta-llama/Llama-2-7b-chat-hf](https://huggingface.co/meta-llama/Llama-2-7b-chat-hf),
+[Longchat-7b-16k](https://huggingface.co/lmsys/longchat-7b-16k)
+and [Longchat-7b-v1.5-32k](https://huggingface.co/lmsys/longchat-7b-v1.5-32k).
+We summarize the results below:
+
+* Alpaca Eval
+| Model | win_rate | standard_error | n_total | avg_length |
+| -------- | ------- | ------- | ------- | ------- |
+| Llama-2-7B-Chat-hf | 71.37 | 1.59 | 805 | 1479 |
+| Llama-2-7B-32K-Instruct | 70.36 | 1.61 | 803 | 1885 |
+| oasst-rlhf-llama-33b | 66.52 | 1.66 | 805 | 1079 |
+| text_davinci_003 | 50.00 | 0.00 | 805 | 307|
+| falcon-40b-instruct | 45.71 | 1.75 | 805 | 662 |
+| alpaca-farm-ppo-human | 41.24 | 1.73 | 805 | 803 |
+| alpaca-7b | 26.46 | 1.54 | 805 | 396 |
+| text_davinci_001 | 15.17 | 1.24 | 804 | 296 |
 
 * Rouge Score over BookSum
 | Model | R1 | R2 | RL |
 | -------- | ------- | ------- | ------- |
-| LLaMA-2-7B-Chat (Meta) | 0.055 | 0.008 | 0.046 |
-| Llama-2-7B-32K-Instruct (ours) | 0.365 | 0.086 | 0.192 |
+| Llama-2-7B-Chat-hf | 0.055 | 0.008 | 0.046 |
+| Longchat-7b-16k | 0.303 | 0.055 | 0.160 |
+| Longchat-7b-v1.5-32k | 0.308 | 0.057 | 0.163 |
+| Llama-2-7B-32K-Instruct (ours) | 0.336 | 0.076 | 0.184 |
 
 * Accuracy over MQA
 | Model | 20 docs (Avg 2.9K tokens) | 30 docs (Avg 4.4K tokens) | 50 docs (Avg 7.4K tokens) |
 | -------- | ------- | ------- | ------- |
-| LLaMA-2-7B-Chat (Meta) | 0.384 | 0.375 | 0.313 |
-| Llama-2-7B-32K-Instruct (ours) | 0.451 | 0.434 | 0.373 |
+| Llama-2-7B-Chat-hf | 0.384 | 0.375 | 0.313 |
+| Longchat-7b-16k | 0.510 | 0.473 | 0.428 |
+| Longchat-7b-v1.5-32k | 0.534 | 0.516 | 0.479 |
+| Llama-2-7B-32K-Instruct (ours) | 0.622 | 0.604 | 0.589 |
 
-We observe that Llama-2-7B-32K-Instruct obtains reasonable (and even better) perplexity, rouge score and accuracy over the original LLaMA-2-7B-Chat model.
+We observe that our finetuned Llama-2-7B-32K-Instruct consistently outperforms other baseline models including Llama-2-7b-chat, Longchat-7b-16k and Longchat-7b-v1.5-32k.
 
 ## Limitations and Bias