Comprehensive overhaul of README.md for better documentation of the model

README.md

@@ -1,35 +1,75 @@
 ---
 license: apache-2.0
----
-
-<!-- This model card has been generated automatically according to the information the Trainer had access to. You
-should probably proofread and complete it, then remove this comment. -->
-
-This model is a fine-tuned model for Chat based on [mosaicml/mpt-7b](https://huggingface.co/mosaicml/mpt-7b) with **max_seq_lenght=2048** on various open source dataset. For the details of the used dataset, please refer to [Intel/neural-chat-dataset-v1-1](https://huggingface.co/datasets/Intel/neural-chat-dataset-v1-1).
+tags:
+- LLMs
+- Intel
+base_model: mosaicml/mpt-7b
+datasets:
+- Intel/neural-chat-dataset-v1-1
+- allenai/real-toxicity-prompts
+language:
+- en
+model-index:
+- name: neural-chat-7b-v1-1
+  results:
+  - task:
+      type: Large Language Model
+      name: Large Language Model
+    dataset:
+      type: Intel/neural-chat-dataset-v1-1
+      name: Intel/neural-chat-dataset-v1-1
+    metrics:
+    - type: Average
+      value: 51.41
+      name: Average
+      verified: true
+    - type: ARC (25-shot)
+      value: 50.09 
+      name: ARC (25-shot)
+      verified: true
+    - type: HellaSwag (10-shot)
+      value: 76.69
+      name: HellaSwag (10-shot)
+      verified: true
+    - type: MMLU (5-shot)
+      value: 38.79
+      name: MMLU (5-shot)
+      verified: true
+    - type: TruthfulQA (0-shot)
+      value: 40.07
+      name: TruthfulQA (0-shot)
+      verified: true
+    - type: Toxicity Rito
+      value: 0.0264
+      name: Toxicity Rito
 
-## Model date
-Neural-chat-7b-v1.1 was trained between June and July 2023.
+---
+## Model Details: Neural-Chat-v1-1
 
-## Evaluation
-We use the same evaluation metrics as [open_llm_leaderboard](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard) which uses [Eleuther AI Language Model Evaluation Harness](https://github.com/EleutherAI/lm-evaluation-harness/tree/master), a unified framework to test generative language models on a large number of different evaluation tasks.
+This model is a fine-tuned model for chat based on [mosaicml/mpt-7b](https://huggingface.co/mosaicml/mpt-7b) with a max sequence length of 2048 on the dataset [Intel/neural-chat-dataset-v1-1](https://huggingface.co/datasets/Intel/neural-chat-dataset-v1-1), which is a compilation of open-source datasets.
 
-| Model | Average ⬆️| ARC (25-s) ⬆️ | HellaSwag (10-s) ⬆️ | MMLU (5-s) ⬆️| TruthfulQA (MC) (0-s) ⬆️ |
-| --- | --- | --- | --- | --- | --- |
-|[mosaicml/mpt-7b](https://huggingface.co/mosaicml/mpt-7b)| 47.4  | 47.61 | 77.56 | 31 | 33.43 |
-| [mosaicml/mpt-7b-chat](https://huggingface.co/mosaicml/mpt-7b-chat) | **49.95** | 46.5 | 75.55 | 37.60 | 40.17 |
-| **Ours** | **51.41** | 50.09 | 76.69 | 38.79 | 40.07 |
+<p align="center">
+  <img src="https://cdn-uploads.huggingface.co/production/uploads/6297f0e30bd2f58c647abb1d/fWCqhGKZQKNuLmvj093rB.jpeg" width="500"/>
+  Prompt of "an image of a brain that has to do with LLMs" from https://clipdrop.co/stable-diffusion-turbo.
+</p>
 
-### Bias evaluation
+| Model Detail | Description |
+| ----------- | ----------- | 
+| Model Authors | Intel. The NeuralChat team with members from DCAI/AISE/AIPT. Core team members: Kaokao Lv, Liang Lv, Chang Wang, Wenxin Zhang, Xuhui Ren, and Haihao Shen. | 
+| Date | July, 2023 | 
+| Version | v1-1 | 
+| Type | 7B Large Language Model | 
+| Paper or Other Resources | Base model: [mosaicml/mpt-7b](https://huggingface.co/mosaicml/mpt-7b); Dataset: [Intel/neural-chat-dataset-v1-1](https://huggingface.co/datasets/Intel/neural-chat-dataset-v1-1) | 
+| License | Apache 2.0 |
+| Questions or Comments | [Community Tab](https://huggingface.co/Intel/neural-chat-7b-v1-1/discussions) and [Intel DevHub Discord](https://discord.gg/rv2Gp55UJQ)|
 
-Following the blog [evaluating-llm-bias](https://huggingface.co/blog/evaluating-llm-bias), we select 10000 samples randomly from [allenai/real-toxicity-prompts](https://huggingface.co/datasets/allenai/real-toxicity-prompts) to evaluate toxicity bias in Language Models
+| Intended Use | Description |
+| ----------- | ----------- | 
+| Primary intended uses | You can use the fine-tuned model for several language-related tasks. Checkout the [LLM Leaderboard](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard) to see this model's performance relative to other LLMs. | 
+| Primary intended users | Anyone doing inference on language-related tasks. | 
+| Out-of-scope uses | This model in most cases will need to be fine-tuned for your particular task.  The model should not be used to intentionally create hostile or alienating environments for people.|
 
-| Model | Toxicity Rito ↓|
-| --- | --- |
-|[mosaicml/mpt-7b](https://huggingface.co/mosaicml/mpt-7b)| 0.027 |
-| **Ours** | 0.0264 |
-
-
-## Training procedure
+## How To Use
 
 ### Training hyperparameters
 
@@ -48,9 +88,10 @@ The following hyperparameters were used during training:
 - lr_scheduler_warmup_ratio: 0.02
 - num_epochs: 3.0
 
-## Inference with transformers
+## Use The Model
 
-```shell
+### Loading the model with Transformers
+```python
 import transformers
 model = transformers.AutoModelForCausalLM.from_pretrained(
   'Intel/neural-chat-7b-v1-1',
@@ -58,10 +99,10 @@ model = transformers.AutoModelForCausalLM.from_pretrained(
 )
 ```
 
-## Inference with INT8
-Follow the instructions [link](https://github.com/intel/intel-extension-for-transformers/tree/main/examples/huggingface/pytorch/text-generation/quantization) to install the necessary dependencies. Use the below command to quantize the model using Intel Neural Compressor [link](https://github.com/intel/neural-compressor) and accelerate the inference.
+### Inference with INT8
+Follow the instructions at the [GitHub repository](https://github.com/intel/intel-extension-for-transformers/tree/main/examples/huggingface/pytorch/text-generation/quantization) to install the necessary dependencies for quantization to INT8. Use the below command to quantize the model using [Intel Neural Compressor](https://github.com/intel/neural-compressor) to accelerate inference.
 
-```shell
+```bash
 python run_generation.py \
     --model Intel/neural-chat-7b-v1-1 \
     --quantize \
@@ -70,6 +111,55 @@ python run_generation.py \
     --ipex
 ```
 
+| Factors | Description | 
+| ----------- | ----------- | 
+| Groups | More details about the dataset can be found at [Intel/neural-chat-dataset-v1-1](https://huggingface.co/datasets/Intel/neural-chat-dataset-v1-1). | 
+| Instrumentation | The performance of the model can vary depending on the inputs to the model. In this case, the prompts provided can drastically change the prediction of the language model. |
+| Environment | - |
+| Card Prompts | Model deployment on varying hardware and software will change model performance.  |
+
+| Metrics | Description | 
+| ----------- | ----------- | 
+| Model performance measures | The model metrics are: ARC, HellaSwag, MMLU, and TruthfulQA. Bias evaluation was also evaluated using using Toxicity Rito (see Quantitative Analyses below). The model performance was evaluated against other LLMs according to the standards at the time the model was published. |
+| Decision thresholds | No decision thresholds were used. | 
+| Approaches to uncertainty and variability | - | 
+
+
+## Training Data
+
+The training data are from [Intel/neural-chat-dataset-v1-1](https://huggingface.co/datasets/Intel/neural-chat-dataset-v1-1). The total number of instruction samples is about 1.1M, and the number of tokens is 326M. This dataset is composed of several other datasets:
+
+| Type         | Language | Dataset                                                                                                                           | Number |
+  |--| ---- |--------|----|
+  | HC3  | en | [HC3](https://huggingface.co/datasets/Hello-SimpleAI/HC3)                                 | 24K   |
+  | dolly  | en | [databricks-dolly-15k](https://huggingface.co/datasets/databricks/databricks-dolly-15k)                                 | 15K   |
+  | alpaca-zh  | zh | [tigerbot-alpaca-zh-0.5m](https://huggingface.co/datasets/TigerResearch/tigerbot-alpaca-zh-0.5m)                                 | 500K   |
+  | alpaca-en  | en | [TigerResearch/tigerbot-alpaca-en-50k](https://huggingface.co/datasets/TigerResearch/tigerbot-alpaca-en-50k)                                  | 50K   |
+  | math     | en | [tigerbot-gsm-8k-en](https://huggingface.co/datasets/TigerResearch/tigerbot-gsm-8k-en)                                           | 8K     |
+  | general     | en | [tigerbot-stackexchange-qa-en-0.5m](https://huggingface.co/datasets/TigerResearch/tigerbot-stackexchange-qa-en-0.5m)             | 500K    |
+
+Note: There is no contamination from the GSM8k test set, as this is not a part of this dataset.
+
+## Quantitative Analyses
+
+### LLM metrics
+We used the same evaluation metrics as [HuggingFaceH4/open_llm_leaderboard](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard), which uses [Eleuther AI Language Model Evaluation Harness](https://github.com/EleutherAI/lm-evaluation-harness/tree/master), a unified framework to test generative language models on a large number of different evaluation tasks.
+
+| Model | Average ⬆️| ARC (25-s) ⬆️ | HellaSwag (10-s) ⬆️ | MMLU (5-s) ⬆️| TruthfulQA (MC) (0-s) ⬆️ |
+| --- | --- | --- | --- | --- | --- |
+|[mosaicml/mpt-7b](https://huggingface.co/mosaicml/mpt-7b)| 47.4  | 47.61 | 77.56 | 31 | 33.43 |
+| [mosaicml/mpt-7b-chat](https://huggingface.co/mosaicml/mpt-7b-chat) | **49.95** | 46.5 | 75.55 | 37.60 | 40.17 |
+| [Intel/neural-chat-dataset-v1-1](https://huggingface.co/Intel/neural-chat-dataset-v1-1) | **51.41** | 50.09 | 76.69 | 38.79 | 40.07 |
+
+### Bias evaluation
+
+Following the blog [evaluating-llm-bias](https://huggingface.co/blog/evaluating-llm-bias), we selected 10000 samples randomly from [allenai/real-toxicity-prompts](https://huggingface.co/datasets/allenai/real-toxicity-prompts) to evaluate toxicity bias. 
+
+| Model | Toxicity Rito ↓|
+| --- | --- |
+|[mosaicml/mpt-7b](https://huggingface.co/mosaicml/mpt-7b)| 0.027 |
+| [Intel/neural-chat-dataset-v1-1](https://huggingface.co/Intel/neural-chat-dataset-v1-1) | 0.0264 |
+
 ### Examples
 
 - code generation
@@ -82,21 +172,20 @@ python run_generation.py \
 ![trip](examples/trip.png)
 
 ## Ethical Considerations and Limitations
-neural-chat-7b-v1-1 can produce factually incorrect output, and should not be relied on to produce factually accurate information. neural-chat-7b-v1-1 was trained on various instruction/chat datasets based on [mosaicml/mpt-7b](https://huggingface.co/mosaicml/mpt-7b). Because of the limitations of the pretrained model and the finetuning datasets, it is possible that this model could generate lewd, biased or otherwise offensive outputs.
+Neural-chat-7b-v1-1 can produce factually incorrect output, and should not be relied on to produce factually accurate information. neural-chat-7b-v1-1 was trained on various instruction/chat datasets based on [mosaicml/mpt-7b](https://huggingface.co/mosaicml/mpt-7b). Because of the limitations of the pretrained model and the finetuning datasets, it is possible that this model could generate lewd, biased or otherwise offensive outputs.
 
-Therefore, before deploying any applications of neural-chat-7b-v1-1, developers should perform safety testing.
+Therefore, before deploying any applications of the model, developers should perform safety testing.
 
-## Disclaimer
-
-The license on this model does not constitute legal advice. We are not responsible for the actions of third parties who use this model. Please cosult an attorney before using this model for commercial purposes.
-
-
-
-## Organizations developing the model
+## Caveats and Recommendations
 
-The NeuralChat team with members from Intel/SATG/AIA/AIPT. Core team members: Kaokao Lv, Liang Lv, Chang Wang, Wenxin Zhang, Xuhui Ren, and Haihao Shen.
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model.
 
-## Useful links
+Here are some useful GitHub repository links to learn more about Intel's open-source AI software:
 * Intel Neural Compressor [link](https://github.com/intel/neural-compressor)
 * Intel Extension for Transformers [link](https://github.com/intel/intel-extension-for-transformers)
 * Intel Extension for PyTorch [link](https://github.com/intel/intel-extension-for-pytorch)
+
+## Disclaimer
+
+The license on this model does not constitute legal advice. We are not responsible for the actions of third parties who use this model. Please cosult an attorney before using this model for commercial purposes.
+