Update README.md

README.md

@@ -18,3 +18,106 @@ OpenCSG stands for Converged resources, Software refined, and Generative LM. The
 
 The vision of OpenCSG is to empower every industry, every company, and every individual to own their models. We adhere to the principles of openness and open source, making the large model software stack of OpenCSG available to the community. We welcome everyone to use, feedback, and collaborative contribute.
 
+
+
+## Model Description
+
+The [StarCoder](https://huggingface.co/bigcode/starcoder) models are 15.5B parameter models trained on 80+ programming languages from [The Stack (v1.2)](https://huggingface.co/datasets/bigcode/the-stack), with opt-out requests excluded.
+<br>
+
+This is the repository for the base 7B version finetuned based on [CodeLlama-7b-hf](https://huggingface.co/codellama/CodeLlama-7b-hf). 
+
+| Model Size    | Base Model                                                                    |
+| --- | ----------------------------------------------------------------------------- |
+| 7B  | [opencsg/Opencsg-CodeLlama-7b-v0.1](https://huggingface.co/opencsg/opencsg-CodeLlama-7b-v0.1) |
+| 13B  | [opencsg/Opencsg-CodeLlama-13b-v0.1](https://huggingface.co/opencsg/opencsg-CodeLlama-13b-v0.1) | 
+| 34B  | [opencsg/Opencsg-CodeLlama-34b-v0.1](https://huggingface.co/opencsg/opencsg-CodeLlama-34b-v0.1) |
+
+
+## Model Eval
+
+HumanEval is the commonest code generation benchmark to evaluate the performance of models, especially on the the compeltion of code exercise cases.
+Somehow, model evaluation is a kind of metaphysics. Different models are sensitive to different decoding methods, paramters and instructions.
+It is impratical for us to manually set specific configuration for each fine-tuned model, because a real LLM should master the universal capability despite the parameters manipulated by users.
+
+Thus, OpenCSG strained our brains to provide a relatively fair method to compare the fine-tuned models on HumanEval benchmark.
+To simplify the comparision, we chosed the Pass@1 metric on python language, but our finetuning dataset includes samples in multi language.
+
+**For fair, we evaluated the fine-tuned and origin codellama models only with the original cases' prompts, not including any other instruction else.**
+
+**Otherwise, we use greedy decoding method for each model during the evaluation.**
+
+| Model     | HumanEval python pass@1                                                 |
+| ---  |----------------------------------------------------------------------------- |
+| CodeLlama-7b-hf  | 30.5%|
+| opencsg-CodeLlama-7b-v0.1(4k) | **42.7%** |
+| CodeLlama-13b-hf  | 36.0%|
+| opencsg-CodeLlama-13b-v0.1(4k) | **45.1%** |
+| CodeLlama-34b-hf  |  48.2%|
+| opencsg-CodeLlama-34b-v0.1(4k)| **48.8%** |
+
+**TODO**
+- we will provide much more benchmark scores on fine-tuned models in future.
+- we will provide different practical problems to evaluate the performance of fine-tuned models in the field of software engineering.
+
+
+
+# Model Usage
+
+
+```python
+from transformers import AutoTokenizer
+import transformers
+import torch
+
+model = "opencsg/opencsg-CodeLlama-7b-v0.1"
+
+tokenizer = AutoTokenizer.from_pretrained(model, trust_remote_code=True)
+pipeline = transformers.pipeline(
+    "text-generation",
+    model=model,
+    torch_dtype=torch.float16,
+    device_map="auto",
+)
+input_text = """def quick_sort(arr):
+    if len(arr) <= 1:
+        return arr
+    pivot = arr[0]
+    left = []
+    right = []
+<FILL_ME>
+        if arr[i] < pivot:
+            left.append(arr[i])
+        else:
+            right.append(arr[i])
+    return quick_sort(left) + [pivot] + quick_sort(right)
+"""
+sequences = pipeline(
+    input_text,
+    do_sample=False,
+    top_k=10,
+    temperature=0.1,
+    top_p=0.95,
+    num_return_sequences=1,
+    eos_token_id=tokenizer1.eos_token_id,
+    max_length=256,
+)
+for seq in sequences:
+    print(seq['generated_text'][len(input_text):])
+```
+# Training
+
+## Basic Model
+
+[codellama-7b-hf](https://huggingface.co/codellama/CodeLlama-7b-hf)
+
+## Hardware
+
+- **GPUs:** 8 Tesla A800 
+- **Training time:** 4 hours
+
+## Software
+
+- **Orchestration:** [Deepspeed](https://github.com/OpenCSGs)
+- **Neural networks:** [PyTorch](https://github.com/pytorch/pytorch)
+- **BP16 if applicable:** [apex](https://github.com/NVIDIA/apex)