Update README.md
Browse files
README.md
CHANGED
@@ -44,7 +44,7 @@ The following table shows the hyper-paramters we used in our training process.
|
|
44 |
| Batch Size | 4M |
|
45 |
| Weight Decay | 0.1 |
|
46 |
|
47 |
-
**Second phase**: We further adjusted the training corpus ratio, incorporating more domain-specific datasets(Math
|
48 |
|
49 |
| Hyper-parameters | |
|
50 |
| --------------------- | ----------- |
|
@@ -59,8 +59,7 @@ The following table shows the hyper-paramters we used in our training process.
|
|
59 |
Our evaluation is based on the framework lm-evaluation-harness and opencompass. The evaluation details are listed as follows:
|
60 |
|
61 |
- Huggingface LLM Leaderboard tasks.
|
62 |
-
-
|
63 |
-
- Other Popular Benchmarks: We report the average accuracies on Big Bench Hard (BBH) (3-shot), HumanEval, MBPP, MATH.
|
64 |
|
65 |
| | MMLU | Winogrande | TruthfulQA | Hellaswag | GSM8K | Arc-C | HumanEval | BBH | Average |
|
66 |
| ------- | ------ | ---------- | ---------- | --------- | ------ | ------ | --------- | ---- | ------- |
|
|
|
44 |
| Batch Size | 4M |
|
45 |
| Weight Decay | 0.1 |
|
46 |
|
47 |
+
**Second phase**: We further adjusted the training corpus ratio, incorporating more domain-specific datasets(Math, Coding), and continued training for 50B tokens.
|
48 |
|
49 |
| Hyper-parameters | |
|
50 |
| --------------------- | ----------- |
|
|
|
59 |
Our evaluation is based on the framework lm-evaluation-harness and opencompass. The evaluation details are listed as follows:
|
60 |
|
61 |
- Huggingface LLM Leaderboard tasks.
|
62 |
+
- Other Popular Benchmarks: We report the average accuracies on Big Bench Hard (BBH) (3-shot), HumanEval.
|
|
|
63 |
|
64 |
| | MMLU | Winogrande | TruthfulQA | Hellaswag | GSM8K | Arc-C | HumanEval | BBH | Average |
|
65 |
| ------- | ------ | ---------- | ---------- | --------- | ------ | ------ | --------- | ---- | ------- |
|