Hugging Face's logo

DevQuasar
/
analytical_reasoning_r16a32_unsloth-Llama-3.2-3B-Instruct-bnb-4bit

Text Generation
Transformers
Safetensors
llama
unsloth
conversational
Eval Results
text-generation-inference
Inference Endpoints
Model card Files Community
csabakecskemeti commited on
Commit
3900a10
•
1 Parent(s): 5a69800

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +19 -53
README.md CHANGED
@@ -31,59 +31,25 @@ model-index:
31
 
32
  ### eval
33
 
34
-
35
-
36
- | Tasks |Version|Filter|n-shot| Metric | |Value | |Stderr|
37
- |----------------------------------------------------------|-------|------|-----:|-----------------------|---|-----:|---|------|
38
- |hellaswag | 1|none | 0|acc |↑ |0.5141|± |0.0050|
39
- | | |none | 0|acc_norm |↑ |0.6793|± |0.0047|
40
- |leaderboard_bbh | N/A| | | | | | | |
41
- | - leaderboard_bbh_boolean_expressions | 1|none | 3|acc_norm |↑ |0.6040|± |0.0310|
42
- | - leaderboard_bbh_causal_judgement | 1|none | 3|acc_norm |↑ |0.5668|± |0.0363|
43
- | - leaderboard_bbh_date_understanding | 1|none | 3|acc_norm |↑ |0.4880|± |0.0317|
44
- | - leaderboard_bbh_disambiguation_qa | 1|none | 3|acc_norm |↑ |0.3760|± |0.0307|
45
- | - leaderboard_bbh_formal_fallacies | 1|none | 3|acc_norm |↑ |0.5400|± |0.0316|
46
- | - leaderboard_bbh_geometric_shapes | 1|none | 3|acc_norm |↑ |0.2200|± |0.0263|
47
- | - leaderboard_bbh_hyperbaton | 1|none | 3|acc_norm |↑ |0.5640|± |0.0314|
48
- | - leaderboard_bbh_logical_deduction_five_objects | 1|none | 3|acc_norm |↑ |0.4560|± |0.0316|
49
- | - leaderboard_bbh_logical_deduction_seven_objects | 1|none | 3|acc_norm |↑ |0.4360|± |0.0314|
50
- | - leaderboard_bbh_logical_deduction_three_objects | 1|none | 3|acc_norm |↑ |0.4880|± |0.0317|
51
- | - leaderboard_bbh_movie_recommendation | 1|none | 3|acc_norm |↑ |0.6360|± |0.0305|
52
- | - leaderboard_bbh_navigate | 1|none | 3|acc_norm |↑ |0.6200|± |0.0308|
53
- | - leaderboard_bbh_object_counting | 1|none | 3|acc_norm |↑ |0.4120|± |0.0312|
54
- | - leaderboard_bbh_penguins_in_a_table | 1|none | 3|acc_norm |↑ |0.3219|± |0.0388|
55
- | - leaderboard_bbh_reasoning_about_colored_objects | 1|none | 3|acc_norm |↑ |0.3440|± |0.0301|
56
- | - leaderboard_bbh_ruin_names | 1|none | 3|acc_norm |↑ |0.3240|± |0.0297|
57
- | - leaderboard_bbh_salient_translation_error_detection | 1|none | 3|acc_norm |↑ |0.3120|± |0.0294|
58
- | - leaderboard_bbh_snarks | 1|none | 3|acc_norm |↑ |0.4494|± |0.0374|
59
- | - leaderboard_bbh_sports_understanding | 1|none | 3|acc_norm |↑ |0.6040|± |0.0310|
60
- | - leaderboard_bbh_temporal_sequences | 1|none | 3|acc_norm |↑ |0.1000|± |0.0190|
61
- | - leaderboard_bbh_tracking_shuffled_objects_five_objects | 1|none | 3|acc_norm |↑ |0.1600|± |0.0232|
62
- | - leaderboard_bbh_tracking_shuffled_objects_seven_objects| 1|none | 3|acc_norm |↑ |0.1200|± |0.0206|
63
- | - leaderboard_bbh_tracking_shuffled_objects_three_objects| 1|none | 3|acc_norm |↑ |0.3440|± |0.0301|
64
- | - leaderboard_bbh_web_of_lies | 1|none | 3|acc_norm |↑ |0.5160|± |0.0317|
65
- |leaderboard_gpqa | N/A| | | | | | | |
66
- | - leaderboard_gpqa_diamond | 1|none | 0|acc_norm |↑ |0.2727|± |0.0317|
67
- | - leaderboard_gpqa_extended | 1|none | 0|acc_norm |↑ |0.2802|± |0.0192|
68
- | - leaderboard_gpqa_main | 1|none | 0|acc_norm |↑ |0.2545|± |0.0206|
69
- |leaderboard_ifeval | 3|none | 0|inst_level_loose_acc |↑ |0.5252|± | N/A|
70
- | | |none | 0|inst_level_strict_acc |↑ |0.4748|± | N/A|
71
- | | |none | 0|prompt_level_loose_acc |↑ |0.3919|± |0.0210|
72
- | | |none | 0|prompt_level_strict_acc|↑ |0.3420|± |0.0204|
73
- |leaderboard_math_hard | N/A| | | | | | | |
74
- | - leaderboard_math_algebra_hard | 2|none | 4|exact_match |↑ |0.2150|± |0.0235|
75
- | - leaderboard_math_counting_and_prob_hard | 2|none | 4|exact_match |↑ |0.0244|± |0.0140|
76
- | - leaderboard_math_geometry_hard | 2|none | 4|exact_match |↑ |0.0606|± |0.0208|
77
- | - leaderboard_math_intermediate_algebra_hard | 2|none | 4|exact_match |↑ |0.0143|± |0.0071|
78
- | - leaderboard_math_num_theory_hard | 2|none | 4|exact_match |↑ |0.0649|± |0.0199|
79
- | - leaderboard_math_prealgebra_hard | 2|none | 4|exact_match |↑ |0.1762|± |0.0275|
80
- | - leaderboard_math_precalculus_hard | 2|none | 4|exact_match |↑ |0.0519|± |0.0192|
81
- |leaderboard_mmlu_pro | 0.1|none | 5|acc |↑ |0.2822|± |0.0041|
82
- |leaderboard_musr | N/A| | | | | | | |
83
- | - leaderboard_musr_murder_mysteries | 1|none | 0|acc_norm |↑ |0.5400|± |0.0316|
84
- | - leaderboard_musr_object_placements | 1|none | 0|acc_norm |↑ |0.2344|± |0.0265|
85
- | - leaderboard_musr_team_allocation | 1|none | 0|acc_norm |↑ |0.3200|± |0.0296|
86
-
87
 
88
 
89
  ### Framework versions
 
31
 
32
  ### eval
33
 
34
+ | Test | Base Model | Fine-Tuned Model | Performance Gain |
35
+ |---|---|---|---|
36
+ | leaderboard_bbh_logical_deduction_seven_objects | 0.252 | 0.436 | 0.184 |
37
+ | leaderboard_bbh_logical_deduction_five_objects | 0.356 | 0.456 | 0.10000000000000003 |
38
+ | leaderboard_musr_team_allocation | 0.22 | 0.32 | 0.1 |
39
+ | leaderboard_bbh_disambiguation_qa | 0.304 | 0.376 | 0.07200000000000001 |
40
+ | leaderboard_gpqa_diamond | 0.2222222222222222 | 0.2727272727272727 | 0.0505050505050505 |
41
+ | leaderboard_bbh_movie_recommendation | 0.596 | 0.636 | 0.040000000000000036 |
42
+ | leaderboard_bbh_formal_fallacies | 0.508 | 0.54 | 0.03200000000000003 |
43
+ | leaderboard_bbh_tracking_shuffled_objects_three_objects | 0.316 | 0.344 | 0.02799999999999997 |
44
+ | leaderboard_bbh_causal_judgement | 0.5454545454545454 | 0.5668449197860963 | 0.021390374331550888 |
45
+ | leaderboard_bbh_web_of_lies | 0.496 | 0.516 | 0.020000000000000018 |
46
+ | leaderboard_math_geometry_hard | 0.045454545454545456 | 0.06060606060606061 | 0.015151515151515152 |
47
+ | leaderboard_math_num_theory_hard | 0.05194805194805195 | 0.06493506493506493 | 0.012987012987012977 |
48
+ | leaderboard_musr_murder_mysteries | 0.528 | 0.54 | 0.01200000000000001 |
49
+ | leaderboard_gpqa_extended | 0.27106227106227104 | 0.2802197802197802 | 0.00915750915750918 |
50
+ | leaderboard_bbh_sports_understanding | 0.596 | 0.604 | 0.008000000000000007 |
51
+ | leaderboard_math_intermediate_algebra_hard | 0.010714285714285714 | 0.014285714285714285 | 0.003571428571428571 |
52
+ | leaderboard_bbh_navigate | 0.62 | 0.62 | 0.0 |
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
53
 
54
 
55
  ### Framework versions