Hugging Face's logo

botbot-ai
/
CabraLlama3-70b

Text Generation
Transformers
Safetensors
Portuguese
llama
portuguese
cabra
llama-3
conversational
Eval Results
text-generation-inference
Inference Endpoints
Model card Files Community
1

Adding the Open Portuguese LLM Leaderboard Evaluation Results

#1
by leaderboard-pt-pr-bot - opened
base: refs/heads/main
←
from: refs/pr/1
Discussion Files changed
+169
-3
Files changed (1) hide show
  1. README.md +169 -3
README.md CHANGED
@@ -1,6 +1,4 @@
1
  ---
2
- datasets:
3
- - botbot-ai/Cabra3k
4
  language:
5
  - pt
6
  license: llama3
@@ -10,6 +8,155 @@ tags:
10
  - llama
11
  - cabra
12
  - llama-3
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
13
  ---
14
  # Cabra Llama-3 70B
15
 
@@ -31,4 +178,23 @@ Dataset interno para fine-tuning. Vamos lançar em breve.
31
 
32
  ### Quantização / GGUF
33
 
34
- Colocamos diversas versões (GGUF) quantanizadas no branch "quantanization".
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
 
 
2
  language:
3
  - pt
4
  license: llama3
 
8
  - llama
9
  - cabra
10
  - llama-3
11
+ datasets:
12
+ - botbot-ai/Cabra3k
13
+ model-index:
14
+ - name: CabraLlama3-70b
15
+ results:
16
+ - task:
17
+ type: text-generation
18
+ name: Text Generation
19
+ dataset:
20
+ name: ENEM Challenge (No Images)
21
+ type: eduagarcia/enem_challenge
22
+ split: train
23
+ args:
24
+ num_few_shot: 3
25
+ metrics:
26
+ - type: acc
27
+ value: 82.02
28
+ name: accuracy
29
+ source:
30
+ url: https://huggingface.co/spaces/eduagarcia/open_pt_llm_leaderboard?query=botbot-ai/CabraLlama3-70b
31
+ name: Open Portuguese LLM Leaderboard
32
+ - task:
33
+ type: text-generation
34
+ name: Text Generation
35
+ dataset:
36
+ name: BLUEX (No Images)
37
+ type: eduagarcia-temp/BLUEX_without_images
38
+ split: train
39
+ args:
40
+ num_few_shot: 3
41
+ metrics:
42
+ - type: acc
43
+ value: 70.1
44
+ name: accuracy
45
+ source:
46
+ url: https://huggingface.co/spaces/eduagarcia/open_pt_llm_leaderboard?query=botbot-ai/CabraLlama3-70b
47
+ name: Open Portuguese LLM Leaderboard
48
+ - task:
49
+ type: text-generation
50
+ name: Text Generation
51
+ dataset:
52
+ name: OAB Exams
53
+ type: eduagarcia/oab_exams
54
+ split: train
55
+ args:
56
+ num_few_shot: 3
57
+ metrics:
58
+ - type: acc
59
+ value: 68.52
60
+ name: accuracy
61
+ source:
62
+ url: https://huggingface.co/spaces/eduagarcia/open_pt_llm_leaderboard?query=botbot-ai/CabraLlama3-70b
63
+ name: Open Portuguese LLM Leaderboard
64
+ - task:
65
+ type: text-generation
66
+ name: Text Generation
67
+ dataset:
68
+ name: Assin2 RTE
69
+ type: assin2
70
+ split: test
71
+ args:
72
+ num_few_shot: 15
73
+ metrics:
74
+ - type: f1_macro
75
+ value: 93.21
76
+ name: f1-macro
77
+ source:
78
+ url: https://huggingface.co/spaces/eduagarcia/open_pt_llm_leaderboard?query=botbot-ai/CabraLlama3-70b
79
+ name: Open Portuguese LLM Leaderboard
80
+ - task:
81
+ type: text-generation
82
+ name: Text Generation
83
+ dataset:
84
+ name: Assin2 STS
85
+ type: eduagarcia/portuguese_benchmark
86
+ split: test
87
+ args:
88
+ num_few_shot: 15
89
+ metrics:
90
+ - type: pearson
91
+ value: 83.32
92
+ name: pearson
93
+ source:
94
+ url: https://huggingface.co/spaces/eduagarcia/open_pt_llm_leaderboard?query=botbot-ai/CabraLlama3-70b
95
+ name: Open Portuguese LLM Leaderboard
96
+ - task:
97
+ type: text-generation
98
+ name: Text Generation
99
+ dataset:
100
+ name: FaQuAD NLI
101
+ type: ruanchaves/faquad-nli
102
+ split: test
103
+ args:
104
+ num_few_shot: 15
105
+ metrics:
106
+ - type: f1_macro
107
+ value: 80.6
108
+ name: f1-macro
109
+ source:
110
+ url: https://huggingface.co/spaces/eduagarcia/open_pt_llm_leaderboard?query=botbot-ai/CabraLlama3-70b
111
+ name: Open Portuguese LLM Leaderboard
112
+ - task:
113
+ type: text-generation
114
+ name: Text Generation
115
+ dataset:
116
+ name: HateBR Binary
117
+ type: ruanchaves/hatebr
118
+ split: test
119
+ args:
120
+ num_few_shot: 25
121
+ metrics:
122
+ - type: f1_macro
123
+ value: 81.62
124
+ name: f1-macro
125
+ source:
126
+ url: https://huggingface.co/spaces/eduagarcia/open_pt_llm_leaderboard?query=botbot-ai/CabraLlama3-70b
127
+ name: Open Portuguese LLM Leaderboard
128
+ - task:
129
+ type: text-generation
130
+ name: Text Generation
131
+ dataset:
132
+ name: PT Hate Speech Binary
133
+ type: hate_speech_portuguese
134
+ split: test
135
+ args:
136
+ num_few_shot: 25
137
+ metrics:
138
+ - type: f1_macro
139
+ value: 72.72
140
+ name: f1-macro
141
+ source:
142
+ url: https://huggingface.co/spaces/eduagarcia/open_pt_llm_leaderboard?query=botbot-ai/CabraLlama3-70b
143
+ name: Open Portuguese LLM Leaderboard
144
+ - task:
145
+ type: text-generation
146
+ name: Text Generation
147
+ dataset:
148
+ name: tweetSentBR
149
+ type: eduagarcia/tweetsentbr_fewshot
150
+ split: test
151
+ args:
152
+ num_few_shot: 25
153
+ metrics:
154
+ - type: f1_macro
155
+ value: 73.85
156
+ name: f1-macro
157
+ source:
158
+ url: https://huggingface.co/spaces/eduagarcia/open_pt_llm_leaderboard?query=botbot-ai/CabraLlama3-70b
159
+ name: Open Portuguese LLM Leaderboard
160
  ---
161
  # Cabra Llama-3 70B
162
 
 
178
 
179
  ### Quantização / GGUF
180
 
181
+ Colocamos diversas versões (GGUF) quantanizadas no branch "quantanization".
182
+
183
+
184
+ # Open Portuguese LLM Leaderboard Evaluation Results
185
+
186
+ Detailed results can be found [here](https://huggingface.co/datasets/eduagarcia-temp/llm_pt_leaderboard_raw_results/tree/main/botbot-ai/CabraLlama3-70b) and on the [🚀 Open Portuguese LLM Leaderboard](https://huggingface.co/spaces/eduagarcia/open_pt_llm_leaderboard)
187
+
188
+ | Metric | Value |
189
+ |--------------------------|---------|
190
+ |Average |**78.44**|
191
+ |ENEM Challenge (No Images)| 82.02|
192
+ |BLUEX (No Images) | 70.10|
193
+ |OAB Exams | 68.52|
194
+ |Assin2 RTE | 93.21|
195
+ |Assin2 STS | 83.32|
196
+ |FaQuAD NLI | 80.60|
197
+ |HateBR Binary | 81.62|
198
+ |PT Hate Speech Binary | 72.72|
199
+ |tweetSentBR | 73.85|
200
+