Hugging Face's logo

t-tech
/
T-pro-it-1.0

Safetensors
Russian
qwen2
Model card Files Community
8
germanjke commited on
Commit
77156fd
•
1 Parent(s): 8340bb9

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +83 -4
README.md CHANGED
@@ -10,18 +10,72 @@ language:
10
 
11
  T-pro-it-1.0 is a model built upon the Qwen 2.5 model family and incorporates both continual pre-training and alignment techniques.
12
 
13
- Detailed model card’s coming soon…
14
-
15
  ### 📚 Dataset
16
 
17
- Detailed model card’s coming soon…
 
 
 
 
 
 
 
 
 
 
18
 
19
  ## 📊 Benchmarks
20
 
21
- Detailed model card’s coming soon…
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
22
 
23
  ## 👨‍💻 Examples of usage
24
 
 
25
 
26
  ```python
27
  from transformers import AutoTokenizer, AutoModelForCausalLM
@@ -77,4 +131,29 @@ Output:
77
  Поиск закономерностей — его цель, открыть тайны бытия.
78
  От распознавания лиц до понимания речи,
79
  Машинное обучение — это ключ, что открывает двери.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
80
  ```
 
10
 
11
  T-pro-it-1.0 is a model built upon the Qwen 2.5 model family and incorporates both continual pre-training and alignment techniques.
12
 
 
 
13
  ### 📚 Dataset
14
 
15
+ Pre-training Stage 1:
16
+ 100B tokens, consisting of diverse Russian data from Common Crawl, books, code, and proprietary datasets, mixed with re-played English data (English added as it is the primary language of the base model).
17
+
18
+ Pre-training Stage 2:
19
+ 40B tokens, a mix of instruction and pre-training data.
20
+
21
+ Supervised Fine-Tuning (SFT):
22
+ 1B tokens, a mix of diverse instruction data.
23
+
24
+ Preference Tuning:
25
+ 1B tokens, training the model to be helpful.
26
 
27
  ## 📊 Benchmarks
28
 
29
+ Proprietary models:
30
+
31
+ | Benchmark | T-pro-it-1.0 | GPT-4o | GPT-4o-mini | GigaChat Max 1.0.26.20 |
32
+ |------------------------------------------------|-----------------------|------------------------------|-----------------------|---------------------|
33
+ | [MERA](https://mera.a-ai.ru) | <u>0.629</u> | **0.642** | 0.57 | 0.588 |
34
+ | [MaMuRaMu](https://mera.a-ai.ru/ru/tasks/22) | <u>0.841</u> | **0.874** | 0.779 | 0.824 |
35
+ | ruMMLU-PRO | <u>0.665</u> | **0.713** | 0.573 | 0.535 |
36
+ | ruGSM8K | **0.941** | <u>0.931</u> | 0.888 | 0.892 |
37
+ | ruMATH | **0.776** | <u>0.771</u> | 0.724 | 0.589 |
38
+ | ruMBPP | **0.805** | <u>0.802</u> | 0.79 | 0.626 |
39
+ | [ruCodeEval](https://mera.a-ai.ru/ru/tasks/23) | 0.432 / 0.626 / 0.677 | <u>0.529 / 0.649 / 0.683</u> | **0.704 / 0.753 / 0.768** | 0.077 / 0.093 / 0.098 |
40
+ | Arena-Hard-Ru | **90.17** | <u>84.87</u> | 81 | - |
41
+ | MT Bench Ru | <u>8.7</u> | **8.706** | 8.45 | 8.53 |
42
+ | Alpaca Eval Ru | <u>47.61</u> | **50** | 45.51 | 38.13 |
43
+
44
+ Open-source models:
45
+
46
+ | Benchmark | T-pro-it-1.0 | Qwen-2.5-32B-Instruct | RuAdapt-Qwen-32B-Instruct-v1 | gemma-2-27b-it | Llama-3.3-70B-Instruct |
47
+ |------------------------------------------------|---------------------------|-------------------------------|------------------------------|------------------------------|------------------------|
48
+ | [MERA](https://mera.a-ai.ru) | **0.629** | 0.578 | <u>0.615</u> | 0.574 | 0.567 |
49
+ | [MaMuRaMu](https://mera.a-ai.ru/ru/tasks/22) | **0.841** | <u>0.824</u> | 0.812 | 0.768 | 0.818 |
50
+ | ruMMLU-PRO | **0.665** | 0.637 | 0.631 | 0.470 | <u>0.653</u> |
51
+ | ruGSM8K | **0.941** | 0.926 | 0.923 | 0.894 | <u>0.934</u> |
52
+ | ruMATH | **0.776** | 0.727 | <u>0.742</u> | 0.538 | 0.636 |
53
+ | ruMBPP | 0.805 | **0.825** | <u>0.813</u> | 0.708 | 0.77 |
54
+ | [ruCodeEval](https://mera.a-ai.ru/ru/tasks/23) | **0.432 / 0.626 / 0.677** | 0.06 / 0.098 / 0.116 | 0.426 / 0.561 / 0.598 | <u>0.259 / 0.586 / 0.689</u> | 0.112 / 0.166 / 0.189 |
55
+ | Arena-Hard-Ru | **90.17** | 74.54 | <u>80.23</u> | 66.4 | 76.51 |
56
+ | MT Bench Ru | **8.7** | 8.15 | 8.39 | 7.96 | <u>8.26</u> |
57
+ | Alpaca Eval Ru | **47.61** | 35.01 | <u>43.15</u> | 38.82 | - |
58
+
59
+
60
+
61
+ | Benchmark | T-pro-it-1.0 | GPT-4o | GPT-4o-mini | Qwen-2.5-32B-Instruct | GigaChat Max 1.0.26.20 | RuAdapt-Qwen-32B-Instruct-v1 | gemma-2-27b-it | Llama-3.3-70B-Instruct |
62
+ |------------------------------------------------|-----------------------|------------------------------|-----------------------|-----------------------|--------------------|------------------------------|-----------------------|------------------------|
63
+ | [MERA](https://mera.a-ai.ru) | <u>0.629</u> | **0.642** | 0.57 | 0.578 | 0.588 | 0.615 | 0.574 | 0.567 |
64
+ | [MaMuRaMu](https://mera.a-ai.ru/ru/tasks/22) | <u>0.841</u> | **0.874** | 0.779 | 0.824 | 0.824 | 0.812 | 0.768 | 0.818 |
65
+ | ruMMLU-PRO | <u>0.665</u> | **0.713** | 0.573 | 0.637 | 0.535 | 0.631 | 0.470 | 0.653 |
66
+ | ruGSM8K | **0.941** | 0.931 | 0.888 | 0.926 | 0.892 | 0.923 | 0.894 | <u>0.934</u> |
67
+ | ruMATH | **0.776** | <u>0.771</u> | 0.724 | 0.727 | 0.589 | 0.742 | 0.538 | 0.636 |
68
+ | ruMBPP | 0.805 | 0.802 | 0.79 | **0.825** | 0.626 | <u>0.813</u> | 0.708 | 0.77 |
69
+ | [ruCodeEval](https://mera.a-ai.ru/ru/tasks/23) | 0.432 / 0.626 / 0.677 | <u>0.529 / 0.649 / 0.683</u> | **0.704 / 0.753 / 0.768** | 0.06 / 0.098 / 0.116 | 0.077 / 0.093 / 0.098 | 0.426 / 0.561 / 0.598 | 0.259 / 0.586 / 0.689 | 0.112 / 0.166 / 0.189 |
70
+ | Arena-Hard-Ru | **90.17** | <u>84.87</u> | 81 | 74.54 | - | 80.23 | 66.4 | 76.51 |
71
+ | MT Bench Ru | <u>8.7</u> | **8.706** | 8.45 | 8.15 | 8.53 | 8.39 | 7.96 | 8.26 |
72
+ | Alpaca Eval Ru | <u>47.61</u> | **50** | 45.51 | 35.01 | 38.13 | 43.15 | 38.82 | - |
73
+
74
+ Detailed evaluation results can be found in our [habr post](https://habr.com/ru/companies/tbank/articles/865582/)
75
 
76
  ## 👨‍💻 Examples of usage
77
 
78
+ ### HF Usage
79
 
80
  ```python
81
  from transformers import AutoTokenizer, AutoModelForCausalLM
 
131
  Поиск закономерностей — его цель, открыть тайны бытия.
132
  От распознавания лиц до понимания речи,
133
  Машинное обучение — это ключ, что открывает двери.
134
+ ```
135
+
136
+ ### VLLM Usage
137
+
138
+ ```python
139
+ from transformers import AutoTokenizer
140
+ from vllm import LLM, SamplingParams
141
+
142
+ model_name = "t-tech/T-pro-it-1.0"
143
+ tokenizer = AutoTokenizer.from_pretrained(model_name)
144
+ llm = LLM(model=model_name)
145
+ sampling_params = SamplingParams(temperature=0.3, max_tokens=8192)
146
+
147
+ prompt = "Напиши стих про машинное обучение"
148
+ messages = [
149
+ {"role": "system", "content": "Ты T-pro, виртуальный ассистент в Т-Технологии. Твоя задача - быть полезным диалоговым ассистентом."},
150
+ {"role": "user", "content": prompt}
151
+ ]
152
+
153
+ prompt_token_ids = tokenizer.apply_chat_template(messages, add_generation_prompt=True)
154
+
155
+ outputs = llm.generate(prompt_token_ids=prompt_token_ids, sampling_params=sampling_params)
156
+
157
+ generated_text = [output.outputs[0].text for output in outputs]
158
+ print(generated_text)
159
  ```