| INFO: 2024-11-18 14:18:48,851: llmtf.base.evaluator: Starting eval on ['darumeru/multiq'] | |
| INFO: 2024-11-18 14:18:48,852: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [145111] | |
| INFO: 2024-11-18 14:18:48,852: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['<|im_end|>'] | |
| INFO: 2024-11-18 14:18:50,653: llmtf.base.evaluator: Starting eval on ['darumeru/parus'] | |
| INFO: 2024-11-18 14:18:50,654: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [145111] | |
| INFO: 2024-11-18 14:18:50,654: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['<|im_end|>'] | |
| INFO: 2024-11-18 14:18:52,696: llmtf.base.darumeru/MultiQ: Loading Dataset: 3.84s | |
| INFO: 2024-11-18 14:18:52,936: llmtf.base.evaluator: Starting eval on ['darumeru/rcb'] | |
| INFO: 2024-11-18 14:18:52,936: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [145111] | |
| INFO: 2024-11-18 14:18:52,936: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['<|im_end|>'] | |
| INFO: 2024-11-18 14:18:53,102: llmtf.base.darumeru/PARus: Loading Dataset: 2.45s | |
| INFO: 2024-11-18 14:18:54,811: llmtf.base.evaluator: Starting eval on ['darumeru/ruopenbookqa'] | |
| INFO: 2024-11-18 14:18:54,811: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [145111] | |
| INFO: 2024-11-18 14:18:54,811: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['<|im_end|>'] | |
| INFO: 2024-11-18 14:18:55,708: llmtf.base.darumeru/RCB: Loading Dataset: 2.77s | |
| INFO: 2024-11-18 14:18:56,455: llmtf.base.darumeru/PARus: Processing Dataset: 3.35s | |
| INFO: 2024-11-18 14:18:56,457: llmtf.base.darumeru/PARus: Results for darumeru/PARus: | |
| INFO: 2024-11-18 14:18:56,470: llmtf.base.darumeru/PARus: {'acc': 0.24} | |
| INFO: 2024-11-18 14:18:56,471: llmtf.base.evaluator: Ended eval | |
| INFO: 2024-11-18 14:18:56,474: llmtf.base.evaluator: | |
| mean darumeru/PARus | |
| 0.240 0.240 | |
| INFO: 2024-11-18 14:18:56,487: llmtf.base.evaluator: Starting eval on ['darumeru/ruworldtree'] | |
| INFO: 2024-11-18 14:18:56,488: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [145111] | |
| INFO: 2024-11-18 14:18:56,488: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['<|im_end|>'] | |
| INFO: 2024-11-18 14:18:58,099: llmtf.base.darumeru/ruOpenBookQA: Loading Dataset: 3.29s | |
| INFO: 2024-11-18 14:18:58,743: llmtf.base.evaluator: Starting eval on ['darumeru/rwsd'] | |
| INFO: 2024-11-18 14:18:58,744: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [145111] | |
| INFO: 2024-11-18 14:18:58,744: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['<|im_end|>'] | |
| INFO: 2024-11-18 14:18:58,925: llmtf.base.darumeru/ruWorldTree: Loading Dataset: 2.44s | |
| INFO: 2024-11-18 14:19:00,968: llmtf.base.evaluator: Starting eval on ['daru/treewayextractive'] | |
| INFO: 2024-11-18 14:19:00,968: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [145111] | |
| INFO: 2024-11-18 14:19:00,968: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['<|im_end|>'] | |
| INFO: 2024-11-18 14:19:01,031: llmtf.base.darumeru/RCB: Processing Dataset: 5.32s | |
| INFO: 2024-11-18 14:19:01,033: llmtf.base.darumeru/RCB: Results for darumeru/RCB: | |
| INFO: 2024-11-18 14:19:01,040: llmtf.base.darumeru/RCB: {'acc': 0.4727272727272727, 'f1_macro': 0.39356669305497743} | |
| INFO: 2024-11-18 14:19:01,041: llmtf.base.evaluator: Ended eval | |
| INFO: 2024-11-18 14:19:01,044: llmtf.base.evaluator: | |
| mean darumeru/PARus darumeru/RCB | |
| 0.337 0.240 0.433 | |
| INFO: 2024-11-18 14:19:01,497: llmtf.base.darumeru/RWSD: Loading Dataset: 2.75s | |
| INFO: 2024-11-18 14:19:01,851: llmtf.base.darumeru/ruWorldTree: Processing Dataset: 2.92s | |
| INFO: 2024-11-18 14:19:01,852: llmtf.base.darumeru/ruWorldTree: Results for darumeru/ruWorldTree: | |
| INFO: 2024-11-18 14:19:01,859: llmtf.base.darumeru/ruWorldTree: {'acc': 0.7714285714285715, 'f1_macro': 0.7726851851851853} | |
| INFO: 2024-11-18 14:19:01,859: llmtf.base.evaluator: Ended eval | |
| INFO: 2024-11-18 14:19:01,863: llmtf.base.evaluator: | |
| mean darumeru/PARus darumeru/RCB darumeru/ruWorldTree | |
| 0.482 0.240 0.433 0.772 | |
| INFO: 2024-11-18 14:19:02,889: llmtf.base.evaluator: Starting eval on ['daru/treewayabstractive'] | |
| INFO: 2024-11-18 14:19:02,890: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [145111] | |
| INFO: 2024-11-18 14:19:02,890: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['<|im_end|>'] | |
| INFO: 2024-11-18 14:19:03,199: llmtf.base.evaluator: Starting eval on ['nlpcoreteam/rummlu'] | |
| INFO: 2024-11-18 14:19:03,199: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [145111] | |
| INFO: 2024-11-18 14:19:03,199: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['<|im_end|>'] | |
| INFO: 2024-11-18 14:19:06,629: llmtf.base.darumeru/RWSD: Processing Dataset: 5.13s | |
| INFO: 2024-11-18 14:19:06,631: llmtf.base.darumeru/RWSD: Results for darumeru/RWSD: | |
| INFO: 2024-11-18 14:19:06,635: llmtf.base.darumeru/RWSD: {'acc': 0.5098039215686274} | |
| INFO: 2024-11-18 14:19:06,636: llmtf.base.evaluator: Ended eval | |
| INFO: 2024-11-18 14:19:06,641: llmtf.base.evaluator: | |
| mean darumeru/PARus darumeru/RCB darumeru/RWSD darumeru/ruWorldTree | |
| 0.489 0.240 0.433 0.510 0.772 | |
| INFO: 2024-11-18 14:19:06,885: llmtf.base.daru/treewayabstractive: Loading Dataset: 4.00s | |
| INFO: 2024-11-18 14:19:07,496: llmtf.base.evaluator: Starting eval on ['darumeru/cp_para_ru'] | |
| INFO: 2024-11-18 14:19:07,497: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [145111] | |
| INFO: 2024-11-18 14:19:07,497: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['<|im_end|>'] | |
| INFO: 2024-11-18 14:19:10,509: llmtf.base.darumeru/cp_para_ru: Loading Dataset: 3.01s | |
| INFO: 2024-11-18 14:19:13,909: llmtf.base.daru/treewayextractive: Loading Dataset: 12.94s | |
| INFO: 2024-11-18 14:19:44,800: llmtf.base.darumeru/ruOpenBookQA: Processing Dataset: 46.70s | |
| INFO: 2024-11-18 14:19:44,801: llmtf.base.darumeru/ruOpenBookQA: Results for darumeru/ruOpenBookQA: | |
| INFO: 2024-11-18 14:19:44,814: llmtf.base.darumeru/ruOpenBookQA: {'acc': 0.615979381443299, 'f1_macro': 0.6154023944317246} | |
| INFO: 2024-11-18 14:19:44,821: llmtf.base.evaluator: Ended eval | |
| INFO: 2024-11-18 14:19:44,826: llmtf.base.evaluator: | |
| mean darumeru/PARus darumeru/RCB darumeru/RWSD darumeru/ruOpenBookQA darumeru/ruWorldTree | |
| 0.514 0.240 0.433 0.510 0.616 0.772 | |
| INFO: 2024-11-18 14:21:05,969: llmtf.base.nlpcoreteam/ruMMLU: Loading Dataset: 122.77s | |
| INFO: 2024-11-18 14:21:35,520: llmtf.base.daru/treewayextractive: Processing Dataset: 141.61s | |
| INFO: 2024-11-18 14:21:35,523: llmtf.base.daru/treewayextractive: Results for daru/treewayextractive: | |
| INFO: 2024-11-18 14:21:35,753: llmtf.base.daru/treewayextractive: {'r-prec': 0.3782488455988456} | |
| INFO: 2024-11-18 14:21:35,793: llmtf.base.evaluator: Ended eval | |
| INFO: 2024-11-18 14:21:35,799: llmtf.base.evaluator: | |
| mean daru/treewayextractive darumeru/PARus darumeru/RCB darumeru/RWSD darumeru/ruOpenBookQA darumeru/ruWorldTree | |
| 0.491 0.378 0.240 0.433 0.510 0.616 0.772 | |
| INFO: 2024-11-18 14:24:06,030: llmtf.base.darumeru/cp_para_ru: Processing Dataset: 295.52s | |
| INFO: 2024-11-18 14:24:06,032: llmtf.base.darumeru/cp_para_ru: Results for darumeru/cp_para_ru: | |
| INFO: 2024-11-18 14:24:06,036: llmtf.base.darumeru/cp_para_ru: {'symbol_per_token': 3.8978256135744003, 'len': 0.8764597602663033, 'lcs': 0.05} | |
| INFO: 2024-11-18 14:24:06,036: llmtf.base.evaluator: Ended eval | |
| INFO: 2024-11-18 14:24:06,041: llmtf.base.evaluator: | |
| mean daru/treewayextractive darumeru/PARus darumeru/RCB darumeru/RWSD darumeru/cp_para_ru darumeru/ruOpenBookQA darumeru/ruWorldTree | |
| 0.428 0.378 0.240 0.433 0.510 0.050 0.616 0.772 | |
| INFO: 2024-11-18 14:24:17,174: llmtf.base.daru/treewayabstractive: Processing Dataset: 310.29s | |
| INFO: 2024-11-18 14:24:17,190: llmtf.base.daru/treewayabstractive: Results for daru/treewayabstractive: | |
| INFO: 2024-11-18 14:24:17,208: llmtf.base.daru/treewayabstractive: {'rouge1': 0.31023763628891676, 'rouge2': 0.09443696323171702} | |
| INFO: 2024-11-18 14:24:17,210: llmtf.base.evaluator: Ended eval | |
| INFO: 2024-11-18 14:24:17,215: llmtf.base.evaluator: | |
| mean daru/treewayabstractive daru/treewayextractive darumeru/PARus darumeru/RCB darumeru/RWSD darumeru/cp_para_ru darumeru/ruOpenBookQA darumeru/ruWorldTree | |
| 0.400 0.202 0.378 0.240 0.433 0.510 0.050 0.616 0.772 | |
| INFO: 2024-11-18 14:26:06,991: llmtf.base.nlpcoreteam/ruMMLU: Processing Dataset: 301.02s | |
| INFO: 2024-11-18 14:26:06,993: llmtf.base.nlpcoreteam/ruMMLU: Results for nlpcoreteam/ruMMLU: | |
| INFO: 2024-11-18 14:26:07,037: llmtf.base.nlpcoreteam/ruMMLU: metric | |
| subject | |
| abstract_algebra 0.280000 | |
| anatomy 0.400000 | |
| astronomy 0.572368 | |
| business_ethics 0.460000 | |
| clinical_knowledge 0.494340 | |
| college_biology 0.375000 | |
| college_chemistry 0.290000 | |
| college_computer_science 0.400000 | |
| college_mathematics 0.400000 | |
| college_medicine 0.491329 | |
| college_physics 0.362745 | |
| computer_security 0.500000 | |
| conceptual_physics 0.421277 | |
| econometrics 0.280702 | |
| electrical_engineering 0.427586 | |
| elementary_mathematics 0.391534 | |
| formal_logic 0.373016 | |
| global_facts 0.230000 | |
| high_school_biology 0.496774 | |
| high_school_chemistry 0.458128 | |
| high_school_computer_science 0.500000 | |
| high_school_european_history 0.600000 | |
| high_school_geography 0.535354 | |
| high_school_government_and_politics 0.518135 | |
| high_school_macroeconomics 0.471795 | |
| high_school_mathematics 0.400000 | |
| high_school_microeconomics 0.462185 | |
| high_school_physics 0.291391 | |
| high_school_psychology 0.614679 | |
| high_school_statistics 0.490741 | |
| high_school_us_history 0.534314 | |
| high_school_world_history 0.624473 | |
| human_aging 0.520179 | |
| human_sexuality 0.519084 | |
| international_law 0.694215 | |
| jurisprudence 0.537037 | |
| logical_fallacies 0.472393 | |
| machine_learning 0.258929 | |
| management 0.640777 | |
| marketing 0.700855 | |
| medical_genetics 0.480000 | |
| miscellaneous 0.533844 | |
| moral_disputes 0.488439 | |
| moral_scenarios 0.268156 | |
| nutrition 0.526144 | |
| philosophy 0.543408 | |
| prehistory 0.475309 | |
| professional_accounting 0.347518 | |
| professional_law 0.345502 | |
| professional_medicine 0.426471 | |
| professional_psychology 0.411765 | |
| public_relations 0.427273 | |
| security_studies 0.542857 | |
| sociology 0.686567 | |
| us_foreign_policy 0.700000 | |
| virology 0.379518 | |
| world_religions 0.538012 | |
| INFO: 2024-11-18 14:26:07,045: llmtf.base.nlpcoreteam/ruMMLU: metric | |
| subject | |
| STEM 0.406471 | |
| humanities 0.499559 | |
| other (business, health, misc.) 0.473641 | |
| social sciences 0.514200 | |
| INFO: 2024-11-18 14:26:07,053: llmtf.base.nlpcoreteam/ruMMLU: {'acc': 0.47346768524963356} | |
| INFO: 2024-11-18 14:26:07,087: llmtf.base.evaluator: Ended eval | |
| INFO: 2024-11-18 14:26:07,096: llmtf.base.evaluator: | |
| mean daru/treewayabstractive daru/treewayextractive darumeru/PARus darumeru/RCB darumeru/RWSD darumeru/cp_para_ru darumeru/ruOpenBookQA darumeru/ruWorldTree nlpcoreteam/ruMMLU | |
| 0.408 0.202 0.378 0.240 0.433 0.510 0.050 0.616 0.772 0.473 | |
| INFO: 2024-11-18 14:29:49,600: llmtf.base.darumeru/MultiQ: Processing Dataset: 656.90s | |
| INFO: 2024-11-18 14:29:49,603: llmtf.base.darumeru/MultiQ: Results for darumeru/MultiQ: | |
| INFO: 2024-11-18 14:29:49,608: llmtf.base.darumeru/MultiQ: {'f1': 0.20613243758223346, 'em': 0.11281070745697896} | |
| INFO: 2024-11-18 14:29:49,612: llmtf.base.evaluator: Ended eval | |
| INFO: 2024-11-18 14:29:49,634: llmtf.base.evaluator: | |
| mean daru/treewayabstractive daru/treewayextractive darumeru/MultiQ darumeru/PARus darumeru/RCB darumeru/RWSD darumeru/cp_para_ru darumeru/ruOpenBookQA darumeru/ruWorldTree nlpcoreteam/ruMMLU | |
| 0.383 0.202 0.378 0.159 0.240 0.433 0.510 0.050 0.616 0.772 0.473 | |
| INFO: 2024-11-18 14:29:55,578: llmtf.base.evaluator: Starting eval on ['nlpcoreteam/enmmlu'] | |
| INFO: 2024-11-18 14:29:55,579: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [145111] | |
| INFO: 2024-11-18 14:29:55,579: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['<|im_end|>'] | |
| INFO: 2024-11-18 14:31:56,928: llmtf.base.nlpcoreteam/enMMLU: Loading Dataset: 121.35s | |
| INFO: 2024-11-18 14:36:48,348: llmtf.base.nlpcoreteam/enMMLU: Processing Dataset: 291.42s | |
| INFO: 2024-11-18 14:36:48,352: llmtf.base.nlpcoreteam/enMMLU: Results for nlpcoreteam/enMMLU: | |
| INFO: 2024-11-18 14:36:48,396: llmtf.base.nlpcoreteam/enMMLU: metric | |
| subject | |
| abstract_algebra 0.350000 | |
| anatomy 0.562963 | |
| astronomy 0.664474 | |
| business_ethics 0.640000 | |
| clinical_knowledge 0.652830 | |
| college_biology 0.652778 | |
| college_chemistry 0.450000 | |
| college_computer_science 0.490000 | |
| college_mathematics 0.310000 | |
| college_medicine 0.618497 | |
| college_physics 0.500000 | |
| computer_security 0.710000 | |
| conceptual_physics 0.587234 | |
| econometrics 0.429825 | |
| electrical_engineering 0.531034 | |
| elementary_mathematics 0.460317 | |
| formal_logic 0.373016 | |
| global_facts 0.270000 | |
| high_school_biology 0.748387 | |
| high_school_chemistry 0.522167 | |
| high_school_computer_science 0.620000 | |
| high_school_european_history 0.733333 | |
| high_school_geography 0.747475 | |
| high_school_government_and_politics 0.808290 | |
| high_school_macroeconomics 0.658974 | |
| high_school_mathematics 0.403704 | |
| high_school_microeconomics 0.684874 | |
| high_school_physics 0.390728 | |
| high_school_psychology 0.822018 | |
| high_school_statistics 0.550926 | |
| high_school_us_history 0.720588 | |
| high_school_world_history 0.742616 | |
| human_aging 0.623318 | |
| human_sexuality 0.687023 | |
| international_law 0.710744 | |
| jurisprudence 0.759259 | |
| logical_fallacies 0.742331 | |
| machine_learning 0.419643 | |
| management 0.747573 | |
| marketing 0.824786 | |
| medical_genetics 0.690000 | |
| miscellaneous 0.708812 | |
| moral_disputes 0.641618 | |
| moral_scenarios 0.252514 | |
| nutrition 0.653595 | |
| philosophy 0.668810 | |
| prehistory 0.675926 | |
| professional_accounting 0.510638 | |
| professional_law 0.397653 | |
| professional_medicine 0.602941 | |
| professional_psychology 0.589869 | |
| public_relations 0.609091 | |
| security_studies 0.673469 | |
| sociology 0.776119 | |
| us_foreign_policy 0.740000 | |
| virology 0.463855 | |
| world_religions 0.783626 | |
| INFO: 2024-11-18 14:36:48,403: llmtf.base.nlpcoreteam/enMMLU: metric | |
| subject | |
| STEM 0.520077 | |
| humanities 0.630926 | |
| other (business, health, misc.) 0.612129 | |
| social sciences 0.685586 | |
| INFO: 2024-11-18 14:36:48,425: llmtf.base.nlpcoreteam/enMMLU: {'acc': 0.6121795307919802} | |
| INFO: 2024-11-18 14:36:48,459: llmtf.base.evaluator: Ended eval | |
| INFO: 2024-11-18 14:36:48,480: llmtf.base.evaluator: | |
| mean daru/treewayabstractive daru/treewayextractive darumeru/MultiQ darumeru/PARus darumeru/RCB darumeru/RWSD darumeru/cp_para_ru darumeru/ruOpenBookQA darumeru/ruWorldTree nlpcoreteam/enMMLU nlpcoreteam/ruMMLU | |
| 0.404 0.202 0.378 0.159 0.240 0.433 0.510 0.050 0.616 0.772 0.612 0.473 | |