| INFO: 2024-10-17 21:30:14,019: llmtf.base.evaluator: Starting eval on ['darumeru/multiq'] | |
| INFO: 2024-10-17 21:30:14,019: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [147077] | |
| INFO: 2024-10-17 21:30:14,019: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['<|im_end|>'] | |
| INFO: 2024-10-17 21:30:20,481: llmtf.base.darumeru/MultiQ: Loading Dataset: 6.46s | |
| INFO: 2024-10-17 21:35:59,593: llmtf.base.darumeru/MultiQ: Processing Dataset: 339.11s | |
| INFO: 2024-10-17 21:35:59,593: llmtf.base.darumeru/MultiQ: Results for darumeru/MultiQ: | |
| INFO: 2024-10-17 21:35:59,594: llmtf.base.darumeru/MultiQ: {'f1': 0.3346248767848689, 'em': 0.22275334608030592} | |
| INFO: 2024-10-17 21:35:59,599: llmtf.base.evaluator: Ended eval | |
| INFO: 2024-10-17 21:35:59,599: llmtf.base.evaluator: | |
| mean darumeru/MultiQ | |
| 0.279 0.279 | |
| INFO: 2024-10-17 21:36:08,809: llmtf.base.evaluator: Starting eval on ['darumeru/parus'] | |
| INFO: 2024-10-17 21:36:08,810: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [147077] | |
| INFO: 2024-10-17 21:36:08,810: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['<|im_end|>'] | |
| INFO: 2024-10-17 21:36:12,969: llmtf.base.darumeru/PARus: Loading Dataset: 4.16s | |
| INFO: 2024-10-17 21:36:18,316: llmtf.base.darumeru/PARus: Processing Dataset: 5.35s | |
| INFO: 2024-10-17 21:36:18,317: llmtf.base.darumeru/PARus: Results for darumeru/PARus: | |
| INFO: 2024-10-17 21:36:18,327: llmtf.base.darumeru/PARus: {'acc': 0.7} | |
| INFO: 2024-10-17 21:36:18,327: llmtf.base.evaluator: Ended eval | |
| INFO: 2024-10-17 21:36:18,328: llmtf.base.evaluator: | |
| mean darumeru/MultiQ darumeru/PARus | |
| 0.489 0.279 0.700 | |
| INFO: 2024-10-17 21:36:27,550: llmtf.base.evaluator: Starting eval on ['darumeru/rcb'] | |
| INFO: 2024-10-17 21:36:27,550: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [147077] | |
| INFO: 2024-10-17 21:36:27,551: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['<|im_end|>'] | |
| INFO: 2024-10-17 21:36:31,450: llmtf.base.darumeru/RCB: Loading Dataset: 3.90s | |
| INFO: 2024-10-17 21:36:38,683: llmtf.base.darumeru/RCB: Processing Dataset: 7.23s | |
| INFO: 2024-10-17 21:36:38,683: llmtf.base.darumeru/RCB: Results for darumeru/RCB: | |
| INFO: 2024-10-17 21:36:38,686: llmtf.base.darumeru/RCB: {'acc': 0.5454545454545454, 'f1_macro': 0.49090309951702227} | |
| INFO: 2024-10-17 21:36:38,687: llmtf.base.evaluator: Ended eval | |
| INFO: 2024-10-17 21:36:38,688: llmtf.base.evaluator: | |
| mean darumeru/MultiQ darumeru/PARus darumeru/RCB | |
| 0.499 0.279 0.700 0.518 | |
| INFO: 2024-10-17 21:36:48,734: llmtf.base.evaluator: Starting eval on ['darumeru/ruopenbookqa'] | |
| INFO: 2024-10-17 21:36:48,735: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [147077] | |
| INFO: 2024-10-17 21:36:48,735: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['<|im_end|>'] | |
| INFO: 2024-10-17 21:36:54,900: llmtf.base.darumeru/ruOpenBookQA: Loading Dataset: 6.17s | |
| INFO: 2024-10-17 21:38:00,519: llmtf.base.darumeru/ruOpenBookQA: Processing Dataset: 65.62s | |
| INFO: 2024-10-17 21:38:00,520: llmtf.base.darumeru/ruOpenBookQA: Results for darumeru/ruOpenBookQA: | |
| INFO: 2024-10-17 21:38:00,532: llmtf.base.darumeru/ruOpenBookQA: {'acc': 0.7302405498281787, 'f1_macro': 0.7304546157096631} | |
| INFO: 2024-10-17 21:38:00,541: llmtf.base.evaluator: Ended eval | |
| INFO: 2024-10-17 21:38:00,542: llmtf.base.evaluator: | |
| mean darumeru/MultiQ darumeru/PARus darumeru/RCB darumeru/ruOpenBookQA | |
| 0.557 0.279 0.700 0.518 0.730 | |
| INFO: 2024-10-17 21:38:09,745: llmtf.base.evaluator: Starting eval on ['darumeru/ruworldtree'] | |
| INFO: 2024-10-17 21:38:09,745: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [147077] | |
| INFO: 2024-10-17 21:38:09,745: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['<|im_end|>'] | |
| INFO: 2024-10-17 21:38:14,102: llmtf.base.darumeru/ruWorldTree: Loading Dataset: 4.36s | |
| INFO: 2024-10-17 21:38:16,932: llmtf.base.darumeru/ruWorldTree: Processing Dataset: 2.83s | |
| INFO: 2024-10-17 21:38:16,933: llmtf.base.darumeru/ruWorldTree: Results for darumeru/ruWorldTree: | |
| INFO: 2024-10-17 21:38:16,936: llmtf.base.darumeru/ruWorldTree: {'acc': 0.9047619047619048, 'f1_macro': 0.9043404138496471} | |
| INFO: 2024-10-17 21:38:16,936: llmtf.base.evaluator: Ended eval | |
| INFO: 2024-10-17 21:38:16,937: llmtf.base.evaluator: | |
| mean darumeru/MultiQ darumeru/PARus darumeru/RCB darumeru/ruOpenBookQA darumeru/ruWorldTree | |
| 0.626 0.279 0.700 0.518 0.730 0.905 | |
| INFO: 2024-10-17 21:38:26,077: llmtf.base.evaluator: Starting eval on ['darumeru/rwsd'] | |
| INFO: 2024-10-17 21:38:26,077: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [147077] | |
| INFO: 2024-10-17 21:38:26,077: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['<|im_end|>'] | |
| INFO: 2024-10-17 21:38:30,781: llmtf.base.darumeru/RWSD: Loading Dataset: 4.70s | |
| INFO: 2024-10-17 21:38:36,497: llmtf.base.darumeru/RWSD: Processing Dataset: 5.72s | |
| INFO: 2024-10-17 21:38:36,497: llmtf.base.darumeru/RWSD: Results for darumeru/RWSD: | |
| INFO: 2024-10-17 21:38:36,498: llmtf.base.darumeru/RWSD: {'acc': 0.6029411764705882} | |
| INFO: 2024-10-17 21:38:36,499: llmtf.base.evaluator: Ended eval | |
| INFO: 2024-10-17 21:38:36,500: llmtf.base.evaluator: | |
| mean darumeru/MultiQ darumeru/PARus darumeru/RCB darumeru/RWSD darumeru/ruOpenBookQA darumeru/ruWorldTree | |
| 0.622 0.279 0.700 0.518 0.603 0.730 0.905 | |
| INFO: 2024-10-17 21:38:45,688: llmtf.base.evaluator: Starting eval on ['daru/treewayextractive'] | |
| INFO: 2024-10-17 21:38:45,688: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [147077] | |
| INFO: 2024-10-17 21:38:45,688: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['<|im_end|>'] | |
| INFO: 2024-10-17 21:39:02,002: llmtf.base.daru/treewayextractive: Loading Dataset: 16.31s | |
| INFO: 2024-10-17 21:42:05,777: llmtf.base.daru/treewayextractive: Processing Dataset: 183.77s | |
| INFO: 2024-10-17 21:42:05,777: llmtf.base.daru/treewayextractive: Results for daru/treewayextractive: | |
| INFO: 2024-10-17 21:42:06,010: llmtf.base.daru/treewayextractive: {'r-prec': 0.3917218614718615} | |
| INFO: 2024-10-17 21:42:06,052: llmtf.base.evaluator: Ended eval | |
| INFO: 2024-10-17 21:42:06,054: llmtf.base.evaluator: | |
| mean daru/treewayextractive darumeru/MultiQ darumeru/PARus darumeru/RCB darumeru/RWSD darumeru/ruOpenBookQA darumeru/ruWorldTree | |
| 0.589 0.392 0.279 0.700 0.518 0.603 0.730 0.905 | |
| INFO: 2024-10-17 21:42:15,170: llmtf.base.evaluator: Starting eval on ['nlpcoreteam/rummlu'] | |
| INFO: 2024-10-17 21:42:15,170: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [147077] | |
| INFO: 2024-10-17 21:42:15,170: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['<|im_end|>'] | |
| INFO: 2024-10-17 21:46:47,282: llmtf.base.nlpcoreteam/ruMMLU: Loading Dataset: 272.11s | |
| INFO: 2024-10-17 21:56:29,398: llmtf.base.nlpcoreteam/ruMMLU: Processing Dataset: 582.12s | |
| INFO: 2024-10-17 21:56:29,399: llmtf.base.nlpcoreteam/ruMMLU: Results for nlpcoreteam/ruMMLU: | |
| INFO: 2024-10-17 21:56:29,464: llmtf.base.nlpcoreteam/ruMMLU: metric | |
| subject | |
| abstract_algebra 0.340000 | |
| anatomy 0.414815 | |
| astronomy 0.611842 | |
| business_ethics 0.610000 | |
| clinical_knowledge 0.554717 | |
| college_biology 0.548611 | |
| college_chemistry 0.380000 | |
| college_computer_science 0.450000 | |
| college_mathematics 0.400000 | |
| college_medicine 0.526012 | |
| college_physics 0.470588 | |
| computer_security 0.620000 | |
| conceptual_physics 0.565957 | |
| econometrics 0.377193 | |
| electrical_engineering 0.537931 | |
| elementary_mathematics 0.529101 | |
| formal_logic 0.365079 | |
| global_facts 0.360000 | |
| high_school_biology 0.664516 | |
| high_school_chemistry 0.487685 | |
| high_school_computer_science 0.700000 | |
| high_school_european_history 0.751515 | |
| high_school_geography 0.722222 | |
| high_school_government_and_politics 0.564767 | |
| high_school_macroeconomics 0.528205 | |
| high_school_mathematics 0.433333 | |
| high_school_microeconomics 0.533613 | |
| high_school_physics 0.403974 | |
| high_school_psychology 0.713761 | |
| high_school_statistics 0.523148 | |
| high_school_us_history 0.661765 | |
| high_school_world_history 0.717300 | |
| human_aging 0.587444 | |
| human_sexuality 0.618321 | |
| international_law 0.735537 | |
| jurisprudence 0.666667 | |
| logical_fallacies 0.564417 | |
| machine_learning 0.392857 | |
| management 0.650485 | |
| marketing 0.752137 | |
| medical_genetics 0.580000 | |
| miscellaneous 0.632184 | |
| moral_disputes 0.583815 | |
| moral_scenarios 0.299441 | |
| nutrition 0.637255 | |
| philosophy 0.617363 | |
| prehistory 0.561728 | |
| professional_accounting 0.386525 | |
| professional_law 0.377445 | |
| professional_medicine 0.481618 | |
| professional_psychology 0.516340 | |
| public_relations 0.500000 | |
| security_studies 0.648980 | |
| sociology 0.756219 | |
| us_foreign_policy 0.720000 | |
| virology 0.439759 | |
| world_religions 0.719298 | |
| INFO: 2024-10-17 21:56:29,473: llmtf.base.nlpcoreteam/ruMMLU: metric | |
| subject | |
| STEM 0.503308 | |
| humanities 0.586259 | |
| other (business, health, misc.) 0.543782 | |
| social sciences 0.599968 | |
| INFO: 2024-10-17 21:56:29,478: llmtf.base.nlpcoreteam/ruMMLU: {'acc': 0.5583294528508019} | |
| INFO: 2024-10-17 21:56:29,516: llmtf.base.evaluator: Ended eval | |
| INFO: 2024-10-17 21:56:29,518: llmtf.base.evaluator: | |
| mean daru/treewayextractive darumeru/MultiQ darumeru/PARus darumeru/RCB darumeru/RWSD darumeru/ruOpenBookQA darumeru/ruWorldTree nlpcoreteam/ruMMLU | |
| 0.586 0.392 0.279 0.700 0.518 0.603 0.730 0.905 0.558 | |
| INFO: 2024-10-17 21:56:39,535: llmtf.base.evaluator: Starting eval on ['nlpcoreteam/enmmlu'] | |
| INFO: 2024-10-17 21:56:39,536: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [147077] | |
| INFO: 2024-10-17 21:56:39,536: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['<|im_end|>'] | |
| INFO: 2024-10-17 21:58:54,966: llmtf.base.nlpcoreteam/enMMLU: Loading Dataset: 135.43s | |
| INFO: 2024-10-17 22:08:04,419: llmtf.base.nlpcoreteam/enMMLU: Processing Dataset: 549.45s | |
| INFO: 2024-10-17 22:08:04,426: llmtf.base.nlpcoreteam/enMMLU: Results for nlpcoreteam/enMMLU: | |
| INFO: 2024-10-17 22:08:04,492: llmtf.base.nlpcoreteam/enMMLU: metric | |
| subject | |
| abstract_algebra 0.380000 | |
| anatomy 0.637037 | |
| astronomy 0.717105 | |
| business_ethics 0.700000 | |
| clinical_knowledge 0.705660 | |
| college_biology 0.715278 | |
| college_chemistry 0.470000 | |
| college_computer_science 0.580000 | |
| college_mathematics 0.330000 | |
| college_medicine 0.664740 | |
| college_physics 0.509804 | |
| computer_security 0.740000 | |
| conceptual_physics 0.642553 | |
| econometrics 0.508772 | |
| electrical_engineering 0.600000 | |
| elementary_mathematics 0.547619 | |
| formal_logic 0.412698 | |
| global_facts 0.360000 | |
| high_school_biology 0.783871 | |
| high_school_chemistry 0.581281 | |
| high_school_computer_science 0.710000 | |
| high_school_european_history 0.800000 | |
| high_school_geography 0.757576 | |
| high_school_government_and_politics 0.854922 | |
| high_school_macroeconomics 0.679487 | |
| high_school_mathematics 0.455556 | |
| high_school_microeconomics 0.773109 | |
| high_school_physics 0.437086 | |
| high_school_psychology 0.844037 | |
| high_school_statistics 0.652778 | |
| high_school_us_history 0.833333 | |
| high_school_world_history 0.843882 | |
| human_aging 0.677130 | |
| human_sexuality 0.786260 | |
| international_law 0.768595 | |
| jurisprudence 0.814815 | |
| logical_fallacies 0.803681 | |
| machine_learning 0.446429 | |
| management 0.786408 | |
| marketing 0.858974 | |
| medical_genetics 0.760000 | |
| miscellaneous 0.795658 | |
| moral_disputes 0.667630 | |
| moral_scenarios 0.311732 | |
| nutrition 0.732026 | |
| philosophy 0.704180 | |
| prehistory 0.712963 | |
| professional_accounting 0.503546 | |
| professional_law 0.457627 | |
| professional_medicine 0.658088 | |
| professional_psychology 0.668301 | |
| public_relations 0.709091 | |
| security_studies 0.697959 | |
| sociology 0.800995 | |
| us_foreign_policy 0.800000 | |
| virology 0.506024 | |
| world_religions 0.801170 | |
| INFO: 2024-10-17 22:08:04,506: llmtf.base.nlpcoreteam/enMMLU: metric | |
| subject | |
| STEM 0.572187 | |
| humanities 0.687100 | |
| other (business, health, misc.) 0.667521 | |
| social sciences 0.740042 | |
| INFO: 2024-10-17 22:08:04,511: llmtf.base.nlpcoreteam/enMMLU: {'acc': 0.6667125709237595} | |
| INFO: 2024-10-17 22:08:04,554: llmtf.base.evaluator: Ended eval | |
| INFO: 2024-10-17 22:08:04,556: llmtf.base.evaluator: | |
| mean daru/treewayextractive darumeru/MultiQ darumeru/PARus darumeru/RCB darumeru/RWSD darumeru/ruOpenBookQA darumeru/ruWorldTree nlpcoreteam/enMMLU nlpcoreteam/ruMMLU | |
| 0.595 0.392 0.279 0.700 0.518 0.603 0.730 0.905 0.667 0.558 | |
| INFO: 2024-10-17 22:08:14,512: llmtf.base.evaluator: Starting eval on ['daru/treewayabstractive'] | |
| INFO: 2024-10-17 22:08:14,513: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [147077] | |
| INFO: 2024-10-17 22:08:14,513: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['<|im_end|>'] | |
| INFO: 2024-10-17 22:08:18,791: llmtf.base.daru/treewayabstractive: Loading Dataset: 4.28s | |
| INFO: 2024-10-17 22:11:46,260: llmtf.base.daru/treewayabstractive: Processing Dataset: 207.47s | |
| INFO: 2024-10-17 22:11:46,260: llmtf.base.daru/treewayabstractive: Results for daru/treewayabstractive: | |
| INFO: 2024-10-17 22:11:46,261: llmtf.base.daru/treewayabstractive: {'rouge1': 0.33109987599556284, 'rouge2': 0.11202889150257295} | |
| INFO: 2024-10-17 22:11:46,262: llmtf.base.evaluator: Ended eval | |
| INFO: 2024-10-17 22:11:46,263: llmtf.base.evaluator: | |
| mean daru/treewayabstractive daru/treewayextractive darumeru/MultiQ darumeru/PARus darumeru/RCB darumeru/RWSD darumeru/ruOpenBookQA darumeru/ruWorldTree nlpcoreteam/enMMLU nlpcoreteam/ruMMLU | |
| 0.557 0.222 0.392 0.279 0.700 0.518 0.603 0.730 0.905 0.667 0.558 | |
| INFO: 2024-10-17 22:11:55,717: llmtf.base.evaluator: Starting eval on ['darumeru/cp_para_ru'] | |
| INFO: 2024-10-17 22:11:55,717: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [147077] | |
| INFO: 2024-10-17 22:11:55,717: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['<|im_end|>'] | |
| INFO: 2024-10-17 22:11:59,846: llmtf.base.darumeru/cp_para_ru: Loading Dataset: 4.13s | |
| INFO: 2024-10-17 22:14:29,975: llmtf.base.darumeru/cp_para_ru: Processing Dataset: 150.13s | |
| INFO: 2024-10-17 22:14:29,975: llmtf.base.darumeru/cp_para_ru: Results for darumeru/cp_para_ru: | |
| INFO: 2024-10-17 22:14:29,976: llmtf.base.darumeru/cp_para_ru: {'symbol_per_token': 3.993754090002875, 'len': 0.9986883734384026, 'lcs': 0.98} | |
| INFO: 2024-10-17 22:14:29,977: llmtf.base.evaluator: Ended eval | |
| INFO: 2024-10-17 22:14:29,977: llmtf.base.evaluator: | |
| mean daru/treewayabstractive daru/treewayextractive darumeru/MultiQ darumeru/PARus darumeru/RCB darumeru/RWSD darumeru/cp_para_ru darumeru/ruOpenBookQA darumeru/ruWorldTree nlpcoreteam/enMMLU nlpcoreteam/ruMMLU | |
| 0.596 0.222 0.392 0.279 0.700 0.518 0.603 0.980 0.730 0.905 0.667 0.558 | |