| INFO: 2024-11-26 20:17:25,790: llmtf.base.evaluator: Starting eval on ['darumeru/multiq', 'darumeru/parus', 'darumeru/rcb', 'darumeru/rwsd', 'darumeru/use'] | |
| INFO: 2024-11-26 20:17:25,791: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [145111] | |
| INFO: 2024-11-26 20:17:25,791: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['<|im_end|>'] | |
| INFO: 2024-11-26 20:17:27,525: llmtf.base.evaluator: Starting eval on ['nlpcoreteam/rummlu'] | |
| INFO: 2024-11-26 20:17:27,525: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [145111] | |
| INFO: 2024-11-26 20:17:27,525: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['<|im_end|>'] | |
| INFO: 2024-11-26 20:17:29,517: llmtf.base.evaluator: Starting eval on ['nlpcoreteam/enmmlu'] | |
| INFO: 2024-11-26 20:17:29,517: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [145111] | |
| INFO: 2024-11-26 20:17:29,517: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['<|im_end|>'] | |
| INFO: 2024-11-26 20:17:30,060: llmtf.base.darumeru/MultiQ: Loading Dataset: 4.27s | |
| INFO: 2024-11-26 20:17:31,597: llmtf.base.evaluator: Starting eval on ['daru/treewayabstractive'] | |
| INFO: 2024-11-26 20:17:31,597: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [145111] | |
| INFO: 2024-11-26 20:17:31,597: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['<|im_end|>'] | |
| INFO: 2024-11-26 20:17:33,345: llmtf.base.evaluator: Starting eval on ['darumeru/cp_para_ru'] | |
| INFO: 2024-11-26 20:17:33,345: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [145111] | |
| INFO: 2024-11-26 20:17:33,345: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['<|im_end|>'] | |
| INFO: 2024-11-26 20:17:35,521: llmtf.base.evaluator: Starting eval on ['vikhrmodels/habr_qa_sbs', 'ruparam', 'shlepa/moviesmc', 'shlepa/musicmc', 'shlepa/lawmc', 'shlepa/booksmc'] | |
| INFO: 2024-11-26 20:17:35,521: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [145111] | |
| INFO: 2024-11-26 20:17:35,521: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['<|im_end|>'] | |
| INFO: 2024-11-26 20:17:36,308: llmtf.base.darumeru/cp_para_ru: Loading Dataset: 2.96s | |
| INFO: 2024-11-26 20:17:36,742: llmtf.base.daru/treewayabstractive: Loading Dataset: 5.14s | |
| INFO: 2024-11-26 20:17:37,659: llmtf.base.evaluator: Starting eval on ['ruopinionne'] | |
| INFO: 2024-11-26 20:17:37,659: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [145111] | |
| INFO: 2024-11-26 20:17:37,660: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['<|im_end|>'] | |
| INFO: 2024-11-26 20:17:37,984: llmtf.base.ruopinionne: Loading Dataset: 0.32s | |
| INFO: 2024-11-26 20:17:39,551: llmtf.base.evaluator: Starting eval on ['nerel'] | |
| INFO: 2024-11-26 20:17:39,551: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [145111] | |
| INFO: 2024-11-26 20:17:39,551: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['<|im_end|>'] | |
| INFO: 2024-11-26 20:17:43,423: llmtf.base.NEREL: Loading Dataset: 3.87s | |
| INFO: 2024-11-26 20:17:48,557: llmtf.base.vikhrmodels/habr_qa_sbs: Loading Dataset: 13.04s | |
| INFO: 2024-11-26 20:18:44,135: llmtf.base.darumeru/cp_para_ru: Processing Dataset: 67.83s | |
| INFO: 2024-11-26 20:18:44,145: llmtf.base.darumeru/cp_para_ru: Results for darumeru/cp_para_ru: | |
| INFO: 2024-11-26 20:18:44,149: llmtf.base.darumeru/cp_para_ru: {'tokens_per_word': 1.905314928817744, 'symbol_per_token': 3.913951487866651, 'len': 0.9904780330832407, 'lcs': 0.8} | |
| INFO: 2024-11-26 20:18:44,150: llmtf.base.evaluator: Ended eval | |
| INFO: 2024-11-26 20:18:44,152: llmtf.base.evaluator: | |
| mean darumeru/cp_para_ru | |
| 0.800 0.800 | |
| INFO: 2024-11-26 20:18:50,857: llmtf.base.NEREL: Processing Dataset: 67.43s | |
| INFO: 2024-11-26 20:18:50,860: llmtf.base.NEREL: Results for NEREL: | |
| INFO: 2024-11-26 20:18:50,864: llmtf.base.NEREL: {'tp': 2.0, 'fp': 27.0, 'fn': 519.0, 'micro-f1': 0.00727272727272595} | |
| INFO: 2024-11-26 20:18:50,865: llmtf.base.evaluator: Ended eval | |
| INFO: 2024-11-26 20:18:50,869: llmtf.base.evaluator: | |
| mean NEREL darumeru/cp_para_ru | |
| 0.404 0.007 0.800 | |
| INFO: 2024-11-26 20:18:57,627: llmtf.base.daru/treewayabstractive: Processing Dataset: 80.88s | |
| INFO: 2024-11-26 20:18:57,629: llmtf.base.daru/treewayabstractive: Results for daru/treewayabstractive: | |
| INFO: 2024-11-26 20:18:57,632: llmtf.base.daru/treewayabstractive: {'rouge1': 0.3138417117532064, 'rouge2': 0.10462617373556911} | |
| INFO: 2024-11-26 20:18:57,634: llmtf.base.evaluator: Ended eval | |
| INFO: 2024-11-26 20:18:57,637: llmtf.base.evaluator: | |
| mean NEREL daru/treewayabstractive darumeru/cp_para_ru | |
| 0.339 0.007 0.209 0.800 | |
| INFO: 2024-11-26 20:19:13,379: llmtf.base.darumeru/MultiQ: Processing Dataset: 103.32s | |
| INFO: 2024-11-26 20:19:13,384: llmtf.base.darumeru/MultiQ: Results for darumeru/MultiQ: | |
| INFO: 2024-11-26 20:19:13,404: llmtf.base.darumeru/MultiQ: {'f1': 0.3016876852043635, 'em': 0.21319311663479923} | |
| INFO: 2024-11-26 20:19:13,412: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [145111] | |
| INFO: 2024-11-26 20:19:13,412: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['<|im_end|>'] | |
| INFO: 2024-11-26 20:19:15,573: llmtf.base.darumeru/PARus: Loading Dataset: 2.16s | |
| INFO: 2024-11-26 20:19:19,861: llmtf.base.darumeru/PARus: Processing Dataset: 4.29s | |
| INFO: 2024-11-26 20:19:19,867: llmtf.base.darumeru/PARus: Results for darumeru/PARus: | |
| INFO: 2024-11-26 20:19:19,887: llmtf.base.darumeru/PARus: {'acc': 0.44} | |
| INFO: 2024-11-26 20:19:19,888: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [145111] | |
| INFO: 2024-11-26 20:19:19,888: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['<|im_end|>'] | |
| INFO: 2024-11-26 20:19:22,295: llmtf.base.darumeru/RCB: Loading Dataset: 2.40s | |
| INFO: 2024-11-26 20:19:23,604: llmtf.base.ruopinionne: Processing Dataset: 105.62s | |
| INFO: 2024-11-26 20:19:23,610: llmtf.base.ruopinionne: Results for ruopinionne: | |
| INFO: 2024-11-26 20:19:23,639: llmtf.base.ruopinionne: {'f1': 0.02701209922104298} | |
| INFO: 2024-11-26 20:19:23,640: llmtf.base.evaluator: Ended eval | |
| INFO: 2024-11-26 20:19:23,656: llmtf.base.evaluator: | |
| mean NEREL daru/treewayabstractive darumeru/MultiQ darumeru/PARus darumeru/cp_para_ru ruopinionne | |
| 0.290 0.007 0.209 0.257 0.440 0.800 0.027 | |
| INFO: 2024-11-26 20:19:27,714: llmtf.base.darumeru/RCB: Processing Dataset: 5.42s | |
| INFO: 2024-11-26 20:19:27,715: llmtf.base.darumeru/RCB: Results for darumeru/RCB: | |
| INFO: 2024-11-26 20:19:27,722: llmtf.base.darumeru/RCB: {'acc': 0.4590909090909091, 'f1_macro': 0.36910715356478985} | |
| INFO: 2024-11-26 20:19:27,724: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [145111] | |
| INFO: 2024-11-26 20:19:27,724: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['<|im_end|>'] | |
| INFO: 2024-11-26 20:19:29,690: llmtf.base.darumeru/RWSD: Loading Dataset: 1.96s | |
| INFO: 2024-11-26 20:19:34,850: llmtf.base.darumeru/RWSD: Processing Dataset: 5.16s | |
| INFO: 2024-11-26 20:19:34,855: llmtf.base.darumeru/RWSD: Results for darumeru/RWSD: | |
| INFO: 2024-11-26 20:19:34,858: llmtf.base.darumeru/RWSD: {'acc': 0.49019607843137253} | |
| INFO: 2024-11-26 20:19:34,859: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [145111] | |
| INFO: 2024-11-26 20:19:34,859: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['<|im_end|>'] | |
| INFO: 2024-11-26 20:19:38,002: llmtf.base.darumeru/USE: Loading Dataset: 3.14s | |
| INFO: 2024-11-26 20:19:52,797: llmtf.base.nlpcoreteam/enMMLU: Loading Dataset: 143.28s | |
| INFO: 2024-11-26 20:19:55,208: llmtf.base.nlpcoreteam/ruMMLU: Loading Dataset: 147.68s | |
| INFO: 2024-11-26 20:20:33,495: llmtf.base.vikhrmodels/habr_qa_sbs: Processing Dataset: 164.94s | |
| INFO: 2024-11-26 20:20:33,496: llmtf.base.vikhrmodels/habr_qa_sbs: Results for vikhrmodels/habr_qa_sbs: | |
| INFO: 2024-11-26 20:20:33,533: llmtf.base.vikhrmodels/habr_qa_sbs: {'acc': 0.547, 'f1_macro': 0.5280856541414141} | |
| INFO: 2024-11-26 20:20:33,546: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [145111] | |
| INFO: 2024-11-26 20:20:33,547: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['<|im_end|>'] | |
| INFO: 2024-11-26 20:20:40,434: llmtf.base.ruparam: Loading Dataset: 6.89s | |
| INFO: 2024-11-26 20:21:07,899: llmtf.base.darumeru/USE: Processing Dataset: 89.90s | |
| INFO: 2024-11-26 20:21:07,900: llmtf.base.darumeru/USE: Results for darumeru/USE: | |
| INFO: 2024-11-26 20:21:07,942: llmtf.base.darumeru/USE: {'grade_norm': 0.06078431372549018} | |
| INFO: 2024-11-26 20:21:07,948: llmtf.base.evaluator: Ended eval | |
| INFO: 2024-11-26 20:21:08,016: llmtf.base.evaluator: | |
| mean NEREL daru/treewayabstractive darumeru/MultiQ darumeru/PARus darumeru/RCB darumeru/RWSD darumeru/USE darumeru/cp_para_ru ruopinionne vikhrmodels/habr_qa_sbs | |
| 0.324 0.007 0.209 0.257 0.440 0.414 0.490 0.061 0.800 0.027 0.538 | |
| INFO: 2024-11-26 20:25:43,772: llmtf.base.ruparam: Processing Dataset: 303.34s | |
| INFO: 2024-11-26 20:25:43,788: llmtf.base.ruparam: Results for ruparam: | |
| INFO: 2024-11-26 20:25:44,038: llmtf.base.ruparam: {'acc': 0.21363220494053065} | |
| INFO: 2024-11-26 20:25:44,053: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [145111] | |
| INFO: 2024-11-26 20:25:44,053: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['<|im_end|>'] | |
| INFO: 2024-11-26 20:25:47,874: llmtf.base.shlepa/movie_mc: Loading Dataset: 3.82s | |
| INFO: 2024-11-26 20:26:05,103: llmtf.base.shlepa/movie_mc: Processing Dataset: 17.22s | |
| INFO: 2024-11-26 20:26:05,119: llmtf.base.shlepa/movie_mc: Results for shlepa/movie_mc: | |
| INFO: 2024-11-26 20:26:05,122: llmtf.base.shlepa/movie_mc: {'acc': 0.22453703703703703} | |
| INFO: 2024-11-26 20:26:05,130: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [145111] | |
| INFO: 2024-11-26 20:26:05,130: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['<|im_end|>'] | |
| INFO: 2024-11-26 20:26:08,736: llmtf.base.shlepa/music_mc: Loading Dataset: 3.60s | |
| INFO: 2024-11-26 20:26:26,413: llmtf.base.shlepa/music_mc: Processing Dataset: 17.68s | |
| INFO: 2024-11-26 20:26:26,416: llmtf.base.shlepa/music_mc: Results for shlepa/music_mc: | |
| INFO: 2024-11-26 20:26:26,435: llmtf.base.shlepa/music_mc: {'acc': 0.24468085106382978} | |
| INFO: 2024-11-26 20:26:26,438: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [145111] | |
| INFO: 2024-11-26 20:26:26,439: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['<|im_end|>'] | |
| INFO: 2024-11-26 20:26:31,058: llmtf.base.shlepa/law_mc: Loading Dataset: 4.62s | |
| INFO: 2024-11-26 20:27:15,973: llmtf.base.shlepa/law_mc: Processing Dataset: 44.91s | |
| INFO: 2024-11-26 20:27:15,980: llmtf.base.shlepa/law_mc: Results for shlepa/law_mc: | |
| INFO: 2024-11-26 20:27:16,000: llmtf.base.shlepa/law_mc: {'acc': 0.537590113285273} | |
| INFO: 2024-11-26 20:27:16,006: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [145111] | |
| INFO: 2024-11-26 20:27:16,006: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['<|im_end|>'] | |
| INFO: 2024-11-26 20:27:19,668: llmtf.base.shlepa/books_mc: Loading Dataset: 3.66s | |
| INFO: 2024-11-26 20:27:39,200: llmtf.base.shlepa/books_mc: Processing Dataset: 19.53s | |
| INFO: 2024-11-26 20:27:39,204: llmtf.base.shlepa/books_mc: Results for shlepa/books_mc: | |
| INFO: 2024-11-26 20:27:39,209: llmtf.base.shlepa/books_mc: {'acc': 0.3112033195020747} | |
| INFO: 2024-11-26 20:27:39,212: llmtf.base.evaluator: Ended eval | |
| INFO: 2024-11-26 20:27:39,231: llmtf.base.evaluator: | |
| mean NEREL daru/treewayabstractive darumeru/MultiQ darumeru/PARus darumeru/RCB darumeru/RWSD darumeru/USE darumeru/cp_para_ru ruopinionne ruparam shlepa/books_mc shlepa/law_mc shlepa/movie_mc shlepa/music_mc vikhrmodels/habr_qa_sbs | |
| 0.318 0.007 0.209 0.257 0.440 0.414 0.490 0.061 0.800 0.027 0.214 0.311 0.538 0.225 0.245 0.538 | |
| INFO: 2024-11-26 20:28:00,389: llmtf.base.nlpcoreteam/enMMLU: Processing Dataset: 487.59s | |
| INFO: 2024-11-26 20:28:00,396: llmtf.base.nlpcoreteam/enMMLU: Results for nlpcoreteam/enMMLU: | |
| INFO: 2024-11-26 20:28:00,445: llmtf.base.nlpcoreteam/enMMLU: metric | |
| subject | |
| abstract_algebra 0.310000 | |
| anatomy 0.518519 | |
| astronomy 0.717105 | |
| business_ethics 0.630000 | |
| clinical_knowledge 0.667925 | |
| college_biology 0.680556 | |
| college_chemistry 0.400000 | |
| college_computer_science 0.510000 | |
| college_mathematics 0.270000 | |
| college_medicine 0.618497 | |
| college_physics 0.558824 | |
| computer_security 0.750000 | |
| conceptual_physics 0.561702 | |
| econometrics 0.438596 | |
| electrical_engineering 0.579310 | |
| elementary_mathematics 0.455026 | |
| formal_logic 0.412698 | |
| global_facts 0.250000 | |
| high_school_biology 0.738710 | |
| high_school_chemistry 0.541872 | |
| high_school_computer_science 0.650000 | |
| high_school_european_history 0.751515 | |
| high_school_geography 0.767677 | |
| high_school_government_and_politics 0.808290 | |
| high_school_macroeconomics 0.638462 | |
| high_school_mathematics 0.433333 | |
| high_school_microeconomics 0.676471 | |
| high_school_physics 0.377483 | |
| high_school_psychology 0.814679 | |
| high_school_statistics 0.527778 | |
| high_school_us_history 0.715686 | |
| high_school_world_history 0.767932 | |
| human_aging 0.641256 | |
| human_sexuality 0.679389 | |
| international_law 0.735537 | |
| jurisprudence 0.777778 | |
| logical_fallacies 0.760736 | |
| machine_learning 0.455357 | |
| management 0.757282 | |
| marketing 0.837607 | |
| medical_genetics 0.690000 | |
| miscellaneous 0.713921 | |
| moral_disputes 0.650289 | |
| moral_scenarios 0.243575 | |
| nutrition 0.663399 | |
| philosophy 0.668810 | |
| prehistory 0.682099 | |
| professional_accounting 0.489362 | |
| professional_law 0.418514 | |
| professional_medicine 0.599265 | |
| professional_psychology 0.588235 | |
| public_relations 0.600000 | |
| security_studies 0.689796 | |
| sociology 0.786070 | |
| us_foreign_policy 0.780000 | |
| virology 0.463855 | |
| world_religions 0.789474 | |
| INFO: 2024-11-26 20:28:00,453: llmtf.base.nlpcoreteam/enMMLU: metric | |
| subject | |
| STEM 0.528725 | |
| humanities 0.644203 | |
| other (business, health, misc.) 0.610063 | |
| social sciences 0.688972 | |
| INFO: 2024-11-26 20:28:00,464: llmtf.base.nlpcoreteam/enMMLU: {'acc': 0.6179910195383496} | |
| INFO: 2024-11-26 20:28:00,501: llmtf.base.evaluator: Ended eval | |
| INFO: 2024-11-26 20:28:00,524: llmtf.base.evaluator: | |
| mean NEREL daru/treewayabstractive darumeru/MultiQ darumeru/PARus darumeru/RCB darumeru/RWSD darumeru/USE darumeru/cp_para_ru nlpcoreteam/enMMLU ruopinionne ruparam shlepa/books_mc shlepa/law_mc shlepa/movie_mc shlepa/music_mc vikhrmodels/habr_qa_sbs | |
| 0.337 0.007 0.209 0.257 0.440 0.414 0.490 0.061 0.800 0.618 0.027 0.214 0.311 0.538 0.225 0.245 0.538 | |
| INFO: 2024-11-26 20:28:16,900: llmtf.base.nlpcoreteam/ruMMLU: Processing Dataset: 501.69s | |
| INFO: 2024-11-26 20:28:16,902: llmtf.base.nlpcoreteam/ruMMLU: Results for nlpcoreteam/ruMMLU: | |
| INFO: 2024-11-26 20:28:16,950: llmtf.base.nlpcoreteam/ruMMLU: metric | |
| subject | |
| abstract_algebra 0.320000 | |
| anatomy 0.414815 | |
| astronomy 0.572368 | |
| business_ethics 0.450000 | |
| clinical_knowledge 0.505660 | |
| college_biology 0.375000 | |
| college_chemistry 0.310000 | |
| college_computer_science 0.380000 | |
| college_mathematics 0.350000 | |
| college_medicine 0.508671 | |
| college_physics 0.431373 | |
| computer_security 0.530000 | |
| conceptual_physics 0.429787 | |
| econometrics 0.298246 | |
| electrical_engineering 0.448276 | |
| elementary_mathematics 0.417989 | |
| formal_logic 0.365079 | |
| global_facts 0.240000 | |
| high_school_biology 0.487097 | |
| high_school_chemistry 0.443350 | |
| high_school_computer_science 0.530000 | |
| high_school_european_history 0.654545 | |
| high_school_geography 0.525253 | |
| high_school_government_and_politics 0.481865 | |
| high_school_macroeconomics 0.430769 | |
| high_school_mathematics 0.392593 | |
| high_school_microeconomics 0.441176 | |
| high_school_physics 0.291391 | |
| high_school_psychology 0.572477 | |
| high_school_statistics 0.416667 | |
| high_school_us_history 0.495098 | |
| high_school_world_history 0.632911 | |
| human_aging 0.488789 | |
| human_sexuality 0.496183 | |
| international_law 0.677686 | |
| jurisprudence 0.574074 | |
| logical_fallacies 0.441718 | |
| machine_learning 0.321429 | |
| management 0.563107 | |
| marketing 0.722222 | |
| medical_genetics 0.470000 | |
| miscellaneous 0.536398 | |
| moral_disputes 0.517341 | |
| moral_scenarios 0.237989 | |
| nutrition 0.526144 | |
| philosophy 0.546624 | |
| prehistory 0.478395 | |
| professional_accounting 0.365248 | |
| professional_law 0.331160 | |
| professional_medicine 0.367647 | |
| professional_psychology 0.415033 | |
| public_relations 0.454545 | |
| security_studies 0.595918 | |
| sociology 0.631841 | |
| us_foreign_policy 0.710000 | |
| virology 0.409639 | |
| world_religions 0.549708 | |
| INFO: 2024-11-26 20:28:16,959: llmtf.base.nlpcoreteam/ruMMLU: metric | |
| subject | |
| STEM 0.413740 | |
| humanities 0.500179 | |
| other (business, health, misc.) 0.469167 | |
| social sciences 0.504442 | |
| INFO: 2024-11-26 20:28:16,967: llmtf.base.nlpcoreteam/ruMMLU: {'acc': 0.47188210698069283} | |
| INFO: 2024-11-26 20:28:17,010: llmtf.base.evaluator: Ended eval | |
| INFO: 2024-11-26 20:28:17,020: llmtf.base.evaluator: | |
| mean NEREL daru/treewayabstractive darumeru/MultiQ darumeru/PARus darumeru/RCB darumeru/RWSD darumeru/USE darumeru/cp_para_ru nlpcoreteam/enMMLU nlpcoreteam/ruMMLU ruopinionne ruparam shlepa/books_mc shlepa/law_mc shlepa/movie_mc shlepa/music_mc vikhrmodels/habr_qa_sbs | |
| 0.345 0.007 0.209 0.257 0.440 0.414 0.490 0.061 0.800 0.618 0.472 0.027 0.214 0.311 0.538 0.225 0.245 0.538 | |