| INFO: 2024-07-13 15:18:25,981: llmtf.base.evaluator: Starting eval on ['darumeru/multiq', 'darumeru/parus', 'darumeru/rcb', 'darumeru/ruopenbookqa', 'darumeru/rutie', 'darumeru/ruworldtree', 'darumeru/rwsd', 'darumeru/use', 'russiannlp/rucola_custom'] |
| INFO: 2024-07-13 15:18:25,995: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [32000] |
| INFO: 2024-07-13 15:18:25,996: llmtf.base.hfmodel: Updated generation_config.stop_strings: [] |
| INFO: 2024-07-13 15:18:26,240: llmtf.base.evaluator: Starting eval on ['darumeru/rummlu'] |
| INFO: 2024-07-13 15:18:26,240: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [32000] |
| INFO: 2024-07-13 15:18:26,240: llmtf.base.hfmodel: Updated generation_config.stop_strings: [] |
| INFO: 2024-07-13 15:18:27,990: llmtf.base.evaluator: Starting eval on ['nlpcoreteam/rummlu'] |
| INFO: 2024-07-13 15:18:27,991: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [32000] |
| INFO: 2024-07-13 15:18:27,991: llmtf.base.hfmodel: Updated generation_config.stop_strings: [] |
| INFO: 2024-07-13 15:18:29,333: llmtf.base.evaluator: Starting eval on ['nlpcoreteam/enmmlu'] |
| INFO: 2024-07-13 15:18:29,333: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [32000] |
| INFO: 2024-07-13 15:18:29,333: llmtf.base.hfmodel: Updated generation_config.stop_strings: [] |
| INFO: 2024-07-13 15:18:29,480: llmtf.base.darumeru/MultiQ: Loading Dataset: 3.48s |
| INFO: 2024-07-13 15:18:30,985: llmtf.base.evaluator: Starting eval on ['daru/treewayabstractive'] |
| INFO: 2024-07-13 15:18:30,985: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [32000] |
| INFO: 2024-07-13 15:18:30,985: llmtf.base.hfmodel: Updated generation_config.stop_strings: [] |
| INFO: 2024-07-13 15:18:33,199: llmtf.base.evaluator: Starting eval on ['daru/treewayextractive'] |
| INFO: 2024-07-13 15:18:33,200: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [32000] |
| INFO: 2024-07-13 15:18:33,200: llmtf.base.hfmodel: Updated generation_config.stop_strings: [] |
| INFO: 2024-07-13 15:18:33,694: llmtf.base.darumeru/ruMMLU: Loading Dataset: 7.45s |
| INFO: 2024-07-13 15:18:35,345: llmtf.base.daru/treewayabstractive: Loading Dataset: 4.36s |
| INFO: 2024-07-13 15:18:35,432: llmtf.base.evaluator: Starting eval on ['darumeru/cp_sent_ru', 'darumeru/cp_sent_en', 'darumeru/cp_para_ru', 'darumeru/cp_para_en'] |
| INFO: 2024-07-13 15:18:35,433: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [32000] |
| INFO: 2024-07-13 15:18:35,433: llmtf.base.hfmodel: Updated generation_config.stop_strings: [] |
| INFO: 2024-07-13 15:18:37,953: llmtf.base.darumeru/cp_sent_ru: Loading Dataset: 2.52s |
| INFO: 2024-07-13 15:18:40,885: llmtf.base.daru/treewayextractive: Loading Dataset: 7.69s |
| INFO: 2024-07-13 15:23:40,040: llmtf.base.evaluator: Starting eval on ['darumeru/parus', 'darumeru/rcb', 'darumeru/ruopenbookqa', 'darumeru/rutie', 'darumeru/ruworldtree', 'darumeru/rwsd', 'russiannlp/rucola_custom'] |
| INFO: 2024-07-13 15:23:40,042: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [32000] |
| INFO: 2024-07-13 15:23:40,042: llmtf.base.hfmodel: Updated generation_config.stop_strings: [] |
| INFO: 2024-07-13 15:23:40,509: llmtf.base.evaluator: Starting eval on ['darumeru/rummlu', 'daru/treewayextractive'] |
| INFO: 2024-07-13 15:23:40,510: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [32000] |
| INFO: 2024-07-13 15:23:40,510: llmtf.base.hfmodel: Updated generation_config.stop_strings: [] |
| INFO: 2024-07-13 15:23:41,090: llmtf.base.evaluator: Starting eval on ['nlpcoreteam/rummlu', 'nlpcoreteam/enmmlu'] |
| INFO: 2024-07-13 15:23:41,091: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [32000] |
| INFO: 2024-07-13 15:23:41,091: llmtf.base.hfmodel: Updated generation_config.stop_strings: [] |
| INFO: 2024-07-13 15:23:42,369: llmtf.base.darumeru/PARus: Loading Dataset: 2.33s |
| INFO: 2024-07-13 15:23:43,206: llmtf.base.evaluator: Starting eval on ['daru/treewayabstractive'] |
| INFO: 2024-07-13 15:23:43,207: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [32000] |
| INFO: 2024-07-13 15:23:43,207: llmtf.base.hfmodel: Updated generation_config.stop_strings: [] |
| INFO: 2024-07-13 15:23:45,405: llmtf.base.evaluator: Starting eval on ['darumeru/multiq', 'darumeru/use'] |
| INFO: 2024-07-13 15:23:45,405: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [32000] |
| INFO: 2024-07-13 15:23:45,405: llmtf.base.hfmodel: Updated generation_config.stop_strings: [] |
| INFO: 2024-07-13 15:23:45,786: llmtf.base.darumeru/PARus: Processing Dataset: 3.42s |
| INFO: 2024-07-13 15:23:45,788: llmtf.base.darumeru/PARus: Results for darumeru/PARus: |
| INFO: 2024-07-13 15:23:45,800: llmtf.base.darumeru/PARus: {'acc': 0.75} |
| INFO: 2024-07-13 15:23:45,801: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [32000] |
| INFO: 2024-07-13 15:23:45,801: llmtf.base.hfmodel: Updated generation_config.stop_strings: [] |
| INFO: 2024-07-13 15:23:47,436: llmtf.base.daru/treewayabstractive: Loading Dataset: 4.23s |
| INFO: 2024-07-13 15:23:47,479: llmtf.base.evaluator: Starting eval on ['darumeru/cp_sent_ru', 'darumeru/cp_para_ru'] |
| INFO: 2024-07-13 15:23:47,479: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [32000] |
| INFO: 2024-07-13 15:23:47,479: llmtf.base.hfmodel: Updated generation_config.stop_strings: [] |
| INFO: 2024-07-13 15:23:47,668: llmtf.base.darumeru/ruMMLU: Loading Dataset: 7.16s |
| INFO: 2024-07-13 15:23:47,812: llmtf.base.darumeru/RCB: Loading Dataset: 2.01s |
| INFO: 2024-07-13 15:23:49,390: llmtf.base.evaluator: Starting eval on ['darumeru/cp_sent_en', 'darumeru/cp_para_en'] |
| INFO: 2024-07-13 15:23:49,390: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [32000] |
| INFO: 2024-07-13 15:23:49,390: llmtf.base.hfmodel: Updated generation_config.stop_strings: [] |
| INFO: 2024-07-13 15:23:49,768: llmtf.base.darumeru/cp_sent_ru: Loading Dataset: 2.29s |
| INFO: 2024-07-13 15:23:51,703: llmtf.base.darumeru/cp_sent_en: Loading Dataset: 2.31s |
| INFO: 2024-07-13 15:23:53,877: llmtf.base.darumeru/RCB: Processing Dataset: 6.06s |
| INFO: 2024-07-13 15:23:53,892: llmtf.base.darumeru/RCB: Results for darumeru/RCB: |
| INFO: 2024-07-13 15:23:53,898: llmtf.base.darumeru/RCB: {'acc': 0.5227272727272727, 'f1_macro': 0.4428418803418803} |
| INFO: 2024-07-13 15:23:53,899: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [32000] |
| INFO: 2024-07-13 15:23:53,899: llmtf.base.hfmodel: Updated generation_config.stop_strings: [] |
| INFO: 2024-07-13 15:23:56,231: llmtf.base.darumeru/ruOpenBookQA: Loading Dataset: 2.33s |
| INFO: 2024-07-13 15:24:00,471: llmtf.base.darumeru/MultiQ: Loading Dataset: 15.07s |
| INFO: 2024-07-13 15:24:35,821: llmtf.base.darumeru/ruOpenBookQA: Processing Dataset: 39.59s |
| INFO: 2024-07-13 15:24:35,822: llmtf.base.darumeru/ruOpenBookQA: Results for darumeru/ruOpenBookQA: |
| INFO: 2024-07-13 15:24:35,835: llmtf.base.darumeru/ruOpenBookQA: {'acc': 0.7323883161512027, 'f1_macro': 0.7329226353930633} |
| INFO: 2024-07-13 15:24:35,842: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [32000] |
| INFO: 2024-07-13 15:24:35,842: llmtf.base.hfmodel: Updated generation_config.stop_strings: [] |
| INFO: 2024-07-13 15:24:40,146: llmtf.base.darumeru/ruTiE: Loading Dataset: 4.30s |
| INFO: 2024-07-13 15:26:07,845: llmtf.base.nlpcoreteam/ruMMLU: Loading Dataset: 146.75s |
| INFO: 2024-07-13 15:26:38,965: llmtf.base.darumeru/cp_sent_en: Processing Dataset: 167.26s |
| INFO: 2024-07-13 15:26:38,982: llmtf.base.darumeru/cp_sent_en: Results for darumeru/cp_sent_en: |
| INFO: 2024-07-13 15:26:38,999: llmtf.base.darumeru/cp_sent_en: {'symbol_per_token': 3.895358639670925, 'len': 0.9974420397589191, 'lcs': 0.9801922792969053} |
| INFO: 2024-07-13 15:26:39,001: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [32000] |
| INFO: 2024-07-13 15:26:39,001: llmtf.base.hfmodel: Updated generation_config.stop_strings: [] |
| INFO: 2024-07-13 15:26:41,305: llmtf.base.darumeru/cp_para_en: Loading Dataset: 2.30s |
| INFO: 2024-07-13 15:27:49,425: llmtf.base.darumeru/cp_sent_ru: Processing Dataset: 239.65s |
| INFO: 2024-07-13 15:27:49,442: llmtf.base.darumeru/cp_sent_ru: Results for darumeru/cp_sent_ru: |
| INFO: 2024-07-13 15:27:49,446: llmtf.base.darumeru/cp_sent_ru: {'symbol_per_token': 2.3734936335344794, 'len': 0.9922334558022529, 'lcs': 0.9153760193869099} |
| INFO: 2024-07-13 15:27:49,448: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [32000] |
| INFO: 2024-07-13 15:27:49,448: llmtf.base.hfmodel: Updated generation_config.stop_strings: [] |
| INFO: 2024-07-13 15:27:51,495: llmtf.base.darumeru/cp_para_ru: Loading Dataset: 2.05s |
| INFO: 2024-07-13 15:28:59,772: llmtf.base.darumeru/ruTiE: Processing Dataset: 259.62s |
| INFO: 2024-07-13 15:28:59,774: llmtf.base.darumeru/ruTiE: Results for darumeru/ruTiE: |
| INFO: 2024-07-13 15:28:59,815: llmtf.base.darumeru/ruTiE: {'acc': 0.5372093023255814} |
| INFO: 2024-07-13 15:28:59,818: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [32000] |
| INFO: 2024-07-13 15:28:59,819: llmtf.base.hfmodel: Updated generation_config.stop_strings: [] |
| INFO: 2024-07-13 15:29:01,873: llmtf.base.darumeru/ruWorldTree: Loading Dataset: 2.05s |
| INFO: 2024-07-13 15:29:04,102: llmtf.base.darumeru/ruWorldTree: Processing Dataset: 2.21s |
| INFO: 2024-07-13 15:29:04,104: llmtf.base.darumeru/ruWorldTree: Results for darumeru/ruWorldTree: |
| INFO: 2024-07-13 15:29:04,109: llmtf.base.darumeru/ruWorldTree: {'acc': 0.8761904761904762, 'f1_macro': 0.8744880624959789} |
| INFO: 2024-07-13 15:29:04,110: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [32000] |
| INFO: 2024-07-13 15:29:04,110: llmtf.base.hfmodel: Updated generation_config.stop_strings: [] |
| INFO: 2024-07-13 15:29:06,342: llmtf.base.darumeru/RWSD: Loading Dataset: 2.23s |
| INFO: 2024-07-13 15:29:13,292: llmtf.base.darumeru/RWSD: Processing Dataset: 6.95s |
| INFO: 2024-07-13 15:29:13,294: llmtf.base.darumeru/RWSD: Results for darumeru/RWSD: |
| INFO: 2024-07-13 15:29:13,298: llmtf.base.darumeru/RWSD: {'acc': 0.5392156862745098} |
| INFO: 2024-07-13 15:29:13,299: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [32000] |
| INFO: 2024-07-13 15:29:13,299: llmtf.base.hfmodel: Updated generation_config.stop_strings: [] |
| INFO: 2024-07-13 15:29:16,878: llmtf.base.russiannlp/rucola_custom: Loading Dataset: 3.58s |
| INFO: 2024-07-13 15:30:03,574: llmtf.base.russiannlp/rucola_custom: Processing Dataset: 46.69s |
| INFO: 2024-07-13 15:30:03,578: llmtf.base.russiannlp/rucola_custom: Results for russiannlp/rucola_custom: |
| INFO: 2024-07-13 15:30:03,605: llmtf.base.russiannlp/rucola_custom: {'acc': 0.7366343738787227, 'mcc': 0.34075509260259335} |
| INFO: 2024-07-13 15:30:03,609: llmtf.base.evaluator: Ended eval |
| INFO: 2024-07-13 15:30:03,635: llmtf.base.evaluator: |
| mean darumeru/PARus darumeru/RCB darumeru/RWSD darumeru/cp_sent_en darumeru/cp_sent_ru darumeru/ruOpenBookQA darumeru/ruTiE darumeru/ruWorldTree russiannlp/rucola_custom |
| 0.716 0.750 0.483 0.539 0.997 0.992 0.733 0.537 0.875 0.539 |
| INFO: 2024-07-13 15:30:38,848: llmtf.base.darumeru/cp_para_en: Processing Dataset: 237.54s |
| INFO: 2024-07-13 15:30:38,850: llmtf.base.darumeru/cp_para_en: Results for darumeru/cp_para_en: |
| INFO: 2024-07-13 15:30:38,867: llmtf.base.darumeru/cp_para_en: {'symbol_per_token': 3.964633931969539, 'len': 0.9963390331388082, 'lcs': 0.873438038674546} |
| INFO: 2024-07-13 15:30:38,867: llmtf.base.evaluator: Ended eval |
| INFO: 2024-07-13 15:30:38,877: llmtf.base.evaluator: |
| mean darumeru/PARus darumeru/RCB darumeru/RWSD darumeru/cp_para_en darumeru/cp_sent_en darumeru/cp_sent_ru darumeru/ruOpenBookQA darumeru/ruTiE darumeru/ruWorldTree russiannlp/rucola_custom |
| 0.732 0.750 0.483 0.539 0.873 0.997 0.992 0.733 0.537 0.875 0.539 |
| INFO: 2024-07-13 15:30:58,082: llmtf.base.darumeru/ruMMLU: Processing Dataset: 430.41s |
| INFO: 2024-07-13 15:30:58,085: llmtf.base.darumeru/ruMMLU: Results for darumeru/ruMMLU: |
| INFO: 2024-07-13 15:30:58,123: llmtf.base.darumeru/ruMMLU: {'acc': 0.4818916492068243} |
| INFO: 2024-07-13 15:30:58,162: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [32000] |
| INFO: 2024-07-13 15:30:58,163: llmtf.base.hfmodel: Updated generation_config.stop_strings: [] |
| INFO: 2024-07-13 15:31:08,009: llmtf.base.daru/treewayextractive: Loading Dataset: 9.84s |
| INFO: 2024-07-13 15:31:29,151: llmtf.base.darumeru/MultiQ: Processing Dataset: 448.68s |
| INFO: 2024-07-13 15:31:29,154: llmtf.base.darumeru/MultiQ: Results for darumeru/MultiQ: |
| INFO: 2024-07-13 15:31:29,174: llmtf.base.darumeru/MultiQ: {'f1': 0.2909161781249439, 'em': 0.16634799235181644} |
| INFO: 2024-07-13 15:31:29,179: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [32000] |
| INFO: 2024-07-13 15:31:29,179: llmtf.base.hfmodel: Updated generation_config.stop_strings: [] |
| INFO: 2024-07-13 15:31:32,357: llmtf.base.darumeru/USE: Loading Dataset: 3.18s |
| INFO: 2024-07-13 15:33:20,030: llmtf.base.nlpcoreteam/ruMMLU: Processing Dataset: 432.18s |
| INFO: 2024-07-13 15:33:20,032: llmtf.base.nlpcoreteam/ruMMLU: Results for nlpcoreteam/ruMMLU: |
| INFO: 2024-07-13 15:33:20,078: llmtf.base.nlpcoreteam/ruMMLU: metric |
| subject |
| abstract_algebra 0.360000 |
| anatomy 0.414815 |
| astronomy 0.565789 |
| business_ethics 0.550000 |
| clinical_knowledge 0.528302 |
| college_biology 0.486111 |
| college_chemistry 0.450000 |
| college_computer_science 0.510000 |
| college_mathematics 0.390000 |
| college_medicine 0.508671 |
| college_physics 0.254902 |
| computer_security 0.580000 |
| conceptual_physics 0.434043 |
| econometrics 0.359649 |
| electrical_engineering 0.489655 |
| elementary_mathematics 0.370370 |
| formal_logic 0.325397 |
| global_facts 0.280000 |
| high_school_biology 0.590323 |
| high_school_chemistry 0.374384 |
| high_school_computer_science 0.600000 |
| high_school_european_history 0.666667 |
| high_school_geography 0.666667 |
| high_school_government_and_politics 0.580311 |
| high_school_macroeconomics 0.438462 |
| high_school_mathematics 0.359259 |
| high_school_microeconomics 0.478992 |
| high_school_physics 0.397351 |
| high_school_psychology 0.625688 |
| high_school_statistics 0.467593 |
| high_school_us_history 0.681373 |
| high_school_world_history 0.713080 |
| human_aging 0.515695 |
| human_sexuality 0.557252 |
| international_law 0.652893 |
| jurisprudence 0.527778 |
| logical_fallacies 0.423313 |
| machine_learning 0.321429 |
| management 0.631068 |
| marketing 0.675214 |
| medical_genetics 0.530000 |
| miscellaneous 0.624521 |
| moral_disputes 0.528902 |
| moral_scenarios 0.231285 |
| nutrition 0.555556 |
| philosophy 0.482315 |
| prehistory 0.490741 |
| professional_accounting 0.382979 |
| professional_law 0.367014 |
| professional_medicine 0.477941 |
| professional_psychology 0.439542 |
| public_relations 0.554545 |
| security_studies 0.612245 |
| sociology 0.686567 |
| us_foreign_policy 0.730000 |
| virology 0.487952 |
| world_religions 0.690058 |
| INFO: 2024-07-13 15:33:20,086: llmtf.base.nlpcoreteam/ruMMLU: metric |
| subject |
| STEM 0.444512 |
| humanities 0.521601 |
| other (business, health, misc.) 0.511622 |
| social sciences 0.560827 |
| INFO: 2024-07-13 15:33:20,093: llmtf.base.nlpcoreteam/ruMMLU: {'acc': 0.5096404231777287} |
| INFO: 2024-07-13 15:33:20,128: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [32000] |
| INFO: 2024-07-13 15:33:20,129: llmtf.base.hfmodel: Updated generation_config.stop_strings: [] |
| INFO: 2024-07-13 15:34:10,886: llmtf.base.darumeru/cp_para_ru: Processing Dataset: 379.39s |
| INFO: 2024-07-13 15:34:10,904: llmtf.base.darumeru/cp_para_ru: Results for darumeru/cp_para_ru: |
| INFO: 2024-07-13 15:34:10,908: llmtf.base.darumeru/cp_para_ru: {'symbol_per_token': 2.469064241004292, 'len': 0.9929789601006123, 'lcs': 0.843045621556421} |
| INFO: 2024-07-13 15:34:10,908: llmtf.base.evaluator: Ended eval |
| INFO: 2024-07-13 15:34:10,938: llmtf.base.evaluator: |
| mean darumeru/MultiQ darumeru/PARus darumeru/RCB darumeru/RWSD darumeru/cp_para_en darumeru/cp_para_ru darumeru/cp_sent_en darumeru/cp_sent_ru darumeru/ruMMLU darumeru/ruOpenBookQA darumeru/ruTiE darumeru/ruWorldTree nlpcoreteam/ruMMLU russiannlp/rucola_custom |
| 0.670 0.229 0.750 0.483 0.539 0.873 0.843 0.997 0.992 0.482 0.733 0.537 0.875 0.510 0.539 |
| INFO: 2024-07-13 15:35:15,961: llmtf.base.nlpcoreteam/enMMLU: Loading Dataset: 115.83s |
| INFO: 2024-07-13 15:36:14,977: llmtf.base.darumeru/USE: Processing Dataset: 282.62s |
| INFO: 2024-07-13 15:36:14,978: llmtf.base.darumeru/USE: Results for darumeru/USE: |
| INFO: 2024-07-13 15:36:14,982: llmtf.base.darumeru/USE: {'grade_norm': 0.06568627450980391} |
| INFO: 2024-07-13 15:36:14,985: llmtf.base.evaluator: Ended eval |
| INFO: 2024-07-13 15:36:14,994: llmtf.base.evaluator: |
| mean darumeru/MultiQ darumeru/PARus darumeru/RCB darumeru/RWSD darumeru/USE darumeru/cp_para_en darumeru/cp_para_ru darumeru/cp_sent_en darumeru/cp_sent_ru darumeru/ruMMLU darumeru/ruOpenBookQA darumeru/ruTiE darumeru/ruWorldTree nlpcoreteam/ruMMLU russiannlp/rucola_custom |
| 0.630 0.229 0.750 0.483 0.539 0.066 0.873 0.843 0.997 0.992 0.482 0.733 0.537 0.875 0.510 0.539 |
| INFO: 2024-07-13 15:39:22,754: llmtf.base.daru/treewayextractive: Processing Dataset: 494.73s |
| INFO: 2024-07-13 15:39:22,774: llmtf.base.daru/treewayextractive: Results for daru/treewayextractive: |
| INFO: 2024-07-13 15:39:23,003: llmtf.base.daru/treewayextractive: {'r-prec': 0.40769011544011546} |
| INFO: 2024-07-13 15:39:23,057: llmtf.base.evaluator: Ended eval |
| INFO: 2024-07-13 15:39:23,070: llmtf.base.evaluator: |
| mean daru/treewayextractive darumeru/MultiQ darumeru/PARus darumeru/RCB darumeru/RWSD darumeru/USE darumeru/cp_para_en darumeru/cp_para_ru darumeru/cp_sent_en darumeru/cp_sent_ru darumeru/ruMMLU darumeru/ruOpenBookQA darumeru/ruTiE darumeru/ruWorldTree nlpcoreteam/ruMMLU russiannlp/rucola_custom |
| 0.616 0.408 0.229 0.750 0.483 0.539 0.066 0.873 0.843 0.997 0.992 0.482 0.733 0.537 0.875 0.510 0.539 |
| INFO: 2024-07-13 15:40:06,403: llmtf.base.nlpcoreteam/enMMLU: Processing Dataset: 290.44s |
| INFO: 2024-07-13 15:40:06,407: llmtf.base.nlpcoreteam/enMMLU: Results for nlpcoreteam/enMMLU: |
| INFO: 2024-07-13 15:40:06,452: llmtf.base.nlpcoreteam/enMMLU: metric |
| subject |
| abstract_algebra 0.340000 |
| anatomy 0.614815 |
| astronomy 0.651316 |
| business_ethics 0.650000 |
| clinical_knowledge 0.675472 |
| college_biology 0.729167 |
| college_chemistry 0.470000 |
| college_computer_science 0.550000 |
| college_mathematics 0.380000 |
| college_medicine 0.641618 |
| college_physics 0.352941 |
| computer_security 0.700000 |
| conceptual_physics 0.553191 |
| econometrics 0.456140 |
| electrical_engineering 0.593103 |
| elementary_mathematics 0.412698 |
| formal_logic 0.476190 |
| global_facts 0.310000 |
| high_school_biology 0.790323 |
| high_school_chemistry 0.458128 |
| high_school_computer_science 0.670000 |
| high_school_european_history 0.769697 |
| high_school_geography 0.792929 |
| high_school_government_and_politics 0.880829 |
| high_school_macroeconomics 0.612821 |
| high_school_mathematics 0.344444 |
| high_school_microeconomics 0.642857 |
| high_school_physics 0.337748 |
| high_school_psychology 0.823853 |
| high_school_statistics 0.467593 |
| high_school_us_history 0.794118 |
| high_school_world_history 0.814346 |
| human_aging 0.721973 |
| human_sexuality 0.732824 |
| international_law 0.735537 |
| jurisprudence 0.722222 |
| logical_fallacies 0.760736 |
| machine_learning 0.455357 |
| management 0.776699 |
| marketing 0.858974 |
| medical_genetics 0.690000 |
| miscellaneous 0.830140 |
| moral_disputes 0.679191 |
| moral_scenarios 0.232402 |
| nutrition 0.709150 |
| philosophy 0.655949 |
| prehistory 0.672840 |
| professional_accounting 0.460993 |
| professional_law 0.468057 |
| professional_medicine 0.709559 |
| professional_psychology 0.619281 |
| public_relations 0.645455 |
| security_studies 0.673469 |
| sociology 0.850746 |
| us_foreign_policy 0.850000 |
| virology 0.512048 |
| world_religions 0.853801 |
| INFO: 2024-07-13 15:40:06,459: llmtf.base.nlpcoreteam/enMMLU: metric |
| subject |
| STEM 0.514223 |
| humanities 0.664237 |
| other (business, health, misc.) 0.654389 |
| social sciences 0.715100 |
| INFO: 2024-07-13 15:40:06,466: llmtf.base.nlpcoreteam/enMMLU: {'acc': 0.6369873391991915} |
| INFO: 2024-07-13 15:40:06,497: llmtf.base.evaluator: Ended eval |
| INFO: 2024-07-13 15:40:06,508: llmtf.base.evaluator: |
| mean daru/treewayextractive darumeru/MultiQ darumeru/PARus darumeru/RCB darumeru/RWSD darumeru/USE darumeru/cp_para_en darumeru/cp_para_ru darumeru/cp_sent_en darumeru/cp_sent_ru darumeru/ruMMLU darumeru/ruOpenBookQA darumeru/ruTiE darumeru/ruWorldTree nlpcoreteam/enMMLU nlpcoreteam/ruMMLU russiannlp/rucola_custom |
| 0.617 0.408 0.229 0.750 0.483 0.539 0.066 0.873 0.843 0.997 0.992 0.482 0.733 0.537 0.875 0.637 0.510 0.539 |
| INFO: 2024-07-13 15:44:34,352: llmtf.base.daru/treewayabstractive: Processing Dataset: 1246.91s |
| INFO: 2024-07-13 15:44:34,354: llmtf.base.daru/treewayabstractive: Results for daru/treewayabstractive: |
| INFO: 2024-07-13 15:44:34,373: llmtf.base.daru/treewayabstractive: {'rouge1': 0.34479017541198337, 'rouge2': 0.12451437402782907} |
| INFO: 2024-07-13 15:44:34,376: llmtf.base.evaluator: Ended eval |
| INFO: 2024-07-13 15:44:34,403: llmtf.base.evaluator: |
| mean daru/treewayabstractive daru/treewayextractive darumeru/MultiQ darumeru/PARus darumeru/RCB darumeru/RWSD darumeru/USE darumeru/cp_para_en darumeru/cp_para_ru darumeru/cp_sent_en darumeru/cp_sent_ru darumeru/ruMMLU darumeru/ruOpenBookQA darumeru/ruTiE darumeru/ruWorldTree nlpcoreteam/enMMLU nlpcoreteam/ruMMLU russiannlp/rucola_custom |
| 0.596 0.235 0.408 0.229 0.750 0.483 0.539 0.066 0.873 0.843 0.997 0.992 0.482 0.733 0.537 0.875 0.637 0.510 0.539 |
|
|
