Hugging Face's logo

Harley-ml
/
Dillion-1.2M

Text Generation
Transformers
Safetensors
PyTorch
English
qwen3_5_text
dillion
small
harley-ml
Model card Files
xet
 Community
1
Harley-ml commited on
Commit
2629df2
·
verified ·
1 Parent(s): eb884ef

Create benchmarks.txt

Browse files
Files changed (1) hide show
  1. benchmarks.txt +87 -0
benchmarks.txt ADDED
@@ -0,0 +1,87 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ | Tasks |Version|Filter|n-shot| Metric | | Value | |Stde|
2
+ |------------------------------------------------------------|------:|------|-----:|---------------|---|-------:|---|----|
3
+ |arc_challenge | 1|none | 0|acc |↑ | 0.1817|± |0.01|
4
+ | | |none | 0|acc_norm |↑ | 0.2244|± |0.01|
5
+ |arc_easy | 1|none | 0|acc |↑ | 0.3144|± |0.00|
6
+ | | |none | 0|acc_norm |↑ | 0.3136|± |0.00|
7
+ |blimp | 2|none | |acc |↑ | 0.6294|± |0.00|
8
+ | - blimp_adjunct_island | 1|none | 0|acc |↑ | 0.5970|± |0.01|
9
+ | - blimp_anaphor_gender_agreement | 1|none | 0|acc |↑ | 0.2480|± |0.01|
10
+ | - blimp_anaphor_number_agreement | 1|none | 0|acc |↑ | 0.5930|± |0.01|
11
+ | - blimp_animate_subject_passive | 1|none | 0|acc |↑ | 0.6450|± |0.01|
12
+ | - blimp_animate_subject_trans | 1|none | 0|acc |↑ | 0.7630|± |0.01|
13
+ | - blimp_causative | 1|none | 0|acc |↑ | 0.5490|± |0.01|
14
+ | - blimp_complex_NP_island | 1|none | 0|acc |↑ | 0.5290|± |0.01|
15
+ | - blimp_coordinate_structure_constraint_complex_left_branch| 1|none | 0|acc |↑ | 0.3190|± |0.01|
16
+ | - blimp_coordinate_structure_constraint_object_extraction | 1|none | 0|acc |↑ | 0.6490|± |0.01|
17
+ | - blimp_determiner_noun_agreement_1 | 1|none | 0|acc |↑ | 0.8720|± |0.01|
18
+ | - blimp_determiner_noun_agreement_2 | 1|none | 0|acc |↑ | 0.8800|± |0.01|
19
+ | - blimp_determiner_noun_agreement_irregular_1 | 1|none | 0|acc |↑ | 0.7290|± |0.01|
20
+ | - blimp_determiner_noun_agreement_irregular_2 | 1|none | 0|acc |↑ | 0.8230|± |0.01|
21
+ | - blimp_determiner_noun_agreement_with_adj_2 | 1|none | 0|acc |↑ | 0.7780|± |0.01|
22
+ | - blimp_determiner_noun_agreement_with_adj_irregular_1 | 1|none | 0|acc |↑ | 0.6510|± |0.01|
23
+ | - blimp_determiner_noun_agreement_with_adj_irregular_2 | 1|none | 0|acc |↑ | 0.7550|± |0.01|
24
+ | - blimp_determiner_noun_agreement_with_adjective_1 | 1|none | 0|acc |↑ | 0.8110|± |0.01|
25
+ | - blimp_distractor_agreement_relational_noun | 1|none | 0|acc |↑ | 0.4350|± |0.01|
26
+ | - blimp_distractor_agreement_relative_clause | 1|none | 0|acc |↑ | 0.3030|± |0.01|
27
+ | - blimp_drop_argument | 1|none | 0|acc |↑ | 0.7290|± |0.01|
28
+ | - blimp_ellipsis_n_bar_1 | 1|none | 0|acc |↑ | 0.4440|± |0.01|
29
+ | - blimp_ellipsis_n_bar_2 | 1|none | 0|acc |↑ | 0.7500|± |0.01|
30
+ | - blimp_existential_there_object_raising | 1|none | 0|acc |↑ | 0.7660|± |0.01|
31
+ | - blimp_existential_there_quantifiers_1 | 1|none | 0|acc |↑ | 0.9390|± |0.00|
32
+ | - blimp_existential_there_quantifiers_2 | 1|none | 0|acc |↑ | 0.4420|± |0.01|
33
+ | - blimp_existential_there_subject_raising | 1|none | 0|acc |↑ | 0.7570|± |0.01|
34
+ | - blimp_expletive_it_object_raising | 1|none | 0|acc |↑ | 0.6800|± |0.01|
35
+ | - blimp_inchoative | 1|none | 0|acc |↑ | 0.4160|± |0.01|
36
+ | - blimp_intransitive | 1|none | 0|acc |↑ | 0.5920|± |0.01|
37
+ | - blimp_irregular_past_participle_adjectives | 1|none | 0|acc |↑ | 0.8590|± |0.01|
38
+ | - blimp_irregular_past_participle_verbs | 1|none | 0|acc |↑ | 0.6770|± |0.01|
39
+ | - blimp_irregular_plural_subject_verb_agreement_1 | 1|none | 0|acc |↑ | 0.7190|± |0.01|
40
+ | - blimp_irregular_plural_subject_verb_agreement_2 | 1|none | 0|acc |↑ | 0.8020|± |0.01|
41
+ | - blimp_left_branch_island_echo_question | 1|none | 0|acc |↑ | 0.4720|± |0.01|
42
+ | - blimp_left_branch_island_simple_question | 1|none | 0|acc |↑ | 0.3430|± |0.01|
43
+ | - blimp_matrix_question_npi_licensor_present | 1|none | 0|acc |↑ | 0.0370|± |0.00|
44
+ | - blimp_npi_present_1 | 1|none | 0|acc |↑ | 0.5970|± |0.01|
45
+ | - blimp_npi_present_2 | 1|none | 0|acc |↑ | 0.4770|± |0.01|
46
+ | - blimp_only_npi_licensor_present | 1|none | 0|acc |↑ | 0.5170|± |0.01|
47
+ | - blimp_only_npi_scope | 1|none | 0|acc |↑ | 0.4840|± |0.01|
48
+ | - blimp_passive_1 | 1|none | 0|acc |↑ | 0.8170|± |0.01|
49
+ | - blimp_passive_2 | 1|none | 0|acc |↑ | 0.7790|± |0.01|
50
+ | - blimp_principle_A_c_command | 1|none | 0|acc |↑ | 0.7350|± |0.01|
51
+ | - blimp_principle_A_case_1 | 1|none | 0|acc |↑ | 1.0000|± | |
52
+ | - blimp_principle_A_case_2 | 1|none | 0|acc |↑ | 0.8370|± |0.01|
53
+ | - blimp_principle_A_domain_1 | 1|none | 0|acc |↑ | 0.9880|± |0.00|
54
+ | - blimp_principle_A_domain_2 | 1|none | 0|acc |↑ | 0.5750|± |0.01|
55
+ | - blimp_principle_A_domain_3 | 1|none | 0|acc |↑ | 0.4810|± |0.01|
56
+ | - blimp_principle_A_reconstruction | 1|none | 0|acc |↑ | 0.5350|± |0.01|
57
+ | - blimp_regular_plural_subject_verb_agreement_1 | 1|none | 0|acc |↑ | 0.7110|± |0.01|
58
+ | - blimp_regular_plural_subject_verb_agreement_2 | 1|none | 0|acc |↑ | 0.7870|± |0.01|
59
+ | - blimp_sentential_negation_npi_licensor_present | 1|none | 0|acc |↑ | 0.9970|± |0.00|
60
+ | - blimp_sentential_negation_npi_scope | 1|none | 0|acc |↑ | 0.4500|± |0.01|
61
+ | - blimp_sentential_subject_island | 1|none | 0|acc |↑ | 0.3070|± |0.01|
62
+ | - blimp_superlative_quantifiers_1 | 1|none | 0|acc |↑ | 0.5800|± |0.01|
63
+ | - blimp_superlative_quantifiers_2 | 1|none | 0|acc |↑ | 0.3040|± |0.01|
64
+ | - blimp_tough_vs_raising_1 | 1|none | 0|acc |↑ | 0.4480|± |0.01|
65
+ | - blimp_tough_vs_raising_2 | 1|none | 0|acc |↑ | 0.6910|± |0.01|
66
+ | - blimp_transitive | 1|none | 0|acc |↑ | 0.7500|± |0.01|
67
+ | - blimp_wh_island | 1|none | 0|acc |↑ | 0.3930|± |0.01|
68
+ | - blimp_wh_questions_object_gap | 1|none | 0|acc |↑ | 0.5950|± |0.01|
69
+ | - blimp_wh_questions_subject_gap | 1|none | 0|acc |↑ | 0.8910|± |0.00|
70
+ | - blimp_wh_questions_subject_gap_long_distance | 1|none | 0|acc |↑ | 0.9390|± |0.00|
71
+ | - blimp_wh_vs_that_no_gap | 1|none | 0|acc |↑ | 0.9700|± |0.00|
72
+ | - blimp_wh_vs_that_no_gap_long_distance | 1|none | 0|acc |↑ | 0.9800|± |0.00|
73
+ | - blimp_wh_vs_that_with_gap | 1|none | 0|acc |↑ | 0.1260|± |0.01|
74
+ | - blimp_wh_vs_that_with_gap_long_distance | 1|none | 0|acc |↑ | 0.0780|± |0.00|
75
+ |hellaswag | 1|none | 0|acc |↑ | 0.2605|± |0.00|
76
+ | | |none | 0|acc_norm |↑ | 0.2676|± |0.00|
77
+ |piqa | 1|none | 0|acc |↑ | 0.5446|± |0.01|
78
+ | | |none | 0|acc_norm |↑ | 0.5310|± |0.01|
79
+ |swag | 1|none | 0|acc |↑ | 0.2851|± |0.00|
80
+ | | |none | 0|acc_norm |↑ | 0.3036|± |0.00|
81
+ |wikitext | 2|none | 0|bits_per_byte |↓ | 1.6161|± | N|
82
+ | | |none | 0|byte_perplexity|↓ | 3.0655|± | N|
83
+ | | |none | 0|word_perplexity|↓ |399.5047|± | N|
84
+
85
+ |Groups|Version|Filter|n-shot|Metric| |Value | |Stderr|
86
+ |------|------:|------|------|------|---|-----:|---|-----:|
87
+ |blimp | 2|none | |acc |↑ |0.6294|± |0.0016|