Enhancing Linguistic Competence of Language Models through Pre-training with Language Learning Tasks
Paper • 2601.03448 • Published • 13
L2T: Pre-training with Language Learning Tasks
Collection of models for "Enhancing Linguistic Competence of Language Models through Pre-training with Language Learning Tasks"
-
l2t-project/ntp_training_data_finewebedu_21b
Updated • 42Note Vanilla training data for full data adaptation analysis (21B raw tokens)
-
l2t-project/l2t-500m-disjoint
0.4B • Updated • 4 -
l2t-project/l2t-500m-disjoint-mix_25
0.4B • Updated • 5 -
l2t-project/l2t-500m-disjoint-mix_75
0.4B • Updated • 1 -
l2t-project/l2t-500m-disjoint-mix_0
0.4B • Updated • 1 -
l2t-project/raw-500m-disjoint
0.4B • Updated • 1 -
l2t-project/l2t-1b-disjoint
1.0B • Updated • 1 -
l2t-project/raw-1b-disjoint
1.0B • Updated • 2 -
l2t-project/l2t-500m-char_count
0.4B • Updated • 1 -
l2t-project/l2t-500m-deletion
0.4B • Updated • 8 -
l2t-project/l2t-500m-half
0.4B • Updated • 2 -
l2t-project/l2t-500m-last
0.4B • Updated • 2 -
l2t-project/l2t-500m-masked_char
0.4B • Updated • 4 -
l2t-project/l2t-500m-masked_word
0.4B • Updated • 1 -
l2t-project/l2t-500m-one
0.4B • Updated • 1 -
l2t-project/l2t-500m-random
0.4B • Updated • 1 -
l2t-project/l2t-500m-reordering
0.4B • Updated • 1 -
l2t-project/l2t-500m-fill_middle
0.4B • Updated • 2 -
l2t-project/l2t-500m-shuffle
0.4B • Updated • 1 -
l2t-project/l2t-500m-space
0.4B • Updated • 1 -
l2t-project/l2t-500m-typo
0.4B • Updated • 1 -
l2t-project/l2t-500m-token_type
0.4B • Updated • 1 -
l2t-project/raw-1b-shared
1.0B • Updated • 2 -
l2t-project/raw-500m-shared
0.4B • Updated • 1 -
l2t-project/l2t-1b-shared
1.0B • Updated • 2 -
l2t-project/l2t-500m-shared
0.4B • Updated • 1