rtferraz
/

domainTokenizer

Model card Files Files and versions

domainTokenizer / src /domain_tokenizer /training

Ctrl+K

Ctrl+K

1 contributor

History: 2 commits

rtferraz's picture

Add data_pipeline.py — tokenize_user_sequences, pack_sequences, prepare_clm_dataset

1dfd4e2 verified 9 days ago

__init__.py

315 Bytes
Phase 2C: Pre-training pipeline — data pipeline, sequence packing, HF Trainer CLM, 124 total tests passing 9 days ago
data_pipeline.py

3.65 kB
Add data_pipeline.py — tokenize_user_sequences, pack_sequences, prepare_clm_dataset 9 days ago