Where to download the tokenizer?
#1
by EarthWorm001 - opened
I'm trying to run the Mamba-2 hybrid using megatron, However, the example scripts require a tokenizer prepared. Where to download the tokenizer?
Thanks!
The tokenizer is uploaded in each repository under the filename mt_nlg_plus_multilingual_ja_zh_the_stack_frac_015_256k.model
rwaleffe changed discussion status to closed
So can I fine tune the model and add tokens to your model?