Chytrej2-Mini

A fully custom pretrained language model built from scratch on the LLaMA architecture trained on 2B tokens of the FineWeb Edu dataset.

Built by PingVortex Labs.

Model Details

Parameters: 20M
Context length: 1024 tokens
Language: English only
Format: Base model
Architecture: LLaMA
License: Apache 2.0

Benchmark

The model achieves score of 35.77% on ARC-Easy benchmark.

Usage

from transformers import LlamaForCausalLM, PreTrainedTokenizerFast

model = LlamaForCausalLM.from_pretrained("pvlabs/Chytrej2-Mini")
tokenizer = PreTrainedTokenizerFast.from_pretrained("pvlabs/Chytrej2-Mini")

prompt = "Neural Networks are"
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=100, repetition_penalty=1.3)
print(tokenizer.decode(outputs[0]))

Made by PingVortex.