README.md

---
license: apache-2.0
datasets:
- HuggingFaceFW/fineweb-edu
language:
- en
pipeline_tag: text-generation
library_name: transformers
tags:
- supra
- chimera
- 50m
- llama
- small
- open
- open-source
- cpu
- tiny
- slm
---

# 🦅 Supra-50M BASE

**Supra-50M** is a compact 50M-parameter BASE causal language model built by SupraLabs, trained from scratch using a Llama-style architecture on 20 billion tokens of high-quality educational web text. Despite being significantly smaller than comparable open models, it achieves competitive or superior results on several key benchmarks. It's our first SupraLabs Scaling Up Plan model.

---

## 🏆 Benchmarks

| Benchmark | Supra-50M *(ours)* | GPT-2 (124M) | SmolLM-135M | OpenELM-270M |
| :--- | :--- | :--- | :--- | :--- |
| **Parameters** | **50M** | 124M *(2.5×)* | 135M *(2.7×)* | 270M *(5.4×)* |
| **BLiMP** (linguistics) | **76.3%** | 63.0% | 69.8% | *(N/A)* |
| **SciQ** (science) | 77.2% | 53.2% | 73.4% | **84.70%** |
| **ARC-Easy** (knowledge) | **52.2%** | 42.0% | 49.2% | 45.08% |
| **PIQA** (logic) | 62.2% | 63.0% | 67.3% | **69.75%** |
| **HellaSwag** (context) | 31.8% | 29.5% | 42.0% | **46.71%** |

![Benchmarks](https://cdn-uploads.huggingface.co/production/uploads/697f2832c2c5e4daa93cece7/tTgsWh576T2sdVzetvZ8C.png)

---

## 🧠 Model Architecture & Hyperparameters

Supra-50M is based on the `LlamaForCausalLM` architecture with the following configuration:

| Hyperparameter | Value |
|---|---|
| Architecture | Llama (decoder-only transformer) |
| Parameters | ~50M |
| `vocab_size` | 32,000 |
| `hidden_size` | 512 |
| `intermediate_size` | 1,408 |
| `num_hidden_layers` | 12 |
| `num_attention_heads` | 8 |
| `num_key_value_heads` | 4 (GQA) |
| `max_position_embeddings` | 1,024 |
| `rope_theta` | 10,000 |
| `tie_word_embeddings` | True |

---

## 📚 Training Data

| Property | Value |
|---|---|
| Dataset | [HuggingFaceFW/fineweb-edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu) (`sample-100BT` split) |
| Total tokens | 20,000,000,000 (20B) |
| Sequence length | 1,024 tokens |
| Storage format | Memory-mapped binary (`uint16`, ~40 GB) |

---

## 🔤 Tokenizer

A custom **Byte-Level BPE** tokenizer was trained from scratch on 500,000 documents sampled from `fineweb-edu (sample-10BT)`.

| Property | Value |
|---|---|
| Type | ByteLevelBPETokenizer |
| Vocabulary size | 32,000 |
| Min frequency | 2 |
| Special tokens | `<s>`, `<pad>`, `</s>`, `<unk>`, `<mask>` |

---

## ⚙️ Training Configuration

| Parameter | Value |
|---|---|
| Epochs | 1 |
| Per-device batch size | 32 |
| Gradient accumulation steps | 4 |
| Effective batch size | 128 × 1,024 tokens |
| Learning rate | 6e-4 |
| LR scheduler | Cosine |
| Warmup ratio | 2% |
| Optimizer | AdamW Fused (`adam_beta1=0.9`, `adam_beta2=0.95`) |
| Weight decay | 0.1 |
| Max grad norm | 1.0 |
| Precision | bfloat16 |
| `torch.compile` | Enabled |
| Hardware | Single GPU |
| Final loss | *3.259* |

---

## 🚀 Inference

```python
from transformers import pipeline
import torch

print("[*] Loading Supra-50M model from Hugging Face Hub...")
pipe = pipeline(
    "text-generation",
    model="SupraLabs/Supra-50M_BASE",
    device_map="auto",
    torch_dtype=torch.float16 if torch.cuda.is_available() else torch.float32
)

def generate_text(prompt, max_new_tokens=150):
    result = pipe(
        prompt,
        max_new_tokens=max_new_tokens,
        do_sample=True,
        temperature=0.5,
        top_k=25,
        top_p=0.9,
        repetition_penalty=1.2,
        pad_token_id=pipe.tokenizer.pad_token_id,
        eos_token_id=pipe.tokenizer.eos_token_id
    )
    return result[0]['generated_text']

# Example
prompt = "The importance of education is"
print(f"\nPrompt: {prompt}")
print("-" * 40)
print("\nOutput:\n" + generate_text(prompt))
```

---

## 💬 Sample Outputs

**Prompt:** `"The main concept of physics is "`
> The main concept of physics is iffy, and the idea that we can make things behave in a certain way. The most important part of physics is called quantum mechanics which states that all particles are made up of energy (energy) and matter (matter). In physics, there are two types of particles: elementary particles and exotic ones. These particles have properties like mass, speed or momentum but they don’t interact with each other to form new objects. This is because these particles do not exist independently from one another. In this case, an exotic particle might be created by adding more energy into its structure than it would take for a normal particle. However, when you add additional energy to an exotic particle, the new object will become smaller and larger until it becomes too large to fit within the existing structure. If you think about how light travels through space, it takes around 20 billion years before the light reaches our eyes. Light waves travel faster than light at high speeds so if we could create some kind of light wave, then we wouldn’t need any special equipment. It just needs a few hundred millionths of a second to produce light rays. So even though the light is moving along the same path as the current, the speed of light is different depending on where the light hits the

**Prompt:** `"Artificial intelligence is "`
> Artificial intelligence is iffy, it can be used to make intelligent machines that could take over the world. What does Artificial Intelligence mean? AI refers to artificial intelligence and machine learning technology which is a type of computer science (also known as artificial intelligence) in which computers are programmed with knowledge about their environment or other objects. The term AI comes from the Greek word "art" meaning "to create." The most common uses for AI include: - Machine Learning This means using algorithms like natural language processing systems to learn how words work together to form sentences such as “I am going to go to the store.” These programs will then use these rules to decide whether they should buy something or not so that you know what’s being sold on the internet. For example, if you purchase an ebook at Amazon, you may want to check its price first before purchasing it. If this happens, your shopping cart might look different than it did when purchased by someone else who bought it earlier. You can also think of AI as a way to help people understand themselves better through training and reasoning rather than simply seeing them doing things differently. In fact, we often see AI models working very well because of the way humans interact with our minds. This ability makes us more effective

**Prompt:** `"Once upon a time, "`
> Once upon a time, ...... I was so excited about the new school year and wanted to make some changes in my life. I had been looking for ways to help me become more self-aware. As an adult, I have always felt that there is no one way of doing things without thinking first. This has led me to start making small changes at home or at work. One such change was to create a space where I could be more mindful and aware of myself as well as other people around me. It’s important to remember that we all need our own personal growth and development. We can do this by taking responsibility for ourselves; being responsible for what happens outside us and keeping it within our control. By creating these smaller steps towards becoming more conscious of yourself, you will see how much better your future looks!The word "treaty" means something like "a treaty made with a king." The French word for "covenant," célèbre (French: cœle), comes from the Latin cecus ("to give up"). A covenant is not a binding agreement but rather an act of mutual understanding between two parties. In general terms, a contract is anything agreed on which someone agrees to agree to receive certain benefits. For example, if a person

---

## 📄 License

This model is released under the [Apache 2.0 License](https://www.apache.org/licenses/LICENSE-2.0).

---

## Third-party validation benchmarks by Datdanboi25
![image](https://cdn-uploads.huggingface.co/production/uploads/697f2832c2c5e4daa93cece7/nB4fwHYhilGEyxwa9DWTD.png)

---

*© SupraLabs 2026 — Project Chimera*