Create README.md

README.md

@@ -0,0 +1,100 @@
+---
+license: apache-2.0
+datasets:
+- HuggingFaceFW/fineweb-edu
+language:
+- en
+pipeline_tag: text-generation
+library_name: transformers
+tags:
+- small
+- cpu
+- supra
+- v2
+- tiny
+- mini
+- open
+- open-source
+---
+
+# 🦅 Supra Mini v2 0.1M
+Supra Mini v2 0.1M is a very tiny base model trained on 700 million tokens of Fineweb-Edu for 3 epochs as the second version of our Supra Mini series.
+
+## Model Config
+
+- Parameters: 167,760 (0.1M)
+- Architecture: Llama
+- Vocab size with custom BPE tokenizer: 2048
+- Hidden Size: 48
+- Intermediate Size: 96
+- Hidden Layers: 3
+- Attention Heads: 4
+- Max Position Embeddings: 256
+- Learning rate: 6e-4
+- Weight Decay: 0.01
+
+## Final Loss
+This model reached a final train loss after 3 epochs of **4.XYZ**.
+
+## Benchmarks
+
+All benchmarks were executed using `lm-eval`.
+
+| Task          | Value        | Random level |
+| :------------ | :----------: | -----------: |
+| Arc_Easy      | 0.XXXX       | 0.25 (25%)   |
+| Wikitext      | XX.XXXX      | -            |
+| BLiMP         | 0.XXXX       | 0.5 (50%)    |
+
+## Examples
+**Prompt:** "Artificial intelligence is "<br>
+**Output:**: "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
+<br><br>
+**Prompt:** "The main concept of physics is "<br>
+**Output:**: "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
+<br><br>
+**Prompt:** "Once upon a time, "<br>
+**Output:**: "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
+
+## Usage
+To use our model, just run this code using HF Transformers to execute the model:
+```python3
+from transformers import pipeline
+import torch
+
+print("[*] Loading Supra Mini v2 0.1M model from Hugging Face Hub...")
+pipe = pipeline(
+    "text-generation", 
+    model="SupraLabs/Supra-Mini-v2-0.1M",
+    device_map="auto",
+    torch_dtype=torch.float16 if torch.cuda.is_available() else torch.float32
+)
+
+def generate_text(prompt, max_length=150):
+    result = pipe(
+        prompt, 
+        max_new_tokens=max_length,
+        do_sample=True,
+        temperature=0.5,
+        top_k=25,
+        top_p=0.9,
+        repetition_penalty=1.2,
+        pad_token_id=pipe.tokenizer.pad_token_id,
+        eos_token_id=pipe.tokenizer.eos_token_id
+    )
+    return result[0]['generated_text']
+
+test_prompt = "The importance of education is"
+print(f"\nPrompt: {test_prompt}")
+print("-" * 30)
+print("\nOutput:\n" + generate_text(test_prompt))
+```
+
+## Training guide
+We trained Supra Mini v2 0.1M on a single T4 GPU in ~2 hours for 3 epochs.<br>
+The full training code can be found in this repo as `run.sh` (easily run the complete pipeline), `train_tokenizer.py` (train costum BPE tokenizer with vocab size of 2048), `train.py` (train the model) and `inference.py` (test the model).<br>
+The model was trained on the first 700 million tokens of Sample-10BT from Fineweb-Edu using streaming tokenization.
+
+## Final thoughts
+As this is the second version of the Supra Mini series, we are very proud to release it today!<br>
+*But:* stay tuned for more models and follow us to support our open-source work! 😊