Update README.md

README.md

@@ -1,3 +1,57 @@
----
-license: mit
----
+---
+license: mit
+datasets:
+- Harley-ml/es-en-words
+language:
+- en
+tags:
+- small
+- small-language-model
+- largeword
+- word-generation
+- harley-ml
+- word
+- words
+- wordgen
+- qwen3
+---
+
+# LargeWord
+
+LargeWord is the largest model in the [WordGen] family and has about 1.59M parameters.
+LargeWord has an instruct version [here].
+
+LargeWord generates pluasible or real words learned from its pretraining dataset.
+
+## Architecture
+
+| Parameter               | Value |
+|-------------------------|-------|
+| hidden_size             | 160   |
+| num_hidden_layers       | 4     |
+| num_attention_heads     | 2     |
+| num_key_value_heads     | 2     |
+| intermediate_size       | 512   |
+| max_position_embeddings | 77    |
+| rope_theta              | 10000.0 |
+| tie_word_embeddings     | True  |
+| vocab_size              | 1204  |
+
+## Training
+
+LargeWord trained on 753,232 words and 4,153,110 tokens. Its goal is to generate plausible-looking or real words.
+
+### Hardware
+
+LargeWord was trained on a NVIDIA RTX 2060 6GB for 2 epochs with a batch size of 8.
+
+### Training Results
+
+| Step | Epoch | Train Loss | Train PPL | Eval Loss | Eval PPL |
+|------|-------|------------|-----------|-----------|----------|
+| 500  | 0.30  | 4.3276     | 75.74     | 2.4190    | 11.23    |
+| 1000 | 0.61  | 1.7151     | 5.56      | 1.4076    | 4.09     |
+| 1500 | 0.91  | 1.3247     | 3.76      | 1.2682    | 3.55     |
+| 2000 | 1.21  | 1.2120     | 3.36      | 1.2026    | 3.33     |
+| 2500 | 1.51  | 1.1619     | 3.20      | 1.1667    | 3.21     |
+| 3000 | 1.82  | 1.1314     | 3.10      | 1.1378    | 3.12     |