Update README.md
Browse files
README.md
CHANGED
|
@@ -30,7 +30,8 @@ Author: Paul Courneya (Harley-ml)
|
|
| 30 |
|
| 31 |
## **Description**
|
| 32 |
|
| 33 |
-
Dillion is a 1.2M parameter language model trained on ~9B tokens of FineWeb-edu.
|
|
|
|
| 34 |
Dillion beats or ties with models much larger than itself such as SupraMini-v4-2M and Tenete-8M.
|
| 35 |
|
| 36 |
## Architecture
|
|
|
|
| 30 |
|
| 31 |
## **Description**
|
| 32 |
|
| 33 |
+
Dillion is a 1.2M parameter language model trained on ~9B tokens of FineWeb-edu.
|
| 34 |
+
Our goal was to make one of the best sub-1.5M parameter LMs through depth (12 layers!) and huge overtraining (~8900 tokens per parameter.)
|
| 35 |
Dillion beats or ties with models much larger than itself such as SupraMini-v4-2M and Tenete-8M.
|
| 36 |
|
| 37 |
## Architecture
|