Create README.md
Browse files
README.md
ADDED
|
@@ -0,0 +1,54 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: apache-2.0
|
| 3 |
+
datasets:
|
| 4 |
+
- roneneldan/TinyStories
|
| 5 |
+
language:
|
| 6 |
+
- en
|
| 7 |
+
pipeline_tag: text-generation
|
| 8 |
+
library_name: transformers
|
| 9 |
+
tags:
|
| 10 |
+
- small
|
| 11 |
+
- tiny
|
| 12 |
+
- story
|
| 13 |
+
- tinystories
|
| 14 |
+
- roneneldan
|
| 15 |
+
- cpu
|
| 16 |
+
- free
|
| 17 |
+
- open-source
|
| 18 |
+
---
|
| 19 |
+
|
| 20 |
+
# 📖 StorySupra 10M
|
| 21 |
+
|
| 22 |
+
## Config
|
| 23 |
+
- Parameters: 12,587,264 (~10M)
|
| 24 |
+
- Hidden Size: 256
|
| 25 |
+
- Intermediate Size: 1024
|
| 26 |
+
- Hidden Layers: 8
|
| 27 |
+
- Attention Heads: 8
|
| 28 |
+
- Max Position Embeddings: 256
|
| 29 |
+
- Vocab Size: 8192
|
| 30 |
+
|
| 31 |
+
## Samples
|
| 32 |
+
Once upon a time , a small bird was flying in the sky . It saw a big tree and wanted to rest under it . But the tree was too high for the bird to reach . The bird tried to fly up , but it could not . Then , a wise old owl flew by and saw the bird struggling . The owl said , " Don ' t worry little bird , I can help you ." The owl used its strong beak to climb the tree and get the bird down . The bird was
|
| 33 |
+
<br><br>
|
| 34 |
+
Once upon a time , there was a little boy named Timmy . He loved to play with his toys and run around outside . One day , he found a shiny penny on the ground . It was so pretty that he picked it up and showed it to his mom . " Look , Mommy ! I found a penny !" he said . His mom smiled and said , " That ' s great , Timmy . But be careful , it ' s very special ." Timmy didn ' t understand what " valuable " meant , but he knew it meant something important . So
|
| 35 |
+
<br><br>
|
| 36 |
+
Once upon a time , there was a lovely princess . She had long , blonde hair and a sparkly crown . One day , she wanted to go for a walk in the forest . She put on her dress and started walking . As she walked , she saw something strange . It was a big , scary bear ! The princess was scared , but she didn ' t want to get away . So she just kept walking until she reached the forest . When she got there , she saw a little rabbit . He was wearing a bright red bow and he looked very friendly .
|
| 37 |
+
|
| 38 |
+
## Training
|
| 39 |
+
- GPU: single RTX 5060 Ti 16GB
|
| 40 |
+
- Time: ~20 minutes
|
| 41 |
+
- Epochs: 3
|
| 42 |
+
- Samples of the dataset: 200k
|
| 43 |
+
|
| 44 |
+
## Dataset
|
| 45 |
+
200k samples of roneneldan/TinyStories
|
| 46 |
+
|
| 47 |
+
## Code
|
| 48 |
+
You can find the full code in this repo as `train.py` and inference.py. Have fun :-)
|
| 49 |
+
|
| 50 |
+
## Usage
|
| 51 |
+
Use this to run the model:
|
| 52 |
+
```python3
|
| 53 |
+
|
| 54 |
+
```
|