| --- |
| datasets: |
| - starhopp3r/TinyChat |
| language: |
| - en |
| pipeline_tag: text-generation |
| tags: |
| - tiny |
| - chat |
| - small |
| - cpu |
| - open |
| - open-source |
| - gpt2 |
| - gpt |
| --- |
| |
| # Welcome to 🤏 Pin Model Series |
| The models in the LH-Tech AI Pin Series are very small models that were trained on starhopp3r/TinyChat. |
|
|
| ## Models overview |
| | Model | Parameters | Training iters | Final Train Loss | Quality | Example Chat | |
| |:----|:----|:----|:----|:----|:----| |
| | Pin-5M | 5.37M | 1000 | 3.170788 | Very Poor | Yes, a bright day is shining and makes everything have a good day a lot. | |
| | Pin-10M | 10.06M | 1500 | 2.562048 | Very Poor | That sounds nice, I agree, it is nice to talk about new ideas. | |
| | Pin-15M | 14.84M | 1500 | 2.358367 | Low | It is hard to see your plans when you want to enjoy the day. | |
| | Pin-20M | 21.03M | 1500 | 2.217588 | Medium | Yes, sunny days are wonderful! I love hearing about the sunshine and the sun's shining on. | |
| | Pin-25M | 26.76M | 1500 | 2.139837 | Medium | Sunny days make everything look brighter, especially with a nice friend who cares. | |
| | Pin-Ultra-25M | 26.76M | 8000 | 1.588281 | Hight | Sunny days are wonderful, they really lift our spirits and make us smile. | |
|
|
| \* All models were prompted with `What is the weather like today?`. |
|
|
| 🚀 **We recommend using Pin-Ultra-25M.** |
|
|
| ## Training |
| ### Training data |
| We trained on [starhopp3r/TinyChat](https://huggingface.co/datasets/starhopp3r/TinyChat) and used the gpt-2 tokenizer. |
|
|
| ### Training code/scripts |
| You can find the full training code for the Pin Model Series in this repo.<br> |
| **Tip:** If you want to train one of these models yourself, make sure to adjust the model config like this: |
| | Model | n\_layer | n\_head | n\_embd | n\_inner | |
| | :---- | :---- | :---- | :---- | :---- | |
| | **Pin-5M** | 4 | 8 | 96 | 384 | |
| | **Pin-10M** | 6 | 8 | 160 | 640 | |
| | **Pin-15M** | *8* | *8* | *208* | *832* | |
| | **Pin-20M** | *10* | *8* | *256* | *1024* | |
| | **Pin-25M** | *12* | *12* | *288* | *1152* | |
| | **Pin-Ultra-25M** | *12* | *12* | *288* | *1152* | |
|
|
| Have fun :D |
|
|
| ### Training details |
| We trained all these models in ~30 minutes on Kaggle 2xT4 GPU in a Kaggle Session (except for Pin-Ultra-25 - it trained for ~4 hours on Kaggle 2xT4).<br> |
| So you are able to easily recreate all of the Pin model without having to launch a 8xH100 cluster 😂 |
|
|
| ## How to use the model |
| You can easily use the favorite model of the Pin series like this: |
|
|
| 1. Download `use.py` from this repo. |
| 2. Adjust the subfolder argument here: |
| ```python |
| answer = run_pin_inference(user_query, model_id="LH-Tech-AI/Pin", subfolder="Pin-Ultra-25M") # use your favorite model here, e.g. "Pin-25M" or "Pin-15M"... |
| ``` |
| 3. Adjust the input prompt here: |
| ```python |
| user_query = "What is the weather like today?" # insert your prompt here |
| ``` |
| 4. Launch it with Torch installed in Python3. |
| 5. Have fun :D |
|
|
| ## Acknowledgements... |
| ...to: |
|
|
| - HF Transformers |
| - Kaggle for the 2xT4 GPU |
| - starhopp3r for his TinyChat dataset |
