File size: 2,403 Bytes
b4de408 dd5b725 e4b0d95 dd5b725 3bcc458 0eb1c3c 3bcc458 cb4be93 a801075 cb4be93 0eb1c3c cb4be93 0eb1c3c a801075 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 | ---
license: apache-2.0
datasets:
- HuggingFaceFW/fineweb-edu
language:
- en
base_model:
- SupraLabs/Supra-Mini-v4-2M
pipeline_tag: text-generation
tags:
- supralabs
- supra
- cpu
- gpu
- distill
- base
- sub-1m
- axionlab
- lh-tech
---
## DistillSupra-0.2M
--------------------------------------------------
**DistillSupra-0.2M** is an ultra-compact causal language model with approximately **0.2 million parameters**, produced by knowledge distillation from [Supra-Mini-v4-2M](https://huggingface.co/SupraLabs/Supra-Mini-v4-2M).
It was trained 500 steps(1 Epoch) for 30 minutes on a GTX 750 Ti 4GB using generated text from the teacher.
The model was **10x** compressed! That's crazy!
## Architecture
| Parameter | Teacher | Student |
|---------------------|---------|---------|
| hidden_size | 64 | 48 |
| intermediate_size | 128 | 96 |
| num_hidden_layers | 5 | 4 |
| num_attention_heads | 8 | 6 |
| vocab_size | 4096 | 4096 |
| Parameters | ~468k | ~289k |
## Some outputs:
Prompt : Throughout history, great civilizations
--------------------------------------------------
Output: Throughout history, great civilizations to in, a be polrain for is with more the the be the for. of be of on (I.er The b M.A-R and or have that not is and the is this they, can for to to. is of a a, to ofs the for and the a. in the is to as of is that an that of and you the which is, the, for in be a are by’ of. and to a m
Prompt : The human brain is capable of
--------------------------------------------------
Output: The human brain is capable ofs in an more that in a new can is the this the a of the pS, the a to the other in not it... and with a to that be are of to for in of of ass. The be of the,.F-s be the of dLal. ins of be and of Sin: and or that a one that to and a a bFed, asRal., the, is a and as
Prompt : The most important principle in science is
--------------------------------------------------
The most important principle in science is a is a this are not for that the to of be digels-LC. to the in a the to, on to,
## Why did supra created this trash?
We are currently researching knowledge distillation and this was the first step! Things will better up!
## Final Thought
Knowledge distillation is a promising thing for us, we believe that LLMs can be helpful even being so small!
|