| --- |
| license: mit |
| datasets: |
| - HuggingFaceFW/fineweb-edu |
| language: |
| - en |
| pipeline_tag: text-generation |
| tags: |
| - micro |
| - nano |
| - small |
| - supra |
| - SupraLabs |
| - gtx |
| - rtx |
| - nvidia |
| - llama |
| - lhtech |
| - axionlab |
| library_name: transformers |
| --- |
| |
| ## **MicroSupra-1k** |
|
|
| So... have you ever seen a model that runs on a 3 dollars hardware? No? If no, Now you're seeing! |
|
|
| MicroSupra-1k is a 1k parameters model trained on 300M Tokens of FineWeb-Edu for 1 minute(Yes! 59 seconds!) on a GTX 750Ti 4GB(AxionLab Hardware) |
|
|
| **Some model outputs:** |
|
|
| [*] Prompt: The main concept of physics is |
| |
| [*] Output: The main concept of physics is a,s and the. thet to, theing.... the,a then,c,i to, thee in b. toed.,,e theyalp the in,er thees- s,el,,,, |
| and, the of ine,,s the of cs of thesss the. f. to. thesining andor dar,,al the,. of p. |
| the.s the.,,s. anded,e. of, ofed, l toinging and themsr the of of. to |
| to thes thes aen,., ofes of a. |
|
|
| [*] Prompt: My name is |
| |
| [*] Output: My name is ed and. as the, to. the, iningt thee the ofingi in |
| the., anda.-eo |
| ofles, b the,er,s fing.ssp the the |
| , of of, the,al, d to the m, the, to toed, |
| seng,,.y. in the,., in and them the thened.sing to |
| the of of andan the the,, the |
| to..,,sing,,.aring the the. of.al.,s ofcal ar s..e and.sssor of, and and. |
|
|
| [*] Prompt: Question: What is the capital of France?\nAnswer: |
| |
| [*] Output: Question: What is the capital of France? |
|
|
| Answer:,. and to the. toc. ofs the m,a thee.. the, f ofling. as.,,y bt, the p, in, the,,ees toed ing to.o, |
| thes. the..,s the.ed and andang,,ed the of,,ms. of, thei the, the,ey,,s l.ing toe the the,se the to, the, the,aror, the of-. in the. the. the,e the of ds to,ic the the aal at the.. |
| ingssy s and and |
|
|
| **🚫What the model CAN'T do:** |
| Think |
| Chat |
| even predict the next token correctly lol |
|
|
| **Why SupraLabs created this???** |
|
|
| Because we are experimenting sizes, experiments(like 1Bit quant, distillation(NEW THINGS ARE COMING WITH DISTILLATION! GET TUNED!), pruning, all to better your experience! We are working to big things!) |
|
|
| **Final thought** |
|
|
| Even without any intelligence, it shows that scaling laws are real. This ant model doesn't know how to talk, but we all know it emotions 🤖🫶 |
|
|
| SupraLabs are excited to work at more things to you! |
