francescofiamingo1/ff-125m-cloud

FF-125M (Cloud Trained)

Author: Francesco Fiamingo
Base architecture: GPT-2-like (125 M parameters)
Training dataset: English CC-News (cleaned & deduplicated)
Tokenizer: ByteLevel BPE – 32 k vocab
Training hardware: NVIDIA A100 80 GB (RunPod)
Loss function: Causal Language Modeling
Final Perplexity: ≈ 2.8
Frameworks: PyTorch | Transformers | W&B | AWS S3

---

🧠 Model Overview

FF-125M is a compact, didactic LLM trained from scratch to reproduce GPT-2-style behavior while maintaining modern training stability.
It serves as the baseline for the upcoming FF-7B bilingual model.

---

⚙️ Usage Example

from transformers import GPT2LMHeadModel, GPT2TokenizerFast
model = GPT2LMHeadModel.from_pretrained("francescofiamingo1/ff-125m-cloud")
tokenizer = GPT2TokenizerFast.from_pretrained("francescofiamingo1/ff-125m-cloud")

prompt = "In a surprising discovery,"
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=60)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))