| --- |
| license: apache-2.0 |
| library_name: vla-foundry |
| tags: |
| - foundry |
| - vla_foundry |
| - llm |
| - text-generation |
| --- |
| |
| # Foundry-LLM-1.2B-1T |
|
|
| A 1.2B parameter language model pretrained on 1T tokens, part of the [VLA Foundry](https://github.com/TRI-ML/vla_foundry) collection. |
|
|
| ## Model Description |
|
|
| - **Architecture:** Transformer (24 layers, 2048 hidden dim, 16 heads, SwiGLU FFN, RoPE, QK-norm) |
| - **Parameters:** 1.2B (non-embedding) |
| - **Tokenizer:** SmolVLM2 (vocab size 49,280) |
| - **Training data:** 1T tokens from DCLM-Baseline-1.0 |
| - **LR schedule:** Warmup + constant for 800B tokens, then 200B tokens of cosine decay |
| - **Sequence length:** 2048 |
|
|
| Continuation of [Foundry-LLM-1.2B-800B](https://huggingface.co/TRI-ML/Foundry-LLM-1.2B-800B) with an additional 200B tokens of cosine-decayed training. |
|
|
| ## Evaluation Results |
|
|
| Multiple-choice reasoning benchmarks: |
|
|
| | HellaSwag | MMLU | ARC-e | ARC-c | PIQA | WinoGrande | OpenBookQA | BoolQ | |
| |---|---|---|---|---|---|---|---| |
| | 66.7 | 26.6 | 71.7 | 39.3 | 77.5 | 62.6 | 40.8 | 65.4 | |
|
|
| ## Usage |
|
|
| ```bash |
| git clone https://github.com/TRI-ML/vla_foundry.git |
| cd vla_foundry |
| pip install -e . |
| ``` |
|
|
| ```python |
| from vla_foundry.models.base_model import BaseModel |
| model = BaseModel.from_pretrained("TRI-ML/Foundry-LLM-1.2B-1T") |
| ``` |
|
|
| ## Links |
|
|
| - **Project page:** [tri-ml.github.io/vla_foundry](https://tri-ml.github.io/vla_foundry/) |
| - **Paper:** [VLA Foundry (arXiv 2604.19728)](https://arxiv.org/abs/2604.19728) |
| - **Code:** [github.com/TRI-ML/vla_foundry](https://github.com/TRI-ML/vla_foundry) |
| - **Collection:** [VLA Foundry collection](https://huggingface.co/collections/TRI-ML/vla-foundry) |
|
|
