File size: 1,566 Bytes
e2d46d2 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 | ---
license: apache-2.0
library_name: vla-foundry
tags:
- foundry
- vla_foundry
- llm
- text-generation
---
# Foundry-LLM-1.2B-800B
A 1.2B parameter language model pretrained on 800B tokens, part of the [VLA Foundry](https://github.com/TRI-ML/vla_foundry) model collection.
## Model Description
- **Architecture:** Transformer (24 layers, 2048 hidden dim, 16 heads, SwiGLU FFN, RoPE, QK-norm)
- **Parameters:** 1.2B (non-embedding)
- **Tokenizer:** SmolVLM2 (vocab size 49,280)
- **Training data:** 800B tokens from DCLM-Baseline-1.0
- **LR schedule:** Warmup + constant (no decay)
- **Sequence length:** 2048
Earlier checkpoint of the Foundry LLM, used as the language backbone for the downstream VLM and VLA models.
## Evaluation Results
Multiple-choice reasoning benchmarks:
| HellaSwag | MMLU | ARC-e | ARC-c | PIQA | WinoGrande | OpenBookQA | BoolQ |
|---|---|---|---|---|---|---|---|
| 64.3 | 26.0 | 70.3 | 37.0 | 75.8 | 60.9 | 40.0 | 63.2 |
## Usage
```bash
git clone https://github.com/TRI-ML/vla_foundry.git
cd vla_foundry
pip install -e .
```
```python
from vla_foundry.models.base_model import BaseModel
model = BaseModel.from_pretrained("TRI-ML/Foundry-LLM-1.2B-800B")
```
## Links
- **Project page:** [tri-ml.github.io/vla_foundry](https://tri-ml.github.io/vla_foundry/)
- **Paper:** [VLA Foundry (arXiv 2604.19728)](https://arxiv.org/abs/2604.19728)
- **Code:** [github.com/TRI-ML/vla_foundry](https://github.com/TRI-ML/vla_foundry)
- **Collection:** [VLA Foundry collection](https://huggingface.co/collections/TRI-ML/vla-foundry)
|