Add model card for SuperApriel-15b-Base

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

.gitattributes

@@ -34,3 +34,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
 tokenizer.json filter=lfs diff=lfs merge=lfs -text
+*.png filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,189 @@
+---
+license: mit
+pipeline_tag: text-generation
+library_name: transformers
+track_downloads: true
+---
+
+# SuperApriel-15b-Base
+
+<img src="assets/super-apriel.png" width="120" alt="thumbnail"/>      `/ˈɑː.pri.əl/`
+
+A 15B-parameter **token-mixer supernet** derived from [Apriel-1.6](https://huggingface.co/ServiceNow-AI/Apriel-1.6-15b-Thinker) via stochastic distillation. Every decoder layer exposes **four trained mixer options**—Full Attention, Sliding Window Attention, Gated DeltaNet, and Kimi Delta Attention—enabling flexible architecture selection from a single checkpoint.
+
+- **Model Size:** 15B parameters
+- **Layers:** 48 decoder layers, each with 4 mixer variants
+- **Context Length:** 262K positions (runtime dependent)
+- **Languages:** English (best)
+
+## Highlights
+
+- **Supernet architecture**: Single checkpoint containing 4 mixer types at every layer, yielding 4⁴⁸ ≈ 7.9 × 10²⁸ possible architectures
+- **Four mixer types**: Full Attention (FA), Sliding Window Attention (SWA, window=4096), Gated DeltaNet (GDN), Kimi Delta Attention (KDA)
+- **Stage 1 distillation checkpoint**: Trained via stochastic distillation from frozen Apriel-1.6 teacher on 266B tokens
+- **Foundation for fine-tuning**: Use this checkpoint to fine-tune on your own data with targeted placement strategies
+
+## Model Overview
+
+SuperApriel-15b-Base is the **Stage 1 (distillation) checkpoint** of the Super Apriel supernet. During training, all four mixer types at each layer were trained simultaneously using stochastic local sampling—each layer's mixer was drawn uniformly from the four types at each training step. Only mixer weights were trained; all shared parameters (FFNs, embeddings, layer norms, vision encoder) remain frozen from the Apriel-1.6 teacher.
+
+This checkpoint is intended as a **foundation for downstream fine-tuning**. For a ready-to-use model with optimized deployment presets, see [SuperApriel-15b-Instruct](https://huggingface.co/ServiceNow-AI/SuperApriel-15b-Instruct).
+
+### Architecture Details
+
+| Component | Details |
+|-----------|---------|
+| Parameters | 15B |
+| Decoder layers | 48 |
+| Query / KV heads | 32 / 8 (grouped-query attention), d_h = 128 |
+| Hidden dimension | 5,120 |
+| FFN width | 14,336 (SiLU-gated) |
+| Vocabulary | 131,072 tokens |
+| Vision encoder | Pixtral (16×16 patches) |
+
+### Mixer Types
+
+| Mixer | Time | Memory | Description |
+|-------|------|--------|-------------|
+| Full Attention (FA) | O(n²) | O(n) KV cache | Standard grouped-query attention |
+| Sliding Window (SWA) | O(w·n) | O(w) | Local window of 4,096 tokens |
+| Gated DeltaNet (GDN) | O(n) | O(1) fixed state | Matrix-valued recurrent state with delta rule |
+| Kimi Delta Attention (KDA) | O(n) | O(1) fixed state | Linear attention with channel-wise gating |
+
+### Training Details
+
+- **Objective**: Stochastic distillation from frozen Apriel-1.6 teacher
+- **Losses**: Activation matching (𝓛_act), Forward KL (weight 0.1), Reverse KL (weight 0.9)
+- **Data**: 266B tokens, curated mixture focused on reasoning and domain-specific data
+- **Sampling**: Uniform local sampling (each layer independently samples a mixer type)
+- **Compute**: Up to 192 H100 GPUs
+- **Training framework**: [Fast-LLM](https://github.com/ServiceNow/Fast-LLM)
+
+## How to Use
+
+Install dependencies:
+
+```bash
+pip install transformers
+```
+
+> **🔴 TODO: There is currently no mechanism to select a placement when using Transformers directly. The model defaults to the all-attention preset during inference. Placement selection requires vLLM with the Fast-LLM plugin (see below). We need to add a Transformers API for placement switching.**
+
+Basic usage with Transformers (all-attention preset):
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+model_name = "ServiceNow-AI/SuperApriel-15b-Base"
+
+tokenizer = AutoTokenizer.from_pretrained(model_name)
+model = AutoModelForCausalLM.from_pretrained(
+    model_name,
+    torch_dtype="auto",
+    device_map="auto",
+    trust_remote_code=True,
+)
+
+prompt = "The capital of France is"
+inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
+
+generated_ids = model.generate(**inputs, max_new_tokens=64)
+output = tokenizer.decode(generated_ids[0], skip_special_tokens=True)
+print(output)
+```
+
+> **Note:** This base model requires `trust_remote_code=True` as it uses custom architecture code for the multi-mixer supernet.
+
+## Use with vLLM
+
+> **🔴 TODO: Confirm the exact vLLM plugin source (Fast-LLM branch/tag?), installation steps, and the CLI/API for selecting a placement. The instructions below are placeholders.**
+
+The supernet is served via a vLLM plugin implemented in [Fast-LLM](https://github.com/ServiceNow/Fast-LLM). Two serving modes are available:
+
+- **`single-preset` mode**: Only the weights of a single selected mixer placement are loaded. Inactive mixer weights are offloaded to CPU, so GPU memory footprint and throughput match a dedicated single-placement model.
+- **`supernet` mode**: The full supernet is loaded into memory, enabling placement switching at a per-request level (5–15s switch time depending on how many layers change mixer type).
+
+### Installation
+
+```bash
+uv venv --python 3.12 --seed
+source .venv/bin/activate
+
+git clone git@github.com:ServiceNow/Fast-LLM.git
+cd Fast-LLM
+uv pip install vllm==0.10.2 --torch-backend=auto
+pip install .
+```
+
+### Running a vLLM Server
+
+```bash
+vllm serve \
+  --model ServiceNow-AI/SuperApriel-15b-Base \
+  --port 8000 \
+  --trust-remote-code
+```
+
+## Intended Use
+
+SuperApriel-15b-Base is designed as a **foundation checkpoint** for:
+
+- Fine-tuning with custom placement strategies on domain-specific data
+- Research on hybrid architectures and mixer placement optimization
+- Placement search and Pareto frontier exploration using the optimization toolkit
+
+It is **not intended** for direct deployment without further fine-tuning or for safety-critical applications without human oversight.
+
+## Limitations
+
+- **Factual accuracy:** May produce incorrect, misleading, or outdated content. Outputs should be verified before use in critical contexts.
+- **Bias:** May reflect societal, cultural, or systemic biases present in training data.
+- **Ethics:** Do not use the model to produce harmful, unlawful, or unethical content.
+- **Language:** Strongest performance is in English. Output quality may degrade in underrepresented languages.
+- **Critical use:** Not suitable for medical, legal, financial, or other high-risk applications without safeguards.
+- **Base model:** This is a distillation checkpoint without instruction tuning. For instruction-following use cases, see [SuperApriel-15b-Instruct](https://huggingface.co/ServiceNow-AI/SuperApriel-15b-Instruct).
+
+## Security and Responsible Use
+
+**Security Responsibilities:**
+Deployers and users are strongly encouraged to align their security practices with established frameworks and regulatory guidelines such as the EU AI Act and the NIST AI Risk Management Framework (RMF).
+
+**Guidelines for Deployers:**
+
+- Regularly conduct robustness assessments to identify and mitigate adversarial inputs.
+- Implement validation and filtering processes to prevent harmful or biased outputs.
+- Continuously perform data privacy checks to guard against unintended data leaks.
+- Document and communicate the model's limitations, intended usage, and known security risks to all end-users.
+- Schedule periodic security reviews and updates to address emerging threats and vulnerabilities.
+
+**Guidelines for Users:**
+
+- Follow established security policies and usage guidelines provided by deployers.
+- Protect and manage sensitive information when interacting with the model.
+- Report anomalies, suspicious behavior, or unsafe outputs to deployers or developers.
+- Maintain human oversight and apply judgment to mitigate potential security or ethical risks during interactions.
+
+**Disclaimer:**
+Users accept responsibility for securely deploying, managing, and using this open-source LLM. The model is provided "as-is," without explicit or implied warranty regarding security or fitness for any specific application or environment.
+
+## Software
+
+- **Training stack:** [Fast-LLM](https://github.com/ServiceNow/Fast-LLM)
+- **Serving:** [Fast-LLM vLLM plugin](https://github.com/ServiceNow/Fast-LLM)
+
+## License
+
+MIT
+
+## Citation
+
+```bibtex
+@misc{super_apriel_2025,
+  title        = {Super Apriel: One Checkpoint, Many Speeds},
+  author       = {ServiceNow Language Models Lab},
+  year         = {2026},
+  eprint       = {TODO},
+  archivePrefix= {arXiv},
+  primaryClass = {cs.CL}
+}
+```