Create README.md

README.md

@@ -0,0 +1,92 @@
+---
+license: other
+language:
+- en
+library_name: transformers
+pipeline_tag: text-generation
+tags:
+- moe
+- mixture-of-experts
+- modularity
+- index
+datasets:
+- allenai/OLMoE-mix-0924
+---
+
+# EMO: Pretraining Mixture of Experts for Emergent Modularity
+
+This page is an **index** for the model checkpoints released alongside [EMO: Pretraining Mixture of Experts for Emergent Modularity](https://arxiv.org/abs/2605.06663). The repository at `allenai/EMO` does not host model weights — pick the checkpoint you want from the table below.
+
+## Released models
+
+### Main release
+
+| Model | Description |
+|---|---|
+| [`allenai/Emo_1b14b_1T`](https://huggingface.co/allenai/Emo_1b14b_1T) | **EMO** — 1B-active / 14B-total MoE pretrained on 1T tokens + 50B-token midtraining anneal. The main model from the paper. |
+
+### Ablation: EMO at smaller scale
+
+| Model | Description |
+|---|---|
+| [`allenai/Emo_1b14b_130B`](https://huggingface.co/allenai/Emo_1b14b_130B) | EMO trained on 130B tokens (Table 1 / Figure 11 ablation). Not midtrained. |
+
+### Architecture-matched standard MoE baselines
+
+These share architecture and data with the EMO models above; only the training objective differs (no document-level expert pool constraint).
+
+| Model | Description |
+|---|---|
+| [`allenai/StdMoE_1b14b_1T`](https://huggingface.co/allenai/StdMoE_1b14b_1T) | Standard MoE — *Reg. MoE* at 1T tokens in the paper. Same setup as `Emo_1b14b_1T`. |
+| [`allenai/StdMoE_1b14b_130B`](https://huggingface.co/allenai/StdMoE_1b14b_130B) | Standard MoE — *Reg. MoE* at 130B tokens. Same setup as `Emo_1b14b_130B`. |
+
+### Memory-matched baselines (Figure 1)
+
+Smaller models trained from scratch at fixed memory budgets, used as comparison points for EMO expert subsets.
+
+| Model | Description |
+|---|---|
+| [`allenai/Dense_1b_130B`](https://huggingface.co/allenai/Dense_1b_130B) | **Dense @ 8** — 1B dense decoder-only Transformer trained on 130B tokens. Active-parameter-matched with 8-expert subsets of the larger EMO/StdMoE models. |
+| [`allenai/StdMoE_1b4b_130B`](https://huggingface.co/allenai/StdMoE_1b4b_130B) | **Reg. MoE @ 32** — 1B-active / 4B-total standard MoE (32 routed experts) trained from scratch on 130B tokens. Memory-matched with 32-expert subsets. |
+
+### EMO-anneal ablation (Appendix B.4)
+
+Tests whether modularity can be induced *after* pretraining by annealing a standard MoE under the EMO objective.
+
+| Model | Description |
+|---|---|
+| [`allenai/StdMoE_1b14b_1T_Preanneal`](https://huggingface.co/allenai/StdMoE_1b14b_1T_Preanneal) | Standard MoE pretrained on 1T tokens, no annealing. The starting point for the EMO-anneal experiment. |
+| [`allenai/StdMoE_1b14b_1T_EmoAnnealed`](https://huggingface.co/allenai/StdMoE_1b14b_1T_EmoAnnealed) | **EMO-anneal** — `StdMoE_1b14b_1T_Preanneal` annealed for 50B tokens under the EMO document-level expert pool objective. |
+
+## Quick start
+
+All checkpoints require `trust_remote_code=True` since they use custom modeling code from the [ryanyxw/transformers](https://github.com/ryanyxw/transformers/tree/flexmoe_v4_57_1) fork. Replace `model_id` with the checkpoint you want from the table above.
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+model_id = "allenai/Emo_1b14b_1T"  # main EMO release
+model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True)
+tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
+
+inputs = tokenizer(["Language modeling is "], return_tensors="pt", return_token_type_ids=False)
+out = model.generate(**inputs, max_new_tokens=100, do_sample=True, temperature=1.0, top_p=0.7)
+print(tokenizer.batch_decode(out, skip_special_tokens=True)[0])
+```
+
+## Citation
+
+```bibtex
+@article{wang2026emo,
+  title  = {EMO: Pretraining Mixture of Experts for Emergent Modularity},
+  author = {Wang, Ryan and Bhagia, Akshita and Min, Sewon},
+  year   = {2026},
+  url    = {https://arxiv.org/abs/2605.06663}
+}
+```
+
+## Links
+
+- Paper: https://arxiv.org/abs/2605.06663
+- Code: https://github.com/allenai/EMO
+- Visualization: https://emovisualization.netlify.app