README.md

---
language:
  - en
  - de
  - es
  - fr
tags:
  - pretraining
  - multilingual
  - language-model
---

# MonoWeb Models

Pretrained language models released alongside the paper:

> **[The Role of Mixed-Language Documents for Multilingual Large Language Model Pretraining](https://arxiv.org/pdf/2601.00364)**


Associated dataset: [UCLNLP/monoweb-dataset](https://huggingface.co/datasets/UCLNLP/monoweb-dataset)

## Model Details

All models are decoder-only transformers with **1.35B parameters**, trained from scratch using the Llama-2 tokenizer (32K vocabulary). Architecture: 24 layers, hidden dimension 2048, 16 attention heads, context length 2048. Training was performed with Megatron-LM for ~143B tokens (34K steps).

## Model Variants

Models are organized by language pair and training data configuration:

| Folder | Language Pair | Training Data |
|---|---|---|
| `ckpt_exp_en_de_baseline` | English–German | FineWeb (full corpus, including bilingual docs) |
| `ckpt_exp_en_de_monoweb` | English–German | MonoWeb (bilingual docs removed) |
| `ckpt_exp_en_de_onlyparallel` | English–German | MonoWeb + parallel docs reintroduced |
| `ckpt_exp_en_de_onlycodeswitch` | English–German | MonoWeb + code-switching docs reintroduced |
| `ckpt_exp_en_es_baseline` | English–Spanish | FineWeb (full corpus, including bilingual docs) |
| `ckpt_exp_en_es_monoweb` | English–Spanish | MonoWeb (bilingual docs removed) |
| `ckpt_exp_en_es_onlyparallel` | English–Spanish | MonoWeb + parallel docs reintroduced |
| `ckpt_exp_en_es_onlycodeswitch` | English–Spanish | MonoWeb + code-switching docs reintroduced |
| `ckpt_exp_en_fr_baseline` | English–French | FineWeb (full corpus, including bilingual docs) |
| `ckpt_exp_en_fr_monoweb` | English–French | MonoWeb (bilingual docs removed) |
| `ckpt_exp_en_fr_onlyparallel` | English–French | MonoWeb + parallel docs reintroduced |
| `ckpt_exp_en_fr_onlycodeswitch` | English–French | MonoWeb + code-switching docs reintroduced |

Each folder contains checkpoints saved every 2,000 steps from `iter_2000` to `iter_36000` (18 checkpoints per model).