Upload README.md with huggingface_hub

README.md

@@ -5,17 +5,187 @@ license: apache-2.0
 library_name: transformers
 tags:
 - biomedical
+- clinical
 - encoder
 - modernbert
 - fill-mask
+base_model:
+- answerdotai/ModernBERT-base
 pipeline_tag: fill-mask
+widget:
+- text: "The patient was diagnosed with [MASK] and started on antibiotics."
+- text: "Mitochondria is the powerhouse of the [MASK]."
 ---
 
 # cpt-en-base
 
-Stealth upload. Model card will be updated before public release.
+*cpt-en-base is available in two sizes: [base](https://huggingface.co/rntc/cpt-en-base) (149M parameters) and [large](https://huggingface.co/almanach/cpt-en-large) (396M parameters). Our code is available in our [GitHub repository](https://github.com/Rian-T/colm2026-clm-detour).*
 
-- Architecture: ModernBERT (22 layers, 768 hidden, 12 heads)
-- Parameters: 149M
-- Context: 8192 tokens
-- Language: English
+## Table of Contents
+
+1. [Model Summary](#model-summary)
+2. [Usage](#usage)
+3. [Training](#training)
+4. [Evaluation](#evaluation)
+5. [License](#license)
+6. [Citation](#citation)
+
+## Model Summary
+
+cpt-en-base is an English biomedical encoder built by continued pretraining of [ModernBERT](https://huggingface.co/answerdotai/ModernBERT-base) using a **CLM detour** recipe. Instead of standard MLM continued pretraining, we temporarily switch to causal language modeling (CLM) before returning to MLM. This produces lasting representational changes in early transformer layers that improve downstream biomedical performance.
+
+cpt-en-base achieves **78.0% average F1** across 11 English biomedical benchmarks (5 Clinical + 6 BigBIO), the highest balanced score across both task families.
+
+| | |
+|---|---|
+| **Architecture** | ModernBERT (FlashAttention, RoPE, alternating local/global attention, unpadding) |
+| **Parameters** | 149M |
+| **Layers** | 22 |
+| **Hidden size** | 768 |
+| **Attention heads** | 12 |
+| **Context length** | 8,192 tokens |
+| **Language** | English |
+| **Base model** | [answerdotai/ModernBERT-base](https://huggingface.co/answerdotai/ModernBERT-base) |
+
+## Usage
+
+You can use this model with the `transformers` library (v4.48.0+):
+
+```bash
+pip install -U transformers>=4.48.0
+```
+
+If your GPU supports it, install Flash Attention for best efficiency:
+
+```bash
+pip install flash-attn
+```
+
+### Masked Language Modeling
+
+```python
+from transformers import AutoTokenizer, AutoModelForMaskedLM
+
+model_id = "rntc/cpt-en-base"
+tokenizer = AutoTokenizer.from_pretrained(model_id)
+model = AutoModelForMaskedLM.from_pretrained(model_id)
+
+text = "The patient was diagnosed with [MASK] and started on antibiotics."
+inputs = tokenizer(text, return_tensors="pt")
+outputs = model(**inputs)
+
+masked_index = inputs["input_ids"][0].tolist().index(tokenizer.mask_token_id)
+predicted_token_id = outputs.logits[0, masked_index].argmax(axis=-1)
+predicted_token = tokenizer.decode(predicted_token_id)
+print("Predicted token:", predicted_token)
+```
+
+### Fine-tuning (Classification, NER, etc.)
+
+```python
+from transformers import AutoTokenizer, AutoModel
+
+model_id = "rntc/cpt-en-base"
+tokenizer = AutoTokenizer.from_pretrained(model_id)
+model = AutoModel.from_pretrained(model_id)
+
+text = "The patient presented with acute myocardial infarction and was treated with percutaneous coronary intervention."
+inputs = tokenizer(text, return_tensors="pt", max_length=8192, truncation=True)
+outputs = model(**inputs)
+# outputs.last_hidden_state: [batch, seq_len, 768]
+```
+
+**Note:** cpt-en-base does not use token type IDs. You can omit the `token_type_ids` parameter.
+
+## Training
+
+### Data
+
+| Corpus | Proportion | Description |
+|--------|------------|-------------|
+| PubMed | 60% | Biomedical abstracts |
+| Med-Inst | 20% | Medical instructions |
+| MIMIC | 20% | Clinical notes |
+| **Total** | **50B tokens** | Single epoch |
+
+### Methodology
+
+cpt-en-base is trained in two phases, initialized from [ModernBERT-base](https://huggingface.co/answerdotai/ModernBERT-base):
+
+* **Phase 1 — CLM detour (50B tokens):** The bidirectional attention mask is replaced with a causal mask, and the model is trained with next-token prediction. This dense training signal (100% of positions) deeply modifies early transformer layers for domain adaptation.
+* **Phase 2 — MLM decay (5B tokens):** Bidirectional attention is restored, and the model is trained with masked language modeling at 15% masking. The learning rate decays from peak to 10% following a 1-sqrt schedule.
+
+Both phases use the same data mix. Training used AdamW (lr=2e-4, beta1=0.9, beta2=0.98), bf16 mixed precision, global batch size of 384 sequences (~3.1M tokens), on 4x H100 GPUs with [Composer](https://github.com/mosaicml/composer).
+
+### Why a CLM Detour?
+
+CLM supervises every token position, producing dense gradient updates that deeply modify early transformer layers (layers 0-7). These changes persist through the MLM decay phase — a phenomenon we call **computational hysteresis**. We provide causal evidence through freeze interventions showing that early-layer modification is both necessary and sufficient for the CLM benefit (double dissociation). See our paper for the full mechanistic analysis.
+
+## Evaluation
+
+English biomedical benchmark results (11 tasks, 5 seeds per model):
+
+### Clinical Tasks
+
+| Model | Ctx | ChemProt | Phenotype | COS | Social Hist. | DEID | **Avg** |
+|-------|-----|----------|-----------|-----|-------------|------|---------|
+| **cpt-en-base** | 8192 | 90.1 | **61.9** | **95.2** | 54.2 | **83.2** | **76.9** |
+| BioClinical-ModernBERT | 8192 | 90.0 | 60.7 | 94.8 | **56.0** | 81.8 | 76.7 |
+| PubMedBERT | 512 | **90.2** | 52.0 | 95.0 | 48.7 | 80.4 | 73.3 |
+| ModernBERT-base | 8192 | 89.5 | 48.4 | 94.0 | 53.1 | 78.3 | 72.7 |
+
+### BigBIO Tasks
+
+| Model | Ctx | AnatEM | BC5CDR | JNLPBA | NCBI | GAD | HoC | **Avg** |
+|-------|-----|--------|--------|--------|------|-----|-----|---------|
+| **cpt-en-base** | 8192 | 81.0 | **89.1** | 74.5 | 80.1 | 78.8 | **70.0** | **78.9** |
+| BioClinical-ModernBERT | 8192 | 79.2 | 88.7 | 74.8 | 78.7 | 75.8 | 67.0 | 77.4 |
+| PubMedBERT | 512 | **83.3** | 89.7 | **74.9** | **82.1** | **79.3** | 71.0 | 80.1 |
+| ModernBERT-base | 8192 | 77.2 | 87.9 | 74.3 | 77.7 | 76.8 | 66.6 | 76.8 |
+
+### Overall
+
+| Model | Clinical | BigBIO | **Overall** |
+|-------|----------|--------|-------------|
+| **cpt-en-base** | **76.9** | **78.9** | **78.0** |
+| BioClinical-ModernBERT | 76.7 | 77.4 | 77.0 |
+| PubMedBERT | 73.3 | 80.1 | 77.0 |
+| ModernBERT-base | 72.7 | 76.8 | 74.9 |
+
+cpt-en-base achieves the highest balanced score (78.0%) across both Clinical and BigBIO task families. PubMedBERT scores higher on short-context BigBIO NER tasks but falls behind on long-context tasks (Phenotype: 52.0% vs 61.9%).
+
+## Intended Use
+
+This model is designed for English biomedical and clinical NLP tasks:
+- Named entity recognition (diseases, chemicals, genes, anatomy)
+- Document classification (clinical phenotyping, relation extraction)
+- De-identification of clinical notes
+- Information extraction from PubMed abstracts and clinical reports
+
+The 8,192-token context is important for long clinical documents (discharge summaries, pathology reports) that are truncated by 512-token models.
+
+## Limitations
+
+- Trained on English biomedical text; not suitable for other languages without further adaptation. See [cpt-fr-base](https://huggingface.co/almanach/cpt-fr-base-base) for French.
+- Encoder model: produces contextualized representations, does not generate text.
+- Clinical text may contain sensitive patterns; users are responsible for compliance with applicable regulations (HIPAA, etc.).
+- The English CLM-MLM improvement (+0.3pp at Base scale) is smaller than in French (+2.9pp) and not statistically significant at Base scale (binomial p=0.27). The practical benefit is clearest at Large scale (+0.8pp) and on long-context tasks.
+
+## License
+
+Apache 2.0
+
+## Citation
+
+```bibtex
+@inproceedings{touchent2026clm,
+  title={A Causal Language Modeling Detour Improves Encoder Continued Pretraining},
+  author={Touchent, Rian and de la Clergerie, {\'E}ric},
+  booktitle={Proceedings of COLM},
+  year={2026}
+}
+```
+
+## Acknowledgments
+
+This work was performed using HPC resources from GENCI-IDRIS (Grant 2024-AD011015883).