Update model card: add model-index, datasets, training details, related models

README.md

@@ -9,16 +9,94 @@ tags:
 - encoder
 - modernbert
 - fill-mask
+datasets:
+- rntc/biomed-enriched
 base_model:
 - answerdotai/ModernBERT-base
 pipeline_tag: fill-mask
 widget:
 - text: "The patient was diagnosed with [MASK] and started on antibiotics."
 - text: "Mitochondria is the powerhouse of the [MASK]."
+model-index:
+- name: cpt-en-base
+  results:
+  - task:
+      type: token-classification
+      name: NER
+    dataset:
+      name: AnatEM
+      type: bigbio/anatem
+    metrics:
+    - type: f1
+      value: 81.0
+  - task:
+      type: token-classification
+      name: NER
+    dataset:
+      name: BC5CDR
+      type: bigbio/bc5cdr
+    metrics:
+    - type: f1
+      value: 89.1
+  - task:
+      type: token-classification
+      name: NER
+    dataset:
+      name: JNLPBA
+      type: bigbio/jnlpba
+    metrics:
+    - type: f1
+      value: 74.5
+  - task:
+      type: token-classification
+      name: NER
+    dataset:
+      name: NCBI Disease
+      type: bigbio/ncbi_disease
+    metrics:
+    - type: f1
+      value: 80.1
+  - task:
+      type: text-classification
+      name: Text Classification
+    dataset:
+      name: GAD
+      type: bigbio/gad
+    metrics:
+    - type: f1
+      value: 78.8
+  - task:
+      type: text-classification
+      name: Text Classification
+    dataset:
+      name: HoC
+      type: bigbio/hallmarks_of_cancer
+    metrics:
+    - type: f1
+      value: 70.0
+  - task:
+      type: text-classification
+      name: Text Classification
+    dataset:
+      name: ChemProt
+      type: bigbio/chemprot
+    metrics:
+    - type: f1
+      value: 90.1
+  - task:
+      type: text-classification
+      name: Text Classification
+    dataset:
+      name: DEID
+      type: n2c2/2006-deid
+    metrics:
+    - type: f1
+      value: 83.2
 ---
 
 # cpt-en-base
 
+*cpt-en is available in two sizes: [base](https://huggingface.co/rntc/cpt-en-base) (149M parameters) and [large](https://huggingface.co/rntc/cpt-en-large) (396M parameters). Our code will be released upon publication.*
 
 ## Table of Contents
 
@@ -31,9 +109,9 @@ widget:
 
 ## Model Summary
 
-cpt-en-base is an English biomedical encoder built by continued pretraining of [ModernBERT](https://huggingface.co/answerdotai/ModernBERT-base) using a **CLM detour** recipe. Instead of standard MLM continued pretraining, we temporarily switch to causal language modeling (CLM) before returning to MLM. This produces lasting representational changes in early transformer layers that improve downstream biomedical performance.
+cpt-en is an English biomedical encoder built by continued pretraining of [ModernBERT](https://huggingface.co/answerdotai/ModernBERT-base) using a **CLM detour** recipe. Instead of standard MLM continued pretraining, we temporarily switch to causal language modeling (CLM) before returning to MLM. This produces lasting representational changes in early transformer layers that improve downstream biomedical performance.
 
-cpt-en-base achieves **78.0% average F1** across 11 English biomedical benchmarks (5 Clinical + 6 BigBIO), the highest balanced score across both task families.
+cpt-en achieves **78.0% average F1** across 11 English biomedical benchmarks (5 Clinical + 6 BigBIO), the highest balanced score across both task families.
 
 | | |
 |---|---|
@@ -94,7 +172,7 @@ outputs = model(**inputs)
 # outputs.last_hidden_state: [batch, seq_len, 768]
 ```
 
-**Note:** cpt-en-base does not use token type IDs. You can omit the `token_type_ids` parameter.
+**Note:** cpt-en does not use token type IDs. You can omit the `token_type_ids` parameter.
 
 ## Training
 
@@ -105,7 +183,7 @@ outputs = model(**inputs)
 | PubMed | 60% | Biomedical abstracts |
 | Med-Inst | 20% | Medical instructions |
 | MIMIC | 20% | Clinical notes |
-| **Total** | **50B tokens** | Single epoch |
+| **Total** | **50B tokens** | |
 
 ### Methodology
 
@@ -114,7 +192,7 @@ cpt-en-base is trained in two phases, initialized from [ModernBERT-base](https:/
 * **Phase 1 — CLM detour (50B tokens):** The bidirectional attention mask is replaced with a causal mask, and the model is trained with next-token prediction. This dense training signal (100% of positions) deeply modifies early transformer layers for domain adaptation.
 * **Phase 2 — MLM decay (5B tokens):** Bidirectional attention is restored, and the model is trained with masked language modeling at 15% masking. The learning rate decays from peak to 10% following a 1-sqrt schedule.
 
-Both phases use the same data mix. Training used AdamW (lr=2e-4, beta1=0.9, beta2=0.98), bf16 mixed precision, global batch size of 384 sequences (~3.1M tokens), on 4x H100 GPUs with [Composer](https://github.com/mosaicml/composer).
+Both phases use the same data mix (55B tokens total). Training used AdamW (lr=2e-4, beta1=0.9, beta2=0.98), bf16 mixed precision, global batch size of 384 sequences (~3.1M tokens), on 4× H100 80GB GPUs with [Composer](https://github.com/mosaicml/composer). Total training time: ~5 GPU-hours.
 
 ### Why a CLM Detour?
 
@@ -129,7 +207,7 @@ English biomedical benchmark results (11 tasks, 5 seeds per model):
 | Model | Ctx | ChemProt | Phenotype | COS | Social Hist. | DEID | **Avg** |
 |-------|-----|----------|-----------|-----|-------------|------|---------|
 | **cpt-en-base** | 8192 | 90.1 | **61.9** | **95.2** | 54.2 | **83.2** | **76.9** |
-| BioClinical-ModernBERT | 8192 | 90.0 | 60.7 | 94.8 | **56.0** | 81.8 | 76.7 |
+| BioClinical-ModernBERT-base | 8192 | 90.0 | 60.7 | 94.8 | **56.0** | 81.8 | 76.7 |
 | PubMedBERT | 512 | **90.2** | 52.0 | 95.0 | 48.7 | 80.4 | 73.3 |
 | ModernBERT-base | 8192 | 89.5 | 48.4 | 94.0 | 53.1 | 78.3 | 72.7 |
 
@@ -138,7 +216,7 @@ English biomedical benchmark results (11 tasks, 5 seeds per model):
 | Model | Ctx | AnatEM | BC5CDR | JNLPBA | NCBI | GAD | HoC | **Avg** |
 |-------|-----|--------|--------|--------|------|-----|-----|---------|
 | **cpt-en-base** | 8192 | 81.0 | **89.1** | 74.5 | 80.1 | 78.8 | **70.0** | **78.9** |
-| BioClinical-ModernBERT | 8192 | 79.2 | 88.7 | 74.8 | 78.7 | 75.8 | 67.0 | 77.4 |
+| BioClinical-ModernBERT-base | 8192 | 79.2 | 88.7 | 74.8 | 78.7 | 75.8 | 67.0 | 77.4 |
 | PubMedBERT | 512 | **83.3** | 89.7 | **74.9** | **82.1** | **79.3** | 71.0 | 80.1 |
 | ModernBERT-base | 8192 | 77.2 | 87.9 | 74.3 | 77.7 | 76.8 | 66.6 | 76.8 |
 
@@ -147,7 +225,7 @@ English biomedical benchmark results (11 tasks, 5 seeds per model):
 | Model | Clinical | BigBIO | **Overall** |
 |-------|----------|--------|-------------|
 | **cpt-en-base** | **76.9** | **78.9** | **78.0** |
-| BioClinical-ModernBERT | 76.7 | 77.4 | 77.0 |
+| BioClinical-ModernBERT-base | 76.7 | 77.4 | 77.0 |
 | PubMedBERT | 73.3 | 80.1 | 77.0 |
 | ModernBERT-base | 72.7 | 76.8 | 74.9 |
 
@@ -163,9 +241,18 @@ This model is designed for English biomedical and clinical NLP tasks:
 
 The 8,192-token context is important for long clinical documents (discharge summaries, pathology reports) that are truncated by 512-token models.
 
+## Related Models
+
+| Model | Language | Parameters |
+|-------|----------|------------|
+| [cpt-en-base](https://huggingface.co/rntc/cpt-en-base) | English | 149M |
+| [cpt-en-large](https://huggingface.co/rntc/cpt-en-large) | English | 396M |
+| [cpt-fr-base](https://huggingface.co/rntc/cpt-fr-base) | French | 150M |
+| [cpt-fr-large](https://huggingface.co/rntc/cpt-fr-large) | French | 350M |
+
 ## Limitations
 
-- Trained on English biomedical text; not suitable for other languages without further adaptation. See [cpt-fr-base](https://huggingface.co/almanach/cpt-fr-base-base) for French.
+- Trained on English biomedical text; not suitable for other languages without further adaptation. See [cpt-fr](https://huggingface.co/rntc/cpt-fr-base) for French.
 - Encoder model: produces contextualized representations, does not generate text.
 - Clinical text may contain sensitive patterns; users are responsible for compliance with applicable regulations (HIPAA, etc.).
 - The English CLM-MLM improvement (+0.3pp at Base scale) is smaller than in French (+2.9pp) and not statistically significant at Base scale (binomial p=0.27). The practical benefit is clearest at Large scale (+0.8pp) and on long-context tasks.
@@ -178,13 +265,13 @@ Apache 2.0
 
 ```bibtex
 @inproceedings{anonymous2026clm,
-  title={Under review},
+  title={A Causal Language Modeling Detour Improves Encoder Continued Pretraining},
   author={Anonymous},
-  booktitle={Under review},
+  booktitle={Proceedings of COLM},
   year={2026}
 }
 ```
 
 ## Acknowledgments
 
-
+This work was performed using HPC resources.