Update README: state-of-the-art biomedical encoder release

README.md

@@ -10,7 +10,7 @@ tags:
 - modernbert
 - fill-mask
 datasets:
-- rntc/biomed-enriched
+- almanach/Biomed-Enriched
 base_model:
 - answerdotai/ModernBERT-base
 pipeline_tag: fill-mask
@@ -18,7 +18,7 @@ widget:
 - text: "The patient was diagnosed with [MASK] and started on antibiotics."
 - text: "Mitochondria is the powerhouse of the [MASK]."
 model-index:
-- name: cpt-en-base
+- name: ModernBERT-bio-base
   results:
   - task:
       type: token-classification
@@ -94,9 +94,9 @@ model-index:
       value: 83.2
 ---
 
-# cpt-en-base
+# ModernBERT-bio-base
 
-*cpt-en is available in two sizes: [base](https://huggingface.co/rntc/cpt-en-base) (149M parameters) and [large](https://huggingface.co/rntc/cpt-en-large) (396M parameters). Our code will be released upon publication.*
+*ModernBERT-bio is available in two sizes: [base](https://huggingface.co/almanach/ModernBERT-bio-base) (149M parameters) and [large](https://huggingface.co/almanach/ModernBERT-bio-large) (396M parameters). Our code is available in our [GitHub repository](https://github.com/Rian-T/colm2026-clm-detour).*
 
 ## Table of Contents
 
@@ -109,9 +109,9 @@ model-index:
 
 ## Model Summary
 
-cpt-en is an English biomedical encoder built by continued pretraining of [ModernBERT](https://huggingface.co/answerdotai/ModernBERT-base) using a **CLM detour** recipe. Instead of standard MLM continued pretraining, we temporarily switch to causal language modeling (CLM) before returning to MLM. This produces lasting representational changes in early transformer layers that improve downstream biomedical performance.
+ModernBERT-bio is an English biomedical encoder built by continued pretraining of [ModernBERT](https://huggingface.co/answerdotai/ModernBERT-base) using a **CLM detour** recipe. Instead of standard MLM continued pretraining, we temporarily switch to causal language modeling (CLM) before returning to MLM. This produces lasting representational changes in early transformer layers that improve downstream biomedical performance.
 
-cpt-en achieves **78.0% average F1** across 11 English biomedical benchmarks (5 Clinical + 6 BigBIO), the highest balanced score across both task families.
+ModernBERT-bio achieves **78.0% average F1** across 11 English biomedical benchmarks (5 Clinical + 6 BigBIO), the highest balanced score across both task families.
 
 | | |
 |---|---|
@@ -143,7 +143,7 @@ pip install flash-attn
 ```python
 from transformers import AutoTokenizer, AutoModelForMaskedLM
 
-model_id = "rntc/cpt-en-base"
+model_id = "almanach/ModernBERT-bio-base"
 tokenizer = AutoTokenizer.from_pretrained(model_id)
 model = AutoModelForMaskedLM.from_pretrained(model_id)
 
@@ -162,7 +162,7 @@ print("Predicted token:", predicted_token)
 ```python
 from transformers import AutoTokenizer, AutoModel
 
-model_id = "rntc/cpt-en-base"
+model_id = "almanach/ModernBERT-bio-base"
 tokenizer = AutoTokenizer.from_pretrained(model_id)
 model = AutoModel.from_pretrained(model_id)
 
@@ -172,7 +172,7 @@ outputs = model(**inputs)
 # outputs.last_hidden_state: [batch, seq_len, 768]
 ```
 
-**Note:** cpt-en does not use token type IDs. You can omit the `token_type_ids` parameter.
+**Note:** ModernBERT-bio does not use token type IDs. You can omit the `token_type_ids` parameter.
 
 ## Training
 
@@ -187,7 +187,7 @@ outputs = model(**inputs)
 
 ### Methodology
 
-cpt-en-base is trained in two phases, initialized from [ModernBERT-base](https://huggingface.co/answerdotai/ModernBERT-base):
+ModernBERT-bio-base is trained in two phases, initialized from [ModernBERT-base](https://huggingface.co/answerdotai/ModernBERT-base):
 
 * **Phase 1 — CLM detour (50B tokens):** The bidirectional attention mask is replaced with a causal mask, and the model is trained with next-token prediction. This dense training signal (100% of positions) deeply modifies early transformer layers for domain adaptation.
 * **Phase 2 — MLM decay (5B tokens):** Bidirectional attention is restored, and the model is trained with masked language modeling at 15% masking. The learning rate decays from peak to 10% following a 1-sqrt schedule.
@@ -206,7 +206,7 @@ English biomedical benchmark results (11 tasks, 5 seeds per model):
 
 | Model | Ctx | ChemProt | Phenotype | COS | Social Hist. | DEID | **Avg** |
 |-------|-----|----------|-----------|-----|-------------|------|---------|
-| **cpt-en-base** | 8192 | 90.1 | **61.9** | **95.2** | 54.2 | **83.2** | **76.9** |
+| **ModernBERT-bio-base** | 8192 | 90.1 | **61.9** | **95.2** | 54.2 | **83.2** | **76.9** |
 | BioClinical-ModernBERT-base | 8192 | 90.0 | 60.7 | 94.8 | **56.0** | 81.8 | 76.7 |
 | PubMedBERT | 512 | **90.2** | 52.0 | 95.0 | 48.7 | 80.4 | 73.3 |
 | ModernBERT-base | 8192 | 89.5 | 48.4 | 94.0 | 53.1 | 78.3 | 72.7 |
@@ -215,7 +215,7 @@ English biomedical benchmark results (11 tasks, 5 seeds per model):
 
 | Model | Ctx | AnatEM | BC5CDR | JNLPBA | NCBI | GAD | HoC | **Avg** |
 |-------|-----|--------|--------|--------|------|-----|-----|---------|
-| **cpt-en-base** | 8192 | 81.0 | **89.1** | 74.5 | 80.1 | 78.8 | **70.0** | **78.9** |
+| **ModernBERT-bio-base** | 8192 | 81.0 | **89.1** | 74.5 | 80.1 | 78.8 | **70.0** | **78.9** |
 | BioClinical-ModernBERT-base | 8192 | 79.2 | 88.7 | 74.8 | 78.7 | 75.8 | 67.0 | 77.4 |
 | PubMedBERT | 512 | **83.3** | 89.7 | **74.9** | **82.1** | **79.3** | 71.0 | 80.1 |
 | ModernBERT-base | 8192 | 77.2 | 87.9 | 74.3 | 77.7 | 76.8 | 66.6 | 76.8 |
@@ -224,12 +224,12 @@ English biomedical benchmark results (11 tasks, 5 seeds per model):
 
 | Model | Clinical | BigBIO | **Overall** |
 |-------|----------|--------|-------------|
-| **cpt-en-base** | **76.9** | **78.9** | **78.0** |
+| **ModernBERT-bio-base** | **76.9** | **78.9** | **78.0** |
 | BioClinical-ModernBERT-base | 76.7 | 77.4 | 77.0 |
 | PubMedBERT | 73.3 | 80.1 | 77.0 |
 | ModernBERT-base | 72.7 | 76.8 | 74.9 |
 
-cpt-en-base achieves the highest balanced score (78.0%) across both Clinical and BigBIO task families. PubMedBERT scores higher on short-context BigBIO NER tasks but falls behind on long-context tasks (Phenotype: 52.0% vs 61.9%).
+ModernBERT-bio-base achieves the highest balanced score (78.0%) across both Clinical and BigBIO task families. PubMedBERT scores higher on short-context BigBIO NER tasks but falls behind on long-context tasks (Phenotype: 52.0% vs 61.9%).
 
 ## Intended Use
 
@@ -245,14 +245,14 @@ The 8,192-token context is important for long clinical documents (discharge summ
 
 | Model | Language | Parameters |
 |-------|----------|------------|
-| [cpt-en-base](https://huggingface.co/rntc/cpt-en-base) | English | 149M |
-| [cpt-en-large](https://huggingface.co/rntc/cpt-en-large) | English | 396M |
-| [cpt-fr-base](https://huggingface.co/rntc/cpt-fr-base) | French | 150M |
-| [cpt-fr-large](https://huggingface.co/rntc/cpt-fr-large) | French | 350M |
+| [ModernBERT-bio-base](https://huggingface.co/almanach/ModernBERT-bio-base) | English | 149M |
+| [ModernBERT-bio-large](https://huggingface.co/almanach/ModernBERT-bio-large) | English | 396M |
+| [ModernCamemBERT-bio-base](https://huggingface.co/almanach/ModernCamemBERT-bio-base) | French | 150M |
+| [ModernCamemBERT-bio-large](https://huggingface.co/almanach/ModernCamemBERT-bio-large) | French | 350M |
 
 ## Limitations
 
-- Trained on English biomedical text; not suitable for other languages without further adaptation. See [cpt-fr](https://huggingface.co/rntc/cpt-fr-base) for French.
+- Trained on English biomedical text; not suitable for other languages without further adaptation. See [ModernCamemBERT-bio](https://huggingface.co/almanach/ModernCamemBERT-bio-base) for French.
 - Encoder model: produces contextualized representations, does not generate text.
 - Clinical text may contain sensitive patterns; users are responsible for compliance with applicable regulations (HIPAA, etc.).
 - The English CLM-MLM improvement (+0.3pp at Base scale) is smaller than in French (+2.9pp) and not statistically significant at Base scale (binomial p=0.27). The practical benefit is clearest at Large scale (+0.8pp) and on long-context tasks.
@@ -264,9 +264,9 @@ Apache 2.0
 ## Citation
 
 ```bibtex
-@inproceedings{anonymous2026clm,
+@inproceedings{touchent2026clm,
   title={A Causal Language Modeling Detour Improves Encoder Continued Pretraining},
-  author={Anonymous},
+  author={Touchent, Rian and de la Clergerie, {\'E}ric},
   booktitle={Proceedings of COLM},
   year={2026}
 }
@@ -274,4 +274,4 @@ Apache 2.0
 
 ## Acknowledgments
 
-This work was performed using HPC resources.
+This work was performed using HPC resources from GENCI-IDRIS (Grant 2024-AD011015883).