Update README: state-of-the-art biomedical encoder release

README.md

@@ -18,7 +18,7 @@ widget:
 - text: "Les patients atteints de <mask> présentent un risque accru de complications cardiovasculaires."
 - text: "Le traitement par <mask> a montré une amélioration significative des symptômes."
 model-index:
-- name: cpt-fr-base
+- name: ModernCamemBERT-bio-base
   results:
   - task:
       type: text-classification
@@ -82,7 +82,7 @@ model-index:
       type: emea
     metrics:
     - type: f1
-      value: 65.9
+      value: 68.6
   - task:
       type: token-classification
       name: NER
@@ -91,12 +91,12 @@ model-index:
       type: medline
     metrics:
     - type: f1
-      value: 58.2
+      value: 61.9
 ---
 
-# cpt-fr-base
+# ModernCamemBERT-bio-base
 
-*cpt-fr is available in two sizes: [base](https://huggingface.co/rntc/cpt-fr-base) (150M parameters) and [large](https://huggingface.co/rntc/cpt-fr-large) (350M parameters). Our code will be released upon publication.*
+*ModernCamemBERT-bio is available in two sizes: [base](https://huggingface.co/almanach/ModernCamemBERT-bio-base) (150M parameters) and [large](https://huggingface.co/almanach/ModernCamemBERT-bio-large) (350M parameters). Our code is available in our [GitHub repository](https://github.com/Rian-T/colm2026-clm-detour).*
 
 ## Table of Contents
 
@@ -109,7 +109,7 @@ model-index:
 
 ## Model Summary
 
-cpt-fr is a French biomedical encoder built by continued pretraining of [ModernCamemBERT](https://huggingface.co/almanach/moderncamembert-base) using a **CLM detour** recipe. Instead of standard MLM continued pretraining, we temporarily switch to causal language modeling (CLM) before returning to MLM. This produces lasting representational changes in early transformer layers that improve downstream biomedical performance by +2.9pp on average across 8 French biomedical tasks.
+ModernCamemBERT-bio is a French biomedical encoder built by continued pretraining of [ModernCamemBERT](https://huggingface.co/almanach/moderncamembert-base) using a **CLM detour** recipe. Instead of standard MLM continued pretraining, we temporarily switch to causal language modeling (CLM) before returning to MLM. This produces lasting representational changes in early transformer layers that improve downstream biomedical performance by +2.8pp on average across 8 French biomedical tasks.
 
 The model uses the ModernBERT architecture with FlashAttention, rotary positional embeddings (RoPE), alternating local/global attention, and unpadding, supporting **8,192-token context** — critical for long clinical documents that exceed the 512-token limit of previous French biomedical models.
 
@@ -143,7 +143,7 @@ pip install flash-attn
 ```python
 from transformers import AutoTokenizer, AutoModelForMaskedLM
 
-model_id = "rntc/cpt-fr-base"
+model_id = "almanach/ModernCamemBERT-bio-base"
 tokenizer = AutoTokenizer.from_pretrained(model_id)
 model = AutoModelForMaskedLM.from_pretrained(model_id)
 
@@ -162,7 +162,7 @@ print("Predicted token:", predicted_token)
 ```python
 from transformers import AutoTokenizer, AutoModel
 
-model_id = "rntc/cpt-fr-base"
+model_id = "almanach/ModernCamemBERT-bio-base"
 tokenizer = AutoTokenizer.from_pretrained(model_id)
 model = AutoModel.from_pretrained(model_id)
 
@@ -172,7 +172,7 @@ outputs = model(**inputs)
 # outputs.last_hidden_state: [batch, seq_len, 768]
 ```
 
-**Note:** cpt-fr does not use token type IDs. You can omit the `token_type_ids` parameter.
+**Note:** ModernCamemBERT-bio does not use token type IDs. You can omit the `token_type_ids` parameter.
 
 ## Training
 
@@ -188,7 +188,7 @@ outputs = model(**inputs)
 
 ### Methodology
 
-cpt-fr is trained in two phases, initialized from [ModernCamemBERT](https://huggingface.co/almanach/moderncamembert-base):
+ModernCamemBERT-bio is trained in two phases, initialized from [ModernCamemBERT](https://huggingface.co/almanach/moderncamembert-base):
 
 * **Phase 1 — CLM detour (10B tokens):** The bidirectional attention mask is replaced with a causal mask, and the model is trained with next-token prediction. This dense training signal (100% of positions) deeply modifies early transformer layers for domain adaptation.
 * **Phase 2 — MLM decay (1B tokens):** Bidirectional attention is restored, and the model is trained with masked language modeling at 15% masking. The learning rate decays from peak to 10% following a 1-sqrt schedule.
@@ -205,13 +205,13 @@ French biomedical benchmark results (8 tasks, 9 seeds per model, macro-averaged
 
 | Model | Ctx | FrACCO-30 | FrACCO-100 | CANTEMIST | DISTEMIST | MedDialog | DiaMed | EMEA | Medline | **Avg** |
 |-------|-----|-----------|------------|-----------|-----------|-----------|--------|------|---------|---------|
-| **cpt-fr-base** | 8192 | **74.8** | **60.1** | **71.0** | **25.5** | 63.6 | **67.4** | 65.9 | 58.2 | **60.8** |
-| MLM baseline (ours) | 8192 | 69.9 | 56.8 | 64.9 | 23.5 | 62.5 | 63.4 | 65.4 | 56.8 | 57.9 |
-| ModernCamemBERT | 8192 | 70.1 | 55.3 | 63.3 | 20.2 | 60.6 | 56.4 | 63.4 | 55.3 | 55.6 |
-| DrBERT | 512 | 53.0 | 35.6 | 37.9 | 21.4 | 63.6 | 57.0 | **68.0** | **62.3** | 49.9 |
-| CamemBERT-bio | 512 | 41.9 | 20.1 | 12.8 | 9.6 | 38.6 | 47.7 | 61.6 | 56.6 | 36.1 |
+| **ModernCamemBERT-bio-base** | 8192 | **74.8** | **60.1** | **71.0** | **25.5** | 63.6 | **67.4** | 68.6 | 61.9 | **61.6** |
+| MLM baseline (ours) | 8192 | 69.9 | 56.8 | 64.9 | 23.5 | 62.5 | 63.4 | 68.5 | 61.4 | 58.9 |
+| ModernCamemBERT | 8192 | 70.1 | 55.3 | 63.3 | 20.2 | 60.6 | 56.4 | 68.0 | 59.7 | 56.7 |
+| DrBERT | 512 | 53.0 | 35.6 | 37.9 | 21.4 | 63.6 | 57.0 | **69.6** | **62.8** | 50.1 |
+| CamemBERT-bio | 512 | 41.9 | 20.1 | 12.8 | 9.6 | 38.6 | 47.7 | **70.8** | **65.2** | 38.3 |
 
-cpt-fr-base outperforms the matched MLM baseline on all 8 tasks (+2.9pp, binomial p=0.004).
+ModernCamemBERT-bio-base outperforms the matched MLM baseline on all 8 tasks (+2.8pp, binomial p=0.004).
 
 ## Intended Use
 
@@ -227,10 +227,10 @@ The 8,192-token context is critical for long clinical documents (discharge summa
 
 | Model | Language | Parameters |
 |-------|----------|------------|
-| [cpt-en-base](https://huggingface.co/rntc/cpt-en-base) | English | 149M |
-| [cpt-en-large](https://huggingface.co/rntc/cpt-en-large) | English | 396M |
-| [cpt-fr-base](https://huggingface.co/rntc/cpt-fr-base) | French | 150M |
-| [cpt-fr-large](https://huggingface.co/rntc/cpt-fr-large) | French | 350M |
+| [ModernBERT-bio-base](https://huggingface.co/almanach/ModernBERT-bio-base) | English | 149M |
+| [ModernBERT-bio-large](https://huggingface.co/almanach/ModernBERT-bio-large) | English | 396M |
+| [ModernCamemBERT-bio-base](https://huggingface.co/almanach/ModernCamemBERT-bio-base) | French | 150M |
+| [ModernCamemBERT-bio-large](https://huggingface.co/almanach/ModernCamemBERT-bio-large) | French | 350M |
 
 ## Limitations
 
@@ -245,9 +245,9 @@ Apache 2.0
 ## Citation
 
 ```bibtex
-@inproceedings{anonymous2026clm,
+@inproceedings{touchent2026clm,
   title={A Causal Language Modeling Detour Improves Encoder Continued Pretraining},
-  author={Anonymous},
+  author={Touchent, Rian and de la Clergerie, {\'E}ric},
   booktitle={Proceedings of COLM},
   year={2026}
 }
@@ -255,4 +255,4 @@ Apache 2.0
 
 ## Acknowledgments
 
-This work was performed using HPC resources.
+This work was performed using HPC resources from GENCI-IDRIS (Grant 2024-AD011015883). We thank the ALMAnaCH team at Inria for the ModernCamemBERT base checkpoint.