Update model card: add model-index, datasets, training details, related models

README.md

@@ -17,10 +17,86 @@ pipeline_tag: fill-mask
 widget:
 - text: "Les patients atteints de <mask> présentent un risque accru de complications cardiovasculaires."
 - text: "Le traitement par <mask> a montré une amélioration significative des symptômes."
+model-index:
+- name: cpt-fr-base
+  results:
+  - task:
+      type: text-classification
+      name: Text Classification
+    dataset:
+      name: FrACCO-30
+      type: rntc/fracco
+    metrics:
+    - type: f1
+      value: 74.8
+  - task:
+      type: text-classification
+      name: Text Classification
+    dataset:
+      name: FrACCO-100
+      type: rntc/fracco
+    metrics:
+    - type: f1
+      value: 60.1
+  - task:
+      type: text-classification
+      name: Text Classification
+    dataset:
+      name: CANTEMIST
+      type: cantemist
+    metrics:
+    - type: f1
+      value: 71.0
+  - task:
+      type: text-classification
+      name: Text Classification
+    dataset:
+      name: DISTEMIST
+      type: distemist
+    metrics:
+    - type: f1
+      value: 25.5
+  - task:
+      type: text-classification
+      name: Text Classification
+    dataset:
+      name: MedDialog
+      type: meddialog
+    metrics:
+    - type: f1
+      value: 63.6
+  - task:
+      type: text-classification
+      name: Text Classification
+    dataset:
+      name: DiaMed
+      type: diamed
+    metrics:
+    - type: f1
+      value: 67.4
+  - task:
+      type: token-classification
+      name: NER
+    dataset:
+      name: EMEA
+      type: emea
+    metrics:
+    - type: f1
+      value: 65.9
+  - task:
+      type: token-classification
+      name: NER
+    dataset:
+      name: Medline
+      type: medline
+    metrics:
+    - type: f1
+      value: 58.2
 ---
 
 # cpt-fr-base
 
+*cpt-fr is available in two sizes: [base](https://huggingface.co/rntc/cpt-fr-base) (150M parameters) and [large](https://huggingface.co/rntc/cpt-fr-large) (350M parameters). Our code will be released upon publication.*
 
 ## Table of Contents
 
@@ -33,7 +109,7 @@ widget:
 
 ## Model Summary
 
-cpt-fr-base is a French biomedical encoder built by continued pretraining of [ModernCamemBERT](https://huggingface.co/almanach/moderncamembert-base) using a **CLM detour** recipe. Instead of standard MLM continued pretraining, we temporarily switch to causal language modeling (CLM) before returning to MLM. This produces lasting representational changes in early transformer layers that improve downstream biomedical performance by +2.9pp on average across 8 French biomedical tasks.
+cpt-fr is a French biomedical encoder built by continued pretraining of [ModernCamemBERT](https://huggingface.co/almanach/moderncamembert-base) using a **CLM detour** recipe. Instead of standard MLM continued pretraining, we temporarily switch to causal language modeling (CLM) before returning to MLM. This produces lasting representational changes in early transformer layers that improve downstream biomedical performance by +2.9pp on average across 8 French biomedical tasks.
 
 The model uses the ModernBERT architecture with FlashAttention, rotary positional embeddings (RoPE), alternating local/global attention, and unpadding, supporting **8,192-token context** — critical for long clinical documents that exceed the 512-token limit of previous French biomedical models.
 
@@ -96,7 +172,7 @@ outputs = model(**inputs)
 # outputs.last_hidden_state: [batch, seq_len, 768]
 ```
 
-**Note:** cpt-fr-base does not use token type IDs. You can omit the `token_type_ids` parameter.
+**Note:** cpt-fr does not use token type IDs. You can omit the `token_type_ids` parameter.
 
 ## Training
 
@@ -112,12 +188,12 @@ outputs = model(**inputs)
 
 ### Methodology
 
-cpt-fr-base is trained in two phases, initialized from [ModernCamemBERT](https://huggingface.co/almanach/moderncamembert-base):
+cpt-fr is trained in two phases, initialized from [ModernCamemBERT](https://huggingface.co/almanach/moderncamembert-base):
 
 * **Phase 1 — CLM detour (10B tokens):** The bidirectional attention mask is replaced with a causal mask, and the model is trained with next-token prediction. This dense training signal (100% of positions) deeply modifies early transformer layers for domain adaptation.
 * **Phase 2 — MLM decay (1B tokens):** Bidirectional attention is restored, and the model is trained with masked language modeling at 15% masking. The learning rate decays from peak to 10% following a 1-sqrt schedule.
 
-Both phases use the same data mix. Training used AdamW (lr=2e-4, beta1=0.9, beta2=0.98), bf16 mixed precision, global batch size of 384 sequences (~3.1M tokens), on 4x H100 GPUs with [Composer](https://github.com/mosaicml/composer).
+Both phases use the same data mix (11B tokens total). Training used AdamW (lr=2e-4, beta1=0.9, beta2=0.98), bf16 mixed precision, global batch size of 384 sequences (~3.1M tokens), on 4× H100 80GB GPUs with [Composer](https://github.com/mosaicml/composer). Total training time: ~5 GPU-hours.
 
 ### Why a CLM Detour?
 
@@ -147,6 +223,15 @@ This model is designed for French biomedical and clinical NLP tasks:
 
 The 8,192-token context is critical for long clinical documents (discharge summaries, oncology reports) that are truncated by 512-token models.
 
+## Related Models
+
+| Model | Language | Parameters |
+|-------|----------|------------|
+| [cpt-en-base](https://huggingface.co/rntc/cpt-en-base) | English | 149M |
+| [cpt-en-large](https://huggingface.co/rntc/cpt-en-large) | English | 396M |
+| [cpt-fr-base](https://huggingface.co/rntc/cpt-fr-base) | French | 150M |
+| [cpt-fr-large](https://huggingface.co/rntc/cpt-fr-large) | French | 350M |
+
 ## Limitations
 
 - Trained on French biomedical text; not suitable for other languages without further adaptation.
@@ -161,13 +246,13 @@ Apache 2.0
 
 ```bibtex
 @inproceedings{anonymous2026clm,
-  title={Under review},
+  title={A Causal Language Modeling Detour Improves Encoder Continued Pretraining},
   author={Anonymous},
-  booktitle={Under review},
+  booktitle={Proceedings of COLM},
   year={2026}
 }
 ```
 
 ## Acknowledgments
 
-
+This work was performed using HPC resources.