model card: fix YaRN context 384 kbp → 393 kbp (65,536 tokens × 6 bp)

README.md

@@ -26,7 +26,7 @@ A generative DNA foundation model from the **Carbon** family.
 
 ## Model Summary
 
-**Carbon-3B** is a 3B-parameter decoder-only autoregressive genomic foundation model trained on DNA and RNA sequences, with a primary focus on eukaryotes. It has a native context length of **32,768 6-mer tokens (≈ 197k DNA base pairs)** and extends to **64,000 tokens (≈ 384 kbp)** at inference time via YaRN. Carbon-3B is designed to be both strong and efficient: on generative tasks (sequence recovery), variant-effect prediction, and motif-perturbation discrimination, it matches the capability of substantially larger single-nucleotide baselines such as Evo2-7B while running several times faster.
+**Carbon-3B** is a 3B-parameter decoder-only autoregressive genomic foundation model trained on DNA and RNA sequences, with a primary focus on eukaryotes. It has a native context length of **32,768 6-mer tokens (≈ 197k DNA base pairs)** and extends to **65,536 tokens (≈ 393 kbp)** at inference time via YaRN. Carbon-3B is designed to be both strong and efficient: on generative tasks (sequence recovery), variant-effect prediction, and motif-perturbation discrimination, it matches the capability of substantially larger single-nucleotide baselines such as Evo2-7B while running several times faster.
 
 Carbon-3B is the **flagship** model of the Carbon family. We also release [**Carbon-8B**](https://huggingface.co/HuggingFaceBio/) for users who need additional capability at higher inference cost, and [**Carbon-500M**](https://huggingface.co/HuggingFaceBio/) — a small generative model intended for speculative decoding alongside Carbon-3B (or Carbon-8B).
 
@@ -34,12 +34,12 @@ Carbon-3B is the **flagship** model of the Carbon family. We also release [**Car
 
 - **3B parameters**, 30 layers, hidden size 3072, GQA (32 heads, 4 KV groups), SwiGLU, RMSNorm.
 - **Hybrid tokenizer**: non-overlapping 6-mer tokenization for DNA combined with the [Qwen3](https://huggingface.co/Qwen/Qwen3-4B) BPE vocabulary for English text. We found 6-mer tokenization to work substantially better than BPE for DNA — hence the hybrid setup, which keeps 6-mer for DNA while preserving a BPE vocabulary for future joint English + DNA training, the direction in which we believe genomic foundation models should converge. Each DNA token encodes 6 nucleotides, so 1 DNA token ≈ 6 bp.
-- **Native context: 32,768 tokens ≈ 197 kbp.** Extendable to 64 k tokens (≈ 384 kbp) at inference time using YaRN.
+- **Native context: 32,768 tokens ≈ 197 kbp.** Extendable to 65,536 tokens (≈ 393 kbp) at inference time using YaRN.
 - **Trained with a Cross-Entropy → Factorised Nucleotide Supervision (FNS) objective schedule** to bridge coarse tokenization and single-nucleotide resolution (see the Carbon technical report).
 - **Metadata-conditioned**: optional species-type and gene-type metadata tokens enable conditional generation.
 - **Efficient inference**: TODO
 
-Across our zero-shot evaluation suite — sequence recovery, four variant-effect-prediction (VEP) benchmarks (ClinVar coding, ClinVar non-coding, BRCA2, TraitGym Mendelian), and two sequence-level perturbation tasks (TATA-box and synonymous codon) — Carbon-3B is competitive with Evo2-7B. It additionally works well on long context and retrieves needles reliably from up to ≈ 384 kbp of distal context on the Genome-NIAH long-context benchmark, while remaining several times faster than Evo2-7B.
+Across our zero-shot evaluation suite — sequence recovery, four variant-effect-prediction (VEP) benchmarks (ClinVar coding, ClinVar non-coding, BRCA2, TraitGym Mendelian), and two sequence-level perturbation tasks (TATA-box and synonymous codon) — Carbon-3B is competitive with Evo2-7B. It additionally works well on long context and retrieves needles reliably from up to ≈ 393 kbp of distal context on the Genome-NIAH long-context benchmark, while remaining several times faster than Evo2-7B.
 
 For full design rationale and ablations, see the Carbon technical report and the [Carbon GitHub repository](https://github.com/huggingface/carbon).
 
@@ -134,15 +134,15 @@ def score(seq: str) -> float:
 
 For batched scoring with attention masking and full reproducible evaluation pipelines (sequence recovery, ClinVar / BRCA2 / TraitGym VEP, TATA / synonymous-codon perturbation, Genome-NIAH), use the official scripts in the [Carbon evaluation directory](https://github.com/huggingface/carbon/tree/main/evaluation) — see [`perturbation_tasks.py`](https://github.com/huggingface/carbon/blob/main/evaluation/perturbation_tasks.py) for the canonical `score_hf` implementation and [`README.md`](https://github.com/huggingface/carbon/blob/main/evaluation/README.md) for run instructions across all tasks.
 
-### Long context: extending to 64 k tokens (≈ 384 kbp) with YaRN
+### Long context: extending to 65,536 tokens (≈ 393 kbp) with YaRN
 
-The released `config.json` is configured for the native 32 k context. To extend to **64 k tokens (≈ 384 kbp)** at inference time, override `max_position_embeddings` and add a YaRN `rope_scaling` block. We recommend a **YaRN factor of 4**, which we observed gives better retrieval quality at 64 k than a tighter factor of 2:
+The released `config.json` is configured for the native 32 k context. To extend to **65,536 tokens (≈ 393 kbp)** at inference time, override `max_position_embeddings` and add a YaRN `rope_scaling` block. We recommend a **YaRN factor of 4**, which we observed gives better retrieval quality at 64 k than a tighter factor of 2:
 
 ```python
 from transformers import AutoConfig, AutoModelForCausalLM
 
 config = AutoConfig.from_pretrained(repo, trust_remote_code=True)
-config.max_position_embeddings = 65536    # 64 k tokens ≈ 384 kbp
+config.max_position_embeddings = 65536    # 65,536 tokens ≈ 393 kbp
 config.rope_scaling = {
     "type": "yarn",
     "factor": 4.0,
@@ -234,8 +234,8 @@ Below are the scores on `niah`:
 Sample sizes: Carbon & GENERator n=500. Evo2-7B n=150 at 16k, n=100 at 32k, n=20 at 64k due to the slow inference speed.
 > TODO: run more 64k samples for Evo2 7B
 
-- **4× longer effective context than Generator-v2-3B.** Generator-v2-3B caps at 16 k tokens (≈ 98 kbp). Carbon-3B has a native context of 32 k tokens (≈ 197 kbp) and extends to 64 k tokens (≈ 384 kbp) at inference time with YaRN. It matches Generator-v2-3B on `niah` at 98 kbp.
-- **Matches Evo2-7B (1 M context) on `niah` at 384 kbp** (64 k tokens) under YaRN, despite being substantially smaller.
+- **4× longer effective context than Generator-v2-3B.** Generator-v2-3B caps at 16 k tokens (≈ 98 kbp). Carbon-3B has a native context of 32 k tokens (≈ 197 kbp) and extends to 65,536 tokens (≈ 393 kbp) at inference time with YaRN. It matches Generator-v2-3B on `niah` at 98 kbp.
+- **Matches Evo2-7B (1 M context) on `niah` at 393 kbp** (64 k tokens) under YaRN, despite being substantially smaller.
 - **YaRN helps at the native-context boundary.** At 32 k tokens (≈ 197 kbp), retrieval quality near the model's native maximum begins to degrade; applying YaRN at inference time smooths the boundary and recovers most of the lost accuracy.
 
 
@@ -290,7 +290,7 @@ The optimizer is AdamW (β₁ = 0.9, β₂ = 0.95, ε = 1e-8, weight decay = 0.1
 ## Limitations
 
 - **Primarily eukaryotic training.** Carbon-3B is trained mostly on eukaryotic genomic and transcript sequence; the pre-training mixture deliberately includes a smaller prokaryotic component (GTDB bacterial genomes) so that continual pre-training on prokaryotic data remains straightforward. Despite this modest prokaryotic share, we observed consistent gains on prokaryotic sequence recovery throughout training, and Carbon-3B already **matches [GENERator-v2-prokaryote-3B](https://huggingface.co/GenerTeam/GENERator-v2-prokaryote-3b-base)** — a model trained specifically on prokaryotes — on the prokaryote sequence-recovery split. We expect that a short continued-training phase on prokaryotic data would deliver substantially stronger prokaryote-specific performance.
-- **YaRN beyond 64 k tokens degrades.** YaRN reliably extends Carbon-3B to 64 k tokens (≈ 384 kbp) at inference time with `factor=4`. Pushing further to 128 k tokens (≈ 786 kbp) causes retrieval quality to drop sharply in our long-context benchmarks.
+- **YaRN beyond 64 k tokens degrades.** YaRN reliably extends Carbon-3B to 65,536 tokens (≈ 393 kbp) at inference time with `factor=4`. Pushing further to 128 k tokens (≈ 786 kbp) causes retrieval quality to drop sharply in our long-context benchmarks.
 
 ## License