File size: 4,362 Bytes
fbabbaf c3a251e fbabbaf c3a251e fbabbaf c3a251e 2c7dd09 c3a251e 2c7dd09 c3a251e 805e7ba c3a251e 2c7dd09 c3a251e 2c7dd09 c3a251e 2c7dd09 c3a251e 2c7dd09 c3a251e 2c7dd09 c3a251e 2c7dd09 c3a251e | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 | ---
library_name: transformers
license: apache-2.0
language:
- dna
tags:
- dna
- genomic
- transformers
- speculative-decoding
---
# Carbon-500M
A small generative DNA model from the **Carbon** family.
**Carbon-500M is intended primarily as a draft model for speculative decoding** — it shares the tokenizer and DNA template format of [Carbon-3B](https://huggingface.co/HuggingFaceBio/Carbon-3B) and [Carbon-8B](https://huggingface.co/HuggingFaceBio/Carbon-8B), so it can be paired with either as the target model to reduce wall-clock generation cost at no quality loss. It is not designed to be competitive with the 3B/8B Carbon models on downstream benchmarks.
For the full design rationale, tokenizer specification, evaluation protocol, and usage notes (DNA tag wrapping, 6-mer constraints, scoring helpers), please refer to the **[Carbon-3B model card](https://huggingface.co/HuggingFaceBio/Carbon-3B)** — this card focuses only on facts specific to Carbon-500M.
> TODO: update teh tokenizer code
## Facts
- **500M-parameter decoder-only autoregressive DNA model** (Llama-style architecture).
- **Hybrid tokenizer** shared with the rest of the Carbon family (6-mer for DNA + Qwen3 BPE for English text; each DNA token ≈ 6 bp).
- **Pre-training tokens:** 600B 6-mer tokens (≈ 3.6 T DNA base pairs).
- **Sequence length:** 8 192tokens (≈ 48 kbp).
- **Loss schedule:** cross-entropy 0 → 300 B tokens, then switch to the hybrid Factorised Nucleotide Supervision (FNS) loss from 300 B → 600 B tokens. The switch happens later than for Carbon-3B because Carbon-500M's training was very stable and tolerated the later transition.
- **Data mixture:** identical to the **decay-phase mixture used by Carbon-3B** — 50 % Generator-style eukaryotic genes / 25 % mature mRNA / 10 % splice-enriched mRNA / 15 % GTDB bacterial genomes. Same weights across the whole 600 B run.
- **Precision:** bfloat16. **Optimizer:** AdamW. **Positional embedding:** RoPE.
- **No long-context training stage** — the model stays at its 8 k-token native context so 48kbp.
- Released as a standard Hugging Face causal LM (`LlamaForCausalLM`).
## How to use
Wrap DNA in `<dna>...</dna>` exactly as for the larger models. See the [Carbon-3B card](https://huggingface.co/HuggingFaceBio/Carbon-3B#tokenizer-working-with-dna-inputs) for tokenizer details.
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
repo = "HuggingFaceBio/Carbon-500M"
tok = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
repo, torch_dtype=torch.bfloat16,
).cuda().eval()
prompt = "<dna>ATGCGCTAGCTACGATCGATCGTAGCTAGCTAGCTAGCTACG"
inputs = tok(prompt, return_tensors="pt", add_special_tokens=False).to("cuda")
out = model.generate(**inputs, max_new_tokens=64, do_sample=False)
print(tok.decode(out[0][inputs.input_ids.shape[1]:]))
```
### Recommended use: speculative decoding with Carbon-3B / Carbon-8B
Carbon-500M is most useful when paired with a larger Carbon model as the verifier. Hugging Face Transformers supports this natively through the `assistant_model` argument:
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
tok = AutoTokenizer.from_pretrained("HuggingFaceBio/Carbon-3B", trust_remote_code=True)
draft = AutoModelForCausalLM.from_pretrained(
"HuggingFaceBio/Carbon-500M", torch_dtype=torch.bfloat16
).cuda().eval()
target = AutoModelForCausalLM.from_pretrained(
"HuggingFaceBio/Carbon-3B", torch_dtype=torch.bfloat16
).cuda().eval()
prompt = "<dna>ATGCGCTAGCTACGATCGATCGTAGCTAGCTAGCTAGCTACG"
inputs = tok(prompt, return_tensors="pt", add_special_tokens=False).to("cuda")
out = target.generate(
**inputs, max_new_tokens=256, do_sample=False,
assistant_model=draft,
)
print(tok.decode(out[0][inputs.input_ids.shape[1]:]))
```
Output is guaranteed identical to greedy decoding with the target model alone; only wall-clock latency is reduced.
## Evaluation
Carbon-500M is benchmarked against ≈ 1B-parameter DNA models on the standard Carbon evaluation suite. See the [Carbon-3B card](https://huggingface.co/HuggingFaceBio/Carbon-3B#evaluation) for the task definitions and methodology.
> TODO Loubna: add one downstream table comparing Carbon-500M to other 1B-class baselines. -->
## License
Apache 2.0. |