library_name: transformers
license: apache-2.0
language:
- dna
tags:
- dna
- genomic
- transformers
- speculative-decoding
Carbon-500M
A small generative DNA model from the Carbon family.
Carbon-500M is intended primarily as a draft model for speculative decoding — it shares the tokenizer and DNA template format of Carbon-3B and Carbon-8B, so it can be paired with either as the target model to reduce wall-clock generation cost at no quality loss. It is not designed to be competitive with the 3B/8B Carbon models on downstream benchmarks.
For the full design rationale, tokenizer specification, evaluation protocol, and usage notes (DNA tag wrapping, 6-mer constraints, scoring helpers), please refer to the Carbon-3B model card — this card focuses only on facts specific to Carbon-500M.
TODO: update teh tokenizer code
Facts
- 500M-parameter decoder-only autoregressive DNA model (Llama-style architecture).
- Hybrid tokenizer shared with the rest of the Carbon family (6-mer for DNA + Qwen3 BPE for English text; each DNA token ≈ 6 bp).
- Pre-training tokens: 600B 6-mer tokens (≈ 3.6 T DNA base pairs).
- Sequence length: 8 192tokens (≈ 48 kbp).
- Loss schedule: cross-entropy 0 → 300 B tokens, then switch to the hybrid Factorised Nucleotide Supervision (FNS) loss from 300 B → 600 B tokens. The switch happens later than for Carbon-3B because Carbon-500M's training was very stable and tolerated the later transition.
- Data mixture: identical to the decay-phase mixture used by Carbon-3B — 50 % Generator-style eukaryotic genes / 25 % mature mRNA / 10 % splice-enriched mRNA / 15 % GTDB bacterial genomes. Same weights across the whole 600 B run.
- Precision: bfloat16. Optimizer: AdamW. Positional embedding: RoPE.
- No long-context training stage — the model stays at its 8 k-token native context so 48kbp.
- Released as a standard Hugging Face causal LM (
LlamaForCausalLM).
How to use
Wrap DNA in <dna>...</dna> exactly as for the larger models. See the Carbon-3B card for tokenizer details.
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
repo = "HuggingFaceBio/Carbon-500M"
tok = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
repo, torch_dtype=torch.bfloat16,
).cuda().eval()
prompt = "<dna>ATGCGCTAGCTACGATCGATCGTAGCTAGCTAGCTAGCTACG"
inputs = tok(prompt, return_tensors="pt", add_special_tokens=False).to("cuda")
out = model.generate(**inputs, max_new_tokens=64, do_sample=False)
print(tok.decode(out[0][inputs.input_ids.shape[1]:]))
Recommended use: speculative decoding with Carbon-3B / Carbon-8B
Carbon-500M is most useful when paired with a larger Carbon model as the verifier. Hugging Face Transformers supports this natively through the assistant_model argument:
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
tok = AutoTokenizer.from_pretrained("HuggingFaceBio/Carbon-3B", trust_remote_code=True)
draft = AutoModelForCausalLM.from_pretrained(
"HuggingFaceBio/Carbon-500M", torch_dtype=torch.bfloat16
).cuda().eval()
target = AutoModelForCausalLM.from_pretrained(
"HuggingFaceBio/Carbon-3B", torch_dtype=torch.bfloat16
).cuda().eval()
prompt = "<dna>ATGCGCTAGCTACGATCGATCGTAGCTAGCTAGCTAGCTACG"
inputs = tok(prompt, return_tensors="pt", add_special_tokens=False).to("cuda")
out = target.generate(
**inputs, max_new_tokens=256, do_sample=False,
assistant_model=draft,
)
print(tok.decode(out[0][inputs.input_ids.shape[1]:]))
Output is guaranteed identical to greedy decoding with the target model alone; only wall-clock latency is reduced.
Evaluation
Carbon-500M is benchmarked against ≈ 1B-parameter DNA models on the standard Carbon evaluation suite. See the Carbon-3B card for the task definitions and methodology.
TODO Loubna: add one downstream table comparing Carbon-500M to other 1B-class baselines. -->
License
Apache 2.0.