pocketz.doi.bio/2026-05-21_200742_github_sness23_carbon
sness23/carbon: The home of Carbon Genomic Foundation Model 🧬
URL: https://github.com/sness23/carbon
Saved: 2026-05-21T03:07:47.995Z
carbon Public forked from huggingface/carbon Pin Watch 0 (0) Fork 0 Star 0 sness23/carbon main t T Add file Code This branch is up to date with huggingface/carbon:main. Contribute Sync fork Folders and files Name Last commit message Last commit date
Latest commit edbeeching Merge pull request huggingface#9 from huggingface/deepstarr-add-kerne… d38abaa · History 127 Commits
evaluation
Merge pull request huggingface#12 from huggingface/MaurizioFiusco-pat…
finetuning
Address finetuning recipe review feedback
logs
Add logs
.gitignore
Track logs directory placeholder
LICENSE
Initial commit
README.md
Revise README for clarity and updates
pyproject.toml
Merge pull request huggingface#9 from huggingface/deepstarr-add-kerne…
tech-report.pdf
Add files via upload
uv.lock
Merge pull request huggingface#9 from huggingface/deepstarr-add-kerne…
Repository files navigation README License Carbon
Genomic foundation models from Hugging Face. Carbon is a family of causal language models trained on 1T tokens of DNA / 6T DNA base pairs from the Carbon Pretraining Corpus, a curated mix of DNA & RNA sequences.
This repo contains:
the eval code for Carbon tasks: sequence recovery, variant effect prediction, and perturbations. We put this together because the zero-shot DNA eval landscape is currently scattered — useful tasks live in different repos, often buried alongside evals that need finetuning or that are already saturated, which makes reproducibility harder. scripts for fine-tuning the Carbon models on downstream tasks. Contents Models Installation Inference Pretraining Evaluation Finetuning Models Model Params Notes HuggingFaceBio/Carbon-500M 500M Draft model for speculative decoding. HuggingFaceBio/Carbon-3B 3B Flagship. Matches or beats Evo2 7B. HuggingFaceBio/Carbon-8B 8B Larger model for more performance.
The Carbon checkpoints use a hybrid tokenizer: BPE for English text and 6-mer for DNA, switched by a
Installation
Install the core runtime dependencies with:
uv sync
To include evaluation dependencies, run:
uv sync --group evaluation
For Evo2-backed evaluation, install the evaluation and Evo2 dependency groups:
uv sync --group evaluation --group evo2 Inference from transformers import AutoModelForCausalLM, AutoTokenizer
modelid = "HuggingFaceBio/Carbon-3B" tok = AutoTokenizer.frompretrained(modelid, trustremotecode=True) model = AutoModelForCausalLM.frompretrained(modelid, trustremotecode=True, torchdtype="bfloat16").to("cuda")
DNA generation: wrap the prompt with so the tokenizer routes to 6-mer mode.
context = "ATGGCCTCGAGCAGCAGCAGCAGCAGCAGCAGCAGCAGCAGCAGCAG"
prompt = f"
out = model.generate(**inputs, maxnewtokens=10, do_sample=False) print(tok.decode(out[0]))
For zero-shot variant scoring, just feed the model the full sequence and read the log-likelihood — see evaluation/vep_eval.py.
Pretraining Training data
Carbon was trained on 1 T tokens (≈ 6 T DNA base pairs) drawn from the Carbon Pretraining Corpus mix of:
Eukaryote genes (animals, plants, fungi, protists) — functional genomic regions, extracted from refSeq from Generator training mix. mRNA transcripts — processed, spliced mRNA from OpenGenome2. Prokaryote genomes — long chromosomal chunks from bacteria and archaea (GTDB v220 + IMG/PR), included as a smaller fraction (~10 % of the training mixture).
The mixture is eukaryote-heavy by design. Carbon's target use case is eukaryote. The prokaryote share is 10% of the pretraining mixture, so the model can be continually pretrained on prokaryote species.
Pretraining code
Carbon was trained with our Megatron-LM fork: huggingface/Megatron-LM-Carbon. The fork adds:
Hybrid loss: the loss for bridging coarse 6-mer tokenization and single-nucleotide resolution. Carbon training scripts Evaluation
This repo ships a suite of seven zero-shot DNA evaluations with reproducible code. The benchmark datasets are available in this collection.
The suite covers four modes of zero-shot evaluation:
Variant effect prediction, with three established benchmarks spanning both coding (BRCA2) and non-coding regulatory variants (TraitGym Mendelian), plus ClinVar for broad pathogenic-vs-benign coverage. A generative task — sequence recovery, ported from the GENERator paper. Two perturbation tasks we built — CAG repeat insertion and synonymous-codon substitution — to probe regulatory-motif awareness and codon-usage structure. Long-context retrieval we built — Genome-NIAH, a needle-in-a-haystack eval adapted to DNA (four tasks × six context lengths up to 786 kbp).
All eval scripts live in evaluation/. Each one runs on Carbon, GENERator, or Evo2 via a single backend flag, so numbers are directly comparable across model families.
Benchmark What it measures Script Sequence recovery Given a DNA context, generate the next 30 bp; score per-base accuracy against the held-out continuation. Training-free generative eval from the GENERator paper. sequencerecovery.py CAG repeat insertion A 30 bp codon-aligned region 60 bp into the CDS exon is replaced with 10 consecutive CAG triplets, mimicking polyglutamine expansion disorders (HD, SCAs, DRPLA). The patch is length- and reading-frame-preserving; all sequence outside is identical. The model should assign higher likelihood to the intact native sequence. perturbationtasks.py --task motifhuman Synonymous codon substitution CDS codons are replaced with the highest-frequency synonym for the target species (human or mouse); amino acid identity is preserved by construction. The model should prefer native codon usage over the codon-optimised variant. Probes coding-sequence structure and species-specific codon bias. perturbationtasks.py --task synhuman / --task synmouse BRCA2 VEP Zero-shot VEP on saturation-mutagenesis BRCA2 (Huang 2025). Centered 8 kb window + full-LL delta. vepeval.py TraitGym Mendelian 3,380 fine-mapped non-coding regulatory variants for 113 Mendelian diseases (Benegas et al. 2025). Centered 8 kb window + full-LL delta. vepeval.py ClinVar Pathogenic vs benign on curated coding + noncoding ClinVar variants. Right-end / next-token scoring with 24 kb left context. clinvarvepeval.py (uses HuggingFaceBio/clinvar-vep-final directly) Genome-NIAH Long-context retrieval: insert a (key, value) pair in a real-genome haystack, ask the model to retrieve the value. Four tasks × six context lengths (up to 786 kbp). genomeniaheval.py
See evaluation/README.md for run commands, DNA-tag flags, and per-benchmark details.
Finetuning
A minimal end-to-end finetuning example (promoter detection from the Nucleotide Transformer downstream benchmark) lives in finetuning/. It uses the standard 🤗 Transformers Trainer with AutoModelForSequenceClassification on top of the Carbon backbone — swap in any other classification dataset by changing one flag.
To specialise Carbon on a new clade (e.g. a specific bacterium or protist that wasn't well represented in the pretraining mix), the same scaffolding works for continual pretraining: load the model with AutoModelForCausalLM, feed it sequences with the
Acknowledgements
Carbon is a joint collaboration between the research teams at Hugging Face, Zhongguancun Academy, and TIGEM/University of Naples “Federico II”.
License
Apache 2.0.
About
The home of Carbon Genomic Foundation Model 🧬
Resources Readme License Apache-2.0 license Activity Stars 0 stars Watchers 0 watching Forks 0 forks
Releases No releases published Create a new release
Packages No packages published Publish your first package
Contributors No contributors
Languages Python 80.6%
Shell 19.4%
Suggested workflows Based on your tech stack SLSA Generic generator Configure SLSA Generic generator Generate SLSA3 provenance for your existing release workflows Django Configure Django Build and Test a Django Project Publish Python Package Configure Publish Python Package Publish a Python Package to PyPI on release. More workflows Dismiss suggestions