06-retraining-cost.md

SignalP 6.0 — What would it cost to remake the weights ourselves?

Short answer: surprisingly little in GPU dollars — roughly \$10–100 to reproduce the released models without hyperparameter search, maybe \$150–2,000 if you also redo the hyperparameter optimization. The expensive-sounding part (the protein language model) is not something you retrain — you download it. The real cost is engineering time, not compute.

The key reframe: fine-tune, don't pretrain

There are two very different things people mean by "make the weights":

| | What it is | Cost | |---|---|---| | Pretrain ProtBERT | Train the 420M-param BERT language model on ~2 billion protein sequences (UniRef100/BFD) | You never do this. It was done once by the ProtTrans group on TPU pods — hundreds of thousands of accelerator-hours, ~\$100k–\$1M-scale. You just from_pretrained("Rostlab/prot_bert") (~1.6 GB download). | | Fine-tune SignalP's head | Adapt that pretrained encoder + train the CRF decoder on ~25k labeled signal-peptide sequences | This is the whole job. Tens of GPU-hours. |

The scripts/train_model.py default is literally --model_name_or_path Rostlab/prot_bert, so the repo confirms it: SignalP 6.0 is a fine-tune.

We already have the training data

Everything needed is in the cloned repo (data/):

before_partitioning.fasta — 25,693 labeled sequences (the full set)
train_set.fasta — 20,290, split into 3 homology-reduced partitions (30% identity)
remainder.fasta — 5,403 held-out
Sequences are truncated to 70 residues (73 tokens with CLS/SEP) → cheap to train on.

Class balance (why the rare types are hard):

| Label | Count | |-------|-------| | NO_SP | 19,036 | | SP (Sec/SPI) | 3,652 | | LIPO (Sec/SPII) | 2,261 | | TAT (Tat/SPI) | 595 | | PILIN (Sec/SPIII) | 113 | | TATLIPO (Tat/SPII) | 36 |

So you do not need to rebuild the dataset (which was the hard scientific work — curating experimentally verified signal peptides and homology-partitioning them).

What the training pipeline actually is

From the scripts, reproducing the shipped weights is a 4-step pipeline:

Fine-tune 6 models (train_model.py). SignalP uses 3 partitions in a nested cross-validation → for each held-out test partition, the other two take turns as validation = 6 models. Each: ProtBERT encoder + linear-chain CRF, Adamax optimizer, slanted-triangular LR, early stopping, region regularization.
Ensemble (average_viterbi_decode.py). The 6 models are averaged → this is the slow / full model (needs ~14 GB RAM to run).
Distill (distill_model.py). Train one smaller student to mimic the 6-model ensemble's per-position marginal probabilities → this is the fast model (the default the CLI ships).
(Optional) Hyperparameter optimization — the scripts reference SigOpt. Doing this faithfully is what multiplies the cost.
Package the weights into the CLI's model_weights/ layout (and optionally signalp6_convert_models gpu).

Compute estimate

Assumptions: ProtBERT ≈ 420M params, 73-token sequences, ~13–17k train seqs/fold, batch 80, early-stopped. These are short sequences, so throughput is high and it fits on a 24 GB GPU (no 80 GB card needed).

| Piece | Estimate | Notes | |-------|----------|-------| | One fine-tuned model | ~2–5 GPU-hr | biggest uncertainty is epochs-to-converge | | Full 6-model ensemble | ~12–30 GPU-hr | trivially parallel across GPUs | | Distillation (+ generating ensemble probs) | ~3–6 GPU-hr | student is smaller | | Reproduce shipped weights, no HPO | ~20–40 GPU-hr | using published/guessed hyperparameters | | Hyperparameter search (SigOpt, ~20–60 trials) | +100–400 GPU-hr | each trial ≈ one fold; can be trimmed | | Faithful reproduction incl. HPO | ~150–450 GPU-hr | up to ~1,000 if thorough |

Dollar cost (mid-2026 cloud rates, approximate)

| GPU / market | \$/hr | No-HPO (~30 hr) | With HPO (~300 hr) | |--------------|------|-----------------|--------------------| | RTX 4090 / community (Vast.ai, RunPod community) | ~\$0.30–0.70 | ~\$10–25 | ~\$100–200 | | A100 80 GB on-demand (RunPod/Lambda) | ~\$1.2–2.5 | ~\$40–100 | ~\$400–750 | | H100 (overkill for 420M) | ~\$2–3.5 | ~\$70–140 | ~\$600–1,000+ |

Bottom line on money: a single 24 GB consumer GPU rented on a spot/community market reproduces the ensemble + fast model for roughly the price of a couple of coffees to a nice dinner (\$10–30). Even the full HPO reproduction is a few hundred dollars, not the six-figure number you'd fear if you had to pretrain the language model.

The real cost is engineering, not GPUs

Budget days-to-weeks of a person's time for:

Old dependency stack — Python ≤3.10, torch<2, a specific older transformers version (the BertModel API and CRF integration were written against it). This is the biggest time-sink to get running.
De-clustering the scripts — they contain hard-coded paths like /work3/felteu/tagging_checkpoints/... and assume a SLURM/SigOpt setup.
Orchestrating the 6-fold CV, ensembling, distillation, and repackaging into the CLI's exact weight format (including CPU/GPU variants).
SigOpt account (or swap in Optuna/Ray Tune) if you want real HPO.

Reproducibility caveat

You will get a model that is close to but not bit-identical to the official weights: the exact LR schedule, dropout, region-regularization weights, and SigOpt-found hyperparameters are not fully published, and torch/transformers version differences shift results. Expect to land within a point or two of the paper's MCCs if you're careful — good enough to validate the method, not a byte-for-byte clone.

Recommended path if you actually want to do it

conda create -n signalp6-train python=3.10; install torch==1.13.* (CUDA), a compatible transformers (~4.x from ~2021–2022), tokenizers, scikit-learn.
Download Rostlab/prot_bert once (cached by HuggingFace).
Get one partition's train working end-to-end on the data already in data/ (fix the hard-coded paths, confirm loss goes down) — this de-risks everything.
Script the 6-model CV, then average_viterbi_decode.py, then distill_model.py.
Skip SigOpt for a first pass — use sensible fixed hyperparameters; only add HPO if you need to match the paper exactly.
Rent a 24 GB GPU (Vast.ai/RunPod) for the runs; expect ~\$15–40 and a day or two of wall-clock for the no-HPO reproduction.

Cheapest sanity check before spending anything: just fine-tune a single fold locally and see the validation MCC climb. If that works, the rest is bookkeeping.