Edit · 06 Retraining Cost

Saving needs a token. Log in or register to get one. Edits are public and licensed CC-BY-SA-4.0.

# SignalP 6.0 — What would it cost to remake the weights ourselves?

Short answer: **surprisingly little in GPU dollars — roughly \$10–100 to reproduce
the released models without hyperparameter search, maybe \$150–2,000 if you also
redo the hyperparameter optimization.** The expensive-sounding part (the protein
language model) is *not* something you retrain — you download it. The real cost is
engineering time, not compute.

## The key reframe: fine-tune, don't pretrain

There are two very different things people mean by "make the weights":

| | What it is | Cost |
|---|---|---|
| **Pretrain ProtBERT** | Train the 420M-param BERT language model on ~2 billion protein sequences (UniRef100/BFD) | **You never do this.** It was done once by the ProtTrans group on TPU pods — hundreds of thousands of accelerator-hours, ~\$100k–\$1M-scale. You just `from_pretrained("Rostlab/prot_bert")` (~1.6 GB download). |
| **Fine-tune SignalP's head** | Adapt that pretrained encoder + train the CRF decoder on ~25k labeled signal-peptide sequences | **This is the whole job.** Tens of GPU-hours. |

The `scripts/train_model.py` default is literally `--model_name_or_path Rostlab/prot_bert`, so the repo confirms it: SignalP 6.0 is a **fine-tune**.

## We already have the training data

Everything needed is in the cloned repo (`data/`):

- `before_partitioning.fasta` — **25,693 labeled sequences** (the full set)
- `train_set.fasta` — 20,290, split into **3 homology-reduced partitions** (30% identity)
- `remainder.fasta` — 5,403 held-out
- Sequences are truncated to **70 residues** (73 tokens with CLS/SEP) → cheap to train on.

Class balance (why the rare types are hard):

| Label | Count |
|-------|-------|
| NO_SP | 19,036 |
| SP (Sec/SPI) | 3,652 |
| LIPO (Sec/SPII) | 2,261 |
| TAT (Tat/SPI) | 595 |
| PILIN (Sec/SPIII) | 113 |
| TATLIPO (Tat/SPII) | **36** |

So you do **not** need to rebuild the dataset (which *was* the hard scientific work —
curating experimentally verified signal peptides and homology-partitioning them).

## What the training pipeline actually is

From the scripts, reproducing the shipped weights is a 4-step pipeline:

1. **Fine-tune 6 models** (`train_model.py`). SignalP uses **3 partitions in a nested
   cross-validation** → for each held-out test partition, the other two take turns as
   validation = **6 models**. Each: ProtBERT encoder + linear-chain CRF, Adamax
   optimizer, slanted-triangular LR, early stopping, region regularization.
2. **Ensemble** (`average_viterbi_decode.py`). The 6 models are averaged → this is the
   **`slow` / full model** (needs ~14 GB RAM to run).
3. **Distill** (`distill_model.py`). Train one smaller student to mimic the 6-model
   ensemble's per-position marginal probabilities → this is the **`fast` model**
   (the default the CLI ships).
4. **(Optional) Hyperparameter optimization** — the scripts reference **SigOpt**. Doing
   this faithfully is what multiplies the cost.
5. **Package** the weights into the CLI's `model_weights/` layout (and optionally
   `signalp6_convert_models gpu`).

## Compute estimate

Assumptions: ProtBERT ≈ **420M params**, 73-token sequences, ~13–17k train seqs/fold,
batch 80, early-stopped. These are short sequences, so throughput is high and it fits
on a **24 GB** GPU (no 80 GB card needed).

| Piece | Estimate | Notes |
|-------|----------|-------|
| One fine-tuned model | **~2–5 GPU-hr** | biggest uncertainty is epochs-to-converge |
| Full 6-model ensemble | **~12–30 GPU-hr** | trivially parallel across GPUs |
| Distillation (+ generating ensemble probs) | **~3–6 GPU-hr** | student is smaller |
| **Reproduce shipped weights, no HPO** | **~20–40 GPU-hr** | using published/guessed hyperparameters |
| Hyperparameter search (SigOpt, ~20–60 trials) | **+100–400 GPU-hr** | each trial ≈ one fold; can be trimmed |
| **Faithful reproduction incl. HPO** | **~150–450 GPU-hr** | up to ~1,000 if thorough |

## Dollar cost (mid-2026 cloud rates, approximate)

| GPU / market | \$/hr | No-HPO (~30 hr) | With HPO (~300 hr) |
|--------------|------|-----------------|--------------------|
| RTX 4090 / community (Vast.ai, RunPod community) | ~\$0.30–0.70 | **~\$10–25** | ~\$100–200 |
| A100 80 GB on-demand (RunPod/Lambda) | ~\$1.2–2.5 | ~\$40–100 | ~\$400–750 |
| H100 (overkill for 420M) | ~\$2–3.5 | ~\$70–140 | ~\$600–1,000+ |

**Bottom line on money:** a single 24 GB consumer GPU rented on a spot/community
market reproduces the ensemble + fast model for roughly the **price of a couple of
coffees to a nice dinner** (\$10–30). Even the full HPO reproduction is a few hundred
dollars, not the six-figure number you'd fear if you had to pretrain the language model.

## The real cost is engineering, not GPUs

Budget days-to-weeks of a person's time for:

- **Old dependency stack** — Python ≤3.10, `torch<2`, a *specific older* `transformers`
  version (the `BertModel` API and CRF integration were written against it). This is the
  biggest time-sink to get running.
- **De-clustering the scripts** — they contain hard-coded paths like
  `/work3/felteu/tagging_checkpoints/...` and assume a SLURM/SigOpt setup.
- **Orchestrating** the 6-fold CV, ensembling, distillation, and repackaging into the
  CLI's exact weight format (including CPU/GPU variants).
- **SigOpt** account (or swap in Optuna/Ray Tune) if you want real HPO.

## Reproducibility caveat

You will get a model that is **close to but not bit-identical** to the official weights:
the exact LR schedule, dropout, region-regularization weights, and SigOpt-found
hyperparameters are not fully published, and torch/transformers version differences
shift results. Expect to land within a point or two of the paper's MCCs if you're
careful — good enough to validate the method, not a byte-for-byte clone.

## Recommended path if you actually want to do it

1. `conda create -n signalp6-train python=3.10`; install `torch==1.13.*` (CUDA),
   a compatible `transformers` (~4.x from ~2021–2022), `tokenizers`, `scikit-learn`.
2. Download `Rostlab/prot_bert` once (cached by HuggingFace).
3. Get **one** partition's train working end-to-end on the data already in `data/`
   (fix the hard-coded paths, confirm loss goes down) — this de-risks everything.
4. Script the 6-model CV, then `average_viterbi_decode.py`, then `distill_model.py`.
5. Skip SigOpt for a first pass — use sensible fixed hyperparameters; only add HPO if
   you need to match the paper exactly.
6. Rent a 24 GB GPU (Vast.ai/RunPod) for the runs; expect **~\$15–40 and a day or two**
   of wall-clock for the no-HPO reproduction.

**Cheapest sanity check before spending anything:** just fine-tune a single fold locally
and see the validation MCC climb. If that works, the rest is bookkeeping.

Cancel

Preview · live