04-benchmarks-and-context.md

SignalP 6.0 — Benchmarks, Limitations & Context

Performance vs SignalP 5.0

Evaluation uses Matthews Correlation Coefficient (MCC), computed per organism group (Archaea, Eukarya, Gram-positive, Gram-negative) in two flavors:

MCC1 — SP type vs. non-SP sequences only (easier).
MCC2 — SP type vs. all other SP types (harder, realistic: the model has to tell the five types apart, not just SP-vs-not).

Headline findings from the paper:

New types become usable. For Sec/SPIII (pilin) and Tat/SPII, v5.0 had no practical capability; v6.0 makes them predictable.
Better at low identity. At sequence identities below ~60%, SignalP 6.0 outperforms SignalP 5.0 — the protein-language-model backbone generalizes to distant homologs where statistics-based models degrade.
Cleavage-site precision improves across all organism groups.
Organism-agnostic use: because it doesn't depend on knowing the source organism, it's the first multi-class SP tool that is directly applicable to metagenomic / unknown-origin sequences.

Limitations (stated by the authors)

Depends on a correct N-terminus. Prediction assumes the start codon / protein start is correctly identified; a wrong gene model start breaks SP prediction.
Tat/SPI in Archaea is roughly on par with SignalP 5.0, not universally improved.
Region (n/h/c) predictions are evaluated qualitatively — there is no ground-truth region annotation, so region borders are "best-effort" (this is why slow mode exists for more accurate borders).
Very rare types (Tat/SPII, n=36) rest on tiny training sets; treat low-support predictions with appropriate caution.

Where SignalP fits — the DTU protein-sorting toolbox

SignalP answers "does this protein enter the secretory pathway, and where is it cut?" — not the full localization story. Companion tools:

| Tool | Question it answers | |------|--------------------| | DeepLoc 2.1 | Subcellular localization of eukaryotic proteins | | DeepLocPro 1.0 | Subcellular localization of prokaryotic proteins | | NetGPI | Does a eukaryotic protein have a GPI anchor? | | TargetP / TatP / LipoP | Older, more specialized sorting-signal predictors |

SignalP lineage (a quick history)

SignalP has been developed at DTU since the 1990s, evolving with ML methods:

SignalP 1.0 (Nielsen et al., 1997) — neural networks; Sec/SPI + cleavage site.
2.0 / 3.0 / 4.0 — added HMMs, improved discrimination of SPs from N-terminal transmembrane helices (4.0 was the long-time standard).
SignalP 5.0 (Almagro Armenteros et al., 2019) — deep learning (BiLSTM+CRF); three types (Sec/SPI, Sec/SPII, Tat/SPI); all domains of life.
SignalP 6.0 (Teufel et al., 2022) — protein language model + CRF; all five types; region decoding; organism-agnostic.

Background reading listed on the DTU page:

SignalP: The Evolution of a Web Server — ch. 17, Protein Bioinformatics (MIMB 2836), 2024.
A Brief History of Protein Sorting Prediction — The Protein Journal, 2019.
Practical Applications of Language Models in Protein Sorting Prediction: SignalP 6.0, DeepLoc 2.1, and DeepLocPro 1.0 — pp. 153–175 in LLMs in Protein Bioinformatics (MIMB 2941), 2025.

References

Teufel F, Almagro Armenteros JJ, Johansen AR, Gíslason MH, Pihl SI, Tsirigos KD, Winther O, Brunak S, von Heijne G, Nielsen H. "SignalP 6.0 predicts all five types of signal peptides using protein language models." Nature Biotechnology 40(7):1023–1025 (2022). DOI: 10.1038/s41587-021-01156-3.
Preprint: bioRxiv 2021.06.09.447770.
PubMed: 34980915 · PMC: PMC9287161
Server: https://services.healthtech.dtu.dk/services/SignalP-6.0/
Code: https://github.com/fteufel/signalp-6.0 · Mirror: https://dtu.biolib.com/SignalP-6
"Behind the Paper" blog: Springer Nature Research Communities.
ProtTrans / ProtBERT (encoder): Elnaggar et al., IEEE TPAMI (2021).