knoxel/conformalesm-paper-starter

ConformalESM: Distribution-Free Uncertainty Quantification for Protein Language Models

Cites: Lin et al. 2022, "Evolutionary Scale Prediction of Atomic Level Protein Structure with a Language Model", Science.

Novelty

This is the first work to apply conformal prediction and temperature scaling to protein language models (PLMs). While 10+ papers apply these techniques to general LLMs (2023-2024), zero have ported them to the protein domain.

Key Results

Method	Accuracy	ECE	Avg Set Size (α=0.10)	Coverage
Baseline ESM-2	61.3%	0.147	—	—
+ Temperature Scaling	61.3%	0.058 (-61%)	—	—
+ Conformal Prediction	—	—	1.79	89.9%
+ Class-Conditional Conformal	—	—	1.63	89.9%

Per-class (α=0.10, class-conditional):

• Coil (C): coverage=90.8%, avg set=1.15 (most confident)
• Helix (H): coverage=88.6%, avg set=1.99
• Sheet (E): coverage=90.1%, avg set=1.94

How to Run

pip install transformers datasets torch scikit-learn
python conformalesm_full.py

No GPU required. Runs in ~5 minutes on CPU.

Dataset

• lamm-mit/protein_secondary_structure_from_PDB (125K sequences)
• 500 calibration / 500 test proteins
• 382,675 total residues evaluated