variants-ner-modernbert-base

This model is a fine-tuned version of answerdotai/ModernBERT-base on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 0.0705
• Precision: 0.8194
• Recall: 0.8887
• F1: 0.8526
• Accuracy: 0.9901

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 3e-05
• train_batch_size: 1
• eval_batch_size: 8
• seed: 42
• gradient_accumulation_steps: 8
• total_train_batch_size: 8
• optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• num_epochs: 30.0

Training results

Training Loss	Epoch	Step	Accuracy	F1	Validation Loss	Precision	Recall
0.6763	1.0	757	0.9843	0.7304	0.0400	0.6640	0.8115
0.2144	2.0	1514	0.9882	0.8050	0.0302	0.7767	0.8353
0.1276	3.0	2271	0.9889	0.7944	0.0272	0.7766	0.8131
0.0865	4.0	3028	0.9891	0.8207	0.0309	0.7947	0.8486
0.0529	5.0	3785	0.9882	0.8090	0.0341	0.7666	0.8564
0.0385	6.0	4542	0.9904	0.8383	0.0337	0.8094	0.8694
0.0212	7.0	5299	0.9901	0.8388	0.0384	0.8121	0.8673
0.0132	8.0	6056	0.9896	0.8273	0.0417	0.8002	0.8564
0.0088	9.0	6813	0.9897	0.8355	0.0473	0.8087	0.8641
0.0055	10.0	7570	0.9898	0.8375	0.0498	0.8078	0.8694
0.0346	11.0	8327	0.0458	0.7958	0.8668	0.8298	0.9891
0.0334	12.0	9084	0.0395	0.7841	0.8618	0.8211	0.9893
0.0241	13.0	9841	0.0433	0.8011	0.8544	0.8269	0.9887
0.0114	14.0	10598	0.0488	0.7874	0.8519	0.8184	0.9892
0.0102	15.0	11355	0.0500	0.8068	0.8698	0.8371	0.9897
0.0087	16.0	12112	0.0557	0.8107	0.8703	0.8395	0.9895
0.0070	17.0	12869	0.0620	0.8016	0.8735	0.8360	0.9892
0.0050	18.0	13626	0.0503	0.8074	0.8597	0.8327	0.9893
0.0089	19.0	14383	0.0561	0.8333	0.8592	0.8460	0.9899
0.0058	20.0	15140	0.0569	0.8051	0.8364	0.8205	0.9891
0.0030	21.0	15897	0.0581	0.8112	0.8682	0.8387	0.9900
0.0015	22.0	16654	0.0608	0.8206	0.8689	0.8441	0.9901
0.0004	23.0	17411	0.0633	0.8180	0.8645	0.8406	0.9899
0.0005	24.0	18168	0.0663	0.8195	0.8793	0.8484	0.9901
0.0001	25.0	18925	0.0705	0.8194	0.8887	0.8526	0.9901
0.0002	26.0	19682	0.0687	0.8254	0.8686	0.8464	0.9901
0.0002	27.0	20439	0.0695	0.8220	0.8784	0.8493	0.9901
0.0000	28.0	21196	0.0717	0.8253	0.8728	0.8484	0.9901
0.0001	29.0	21953	0.0735	0.8245	0.8776	0.8502	0.9901
0.0000	30.0	22710	0.0741	0.8254	0.8769	0.8503	0.9901

Framework versions

• Transformers 5.2.0
• Pytorch 2.10.0+cu128
• Datasets 4.6.0
• Tokenizers 0.22.2

NER Performance Analysis

Evaluation Dataset

Dataset: gyorilab/variants_ner_benchmark

Notes

Model labels not present in the evaluation dataset were mapped to O: B-Gene, I-Gene

Overall Seqeval Metrics

Metric	Value
Precision	0.9132
Recall	0.9307
F1	0.9218
Accuracy	0.9944

Entity-level Classification Report

Entity	Precision	Recall	F1	Support
CopyNumberVariant	0.8283	0.8586	0.8432	191
DNAMutation	0.8597	0.8944	0.8767	322
ProteinMutation	0.9182	0.9359	0.9270	468
SNP	1.0000	0.9920	0.9960	375
micro avg	0.9132	0.9307	0.9218	1,356
macro avg	0.9016	0.9202	0.9107	1,356
weighted avg	0.9143	0.9307	0.9223	1,356

Token-level Confusion Matrix (Entity Type, rows=true, cols=pred)

True \ Pred	O	CopyNumberVariant	DNAMutation	ProteinMutation	SNP
O	90,326	48	32	61	0
CopyNumberVariant	204	2,257	16	0	0
DNAMutation	56	0	1,543	14	0
ProteinMutation	78	0	7	1,528	0
SNP	11	0	0	0	1,544

Entity Instance Counts

Entity Type	True Instances	Pred Instances
CopyNumberVariant	191	198
DNAMutation	322	335
ProteinMutation	468	477
SNP	375	372

Sequence Truncation Summary

Metric	Value
Model max sequence length	8192
Sequences classified as whole	383 / 383
Sequences truncated	0 / 383
Entities not fully evaluated due to truncation	0 / 1,356