jmroth/nlp-reranker-finetuned-v3-2epoch

CrossEncoder based on cross-encoder/ms-marco-MiniLM-L6-v2

This is a Cross Encoder model finetuned from cross-encoder/ms-marco-MiniLM-L6-v2 using the sentence-transformers library. It computes scores for pairs of texts, which can be used for text reranking and semantic search.

Model Details

Model Description

• Model Type: Cross Encoder
• Base model: cross-encoder/ms-marco-MiniLM-L6-v2
• Maximum Sequence Length: 512 tokens
• Number of Output Labels: 1 label
• Supported Modality: Text

Model Sources

• Documentation: Sentence Transformers Documentation
• Documentation: Cross Encoder Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Cross Encoders on Hugging Face

Full Model Architecture

CrossEncoder(
  (0): Transformer({'transformer_task': 'sequence-classification', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'logits'}}, 'module_output_name': 'scores', 'architecture': 'BertForSequenceClassification'})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import CrossEncoder

# Download from the 🤗 Hub
model = CrossEncoder("jmroth/nlp-reranker-finetuned")
# Get scores for pairs of inputs
pairs = [
    ['Not only is there no scientific evidence that CO2 is a pollutant, higher CO2 concentrations actually help ecosystems support more plant and animal life.', 'Higher carbon dioxide concentrations will favourably affect plant growth and demand for water.'],
    ['Not only is there no scientific evidence that CO2 is a pollutant, higher CO2 concentrations actually help ecosystems support more plant and animal life.', 'Plants can grow as much as 50 percent faster in concentrations of 1,000 ppm CO 2 when compared with ambient conditions, though this assumes no change in climate and no limitation on other nutrients.'],
    ['Not only is there no scientific evidence that CO2 is a pollutant, higher CO2 concentrations actually help ecosystems support more plant and animal life.', 'At very high concentrations (100 times atmospheric concentration, or greater), carbon dioxide can be toxic to animal life, so raising the concentration to 10,000 ppm (1%) or higher for several hours will eliminate pests such as whiteflies and spider mites in a greenhouse.'],
    ['Not only is there no scientific evidence that CO2 is a pollutant, higher CO2 concentrations actually help ecosystems support more plant and animal life.', 'Use of fertilizers are beneficial in providing nutrients to plants although they have some negative environmental effects.'],
    ['Not only is there no scientific evidence that CO2 is a pollutant, higher CO2 concentrations actually help ecosystems support more plant and animal life.', 'Studies have shown that higher CO2 levels lead to reduced plant uptake of nitrogen (and a smaller number showing the same for trace elements such as zinc) resulting in crops with lower nutritional value.'],
]
scores = model.predict(pairs)
print(scores)
# [0.3819 0.4154 0.5552 0.146  0.6721]

# Or rank different texts based on similarity to a single text
ranks = model.rank(
    'Not only is there no scientific evidence that CO2 is a pollutant, higher CO2 concentrations actually help ecosystems support more plant and animal life.',
    [
        'Higher carbon dioxide concentrations will favourably affect plant growth and demand for water.',
        'Plants can grow as much as 50 percent faster in concentrations of 1,000 ppm CO 2 when compared with ambient conditions, though this assumes no change in climate and no limitation on other nutrients.',
        'At very high concentrations (100 times atmospheric concentration, or greater), carbon dioxide can be toxic to animal life, so raising the concentration to 10,000 ppm (1%) or higher for several hours will eliminate pests such as whiteflies and spider mites in a greenhouse.',
        'Use of fertilizers are beneficial in providing nutrients to plants although they have some negative environmental effects.',
        'Studies have shown that higher CO2 levels lead to reduced plant uptake of nitrogen (and a smaller number showing the same for trace elements such as zinc) resulting in crops with lower nutritional value.',
    ]
)
# [{'corpus_id': ..., 'score': ...}, {'corpus_id': ..., 'score': ...}, ...]

Training Details

Training Dataset

Unnamed Dataset

• Size: 18,858 training samples
• Columns: sentence1, sentence2, and label
• Approximate statistics based on the first 1000 samples:

  	sentence1	sentence2	label
  type	string	string	float
  details	min: 9 tokens mean: 26.48 tokens max: 54 tokens	min: 4 tokens mean: 33.89 tokens max: 475 tokens	min: 0.0 mean: 0.24 max: 1.0

• Samples:

  sentence1	sentence2	label
  Not only is there no scientific evidence that CO2 is a pollutant, higher CO2 concentrations actually help ecosystems support more plant and animal life.	Higher carbon dioxide concentrations will favourably affect plant growth and demand for water.	1.0
  Not only is there no scientific evidence that CO2 is a pollutant, higher CO2 concentrations actually help ecosystems support more plant and animal life.	Plants can grow as much as 50 percent faster in concentrations of 1,000 ppm CO 2 when compared with ambient conditions, though this assumes no change in climate and no limitation on other nutrients.	1.0
  Not only is there no scientific evidence that CO2 is a pollutant, higher CO2 concentrations actually help ecosystems support more plant and animal life.	At very high concentrations (100 times atmospheric concentration, or greater), carbon dioxide can be toxic to animal life, so raising the concentration to 10,000 ppm (1%) or higher for several hours will eliminate pests such as whiteflies and spider mites in a greenhouse.	1.0

• Loss: BinaryCrossEntropyLoss with these parameters:

  {
      "activation_fn": "torch.nn.modules.linear.Identity",
      "pos_weight": null
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 16
• learning_rate: 1e-06
• weight_decay: 0.01
• num_train_epochs: 2
• warmup_steps: 0.1
• fp16: True

All Hyperparameters

Click to expand

• do_predict: False
• prediction_loss_only: True
• per_device_train_batch_size: 16
• per_device_eval_batch_size: 8
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 1e-06
• weight_decay: 0.01
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 2
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: None
• warmup_ratio: None
• warmup_steps: 0.1
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• enable_jit_checkpoint: False
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• use_cpu: False
• seed: 42
• data_seed: None
• bf16: False
• fp16: True
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: -1
• ddp_backend: None
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch_fused
• optim_args: None
• group_by_length: False
• length_column_name: length
• project: huggingface
• trackio_space_id: trackio
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• hub_revision: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_for_metrics: []
• eval_do_concat_batches: True
• auto_find_batch_size: False
• full_determinism: False
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_num_input_tokens_seen: no
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• liger_kernel_config: None
• eval_use_gather_object: False
• average_tokens_across_devices: True
• use_cache: False
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

Click to expand

Epoch	Step	Training Loss
0.0085	10	1.0225
0.0170	20	1.2386
0.0254	30	1.4291
0.0339	40	1.1493
0.0424	50	1.2889
0.0509	60	1.1320
0.0594	70	1.1917
0.0679	80	0.9609
0.0763	90	0.9924
0.0848	100	0.8889
0.0933	110	1.1595
0.1018	120	1.1698
0.1103	130	0.7514
0.1187	140	1.1247
0.1272	150	1.4985
0.1357	160	0.8702
0.1442	170	1.0263
0.1527	180	1.0476
0.1612	190	0.8893
0.1696	200	1.0815
0.1781	210	1.2692
0.1866	220	0.9813
0.1951	230	0.9644
0.2036	240	1.3115
0.2120	250	0.8838
0.2205	260	1.2384
0.2290	270	1.0969
0.2375	280	1.1088
0.2460	290	1.1837
0.2545	300	1.1290
0.2629	310	1.4019
0.2714	320	1.0076
0.2799	330	0.7441
0.2884	340	0.8919
0.2969	350	0.8093
0.3053	360	0.9434
0.3138	370	0.9470
0.3223	380	0.9463
0.3308	390	0.9569
0.3393	400	0.7941
0.3478	410	0.8773
0.3562	420	0.7699
0.3647	430	0.8987
0.3732	440	0.9256
0.3817	450	0.7964
0.3902	460	0.6621
0.3986	470	0.8111
0.4071	480	0.7892
0.4156	490	0.9585
0.4241	500	0.9916
0.4326	510	0.9482
0.4411	520	0.8684
0.4495	530	0.7502
0.4580	540	0.8336
0.4665	550	1.1090
0.4750	560	0.9581
0.4835	570	0.8101
0.4919	580	0.9455
0.5004	590	0.8295
0.5089	600	0.7811
0.5174	610	0.7830
0.5259	620	0.5916
0.5344	630	0.7764
0.5428	640	0.9287
0.5513	650	0.6775
0.5598	660	0.7411
0.5683	670	0.5842
0.5768	680	0.7475
0.5852	690	0.8193
0.5937	700	0.8532
0.6022	710	0.5871
0.6107	720	0.7317
0.6192	730	0.7078
0.6277	740	0.6542
0.6361	750	0.7618
0.6446	760	0.8174
0.6531	770	0.7401
0.6616	780	0.6917
0.6701	790	0.6621
0.6785	800	0.9183
0.6870	810	0.5937
0.6955	820	0.7405
0.7040	830	0.5551
0.7125	840	0.6287
0.7209	850	0.6417
0.7294	860	0.7638
0.7379	870	0.8658
0.7464	880	0.7059
0.7549	890	0.8904
0.7634	900	0.8676
0.7718	910	0.7325
0.7803	920	0.6385
0.7888	930	0.6598
0.7973	940	0.6510
0.8058	950	0.7248
0.8142	960	0.7579
0.8227	970	0.7303
0.8312	980	0.7443
0.8397	990	0.8274
0.8482	1000	0.6730
0.8567	1010	0.7113
0.8651	1020	0.7275
0.8736	1030	0.7184
0.8821	1040	0.6673
0.8906	1050	0.7256
0.8991	1060	0.5711
0.9075	1070	0.6401
0.9160	1080	0.7851
0.9245	1090	0.6318
0.9330	1100	0.6304
0.9415	1110	0.7506
0.9500	1120	0.6707
0.9584	1130	0.6347
0.9669	1140	0.6550
0.9754	1150	0.7638
0.9839	1160	0.6765
0.9924	1170	0.7170
1.0008	1180	0.5122
1.0093	1190	0.5915
1.0178	1200	0.6447
1.0263	1210	0.6607
1.0348	1220	0.6470
1.0433	1230	0.7441
1.0517	1240	0.7242
1.0602	1250	0.6026
1.0687	1260	0.5547
1.0772	1270	0.7711
1.0857	1280	0.7076
1.0941	1290	0.7744
1.1026	1300	0.6116
1.1111	1310	0.5839
1.1196	1320	0.7205
1.1281	1330	0.7353
1.1366	1340	0.5883
1.1450	1350	0.6570
1.1535	1360	0.4835
1.1620	1370	0.7106
1.1705	1380	0.7762
1.1790	1390	0.6035
1.1874	1400	0.7114
1.1959	1410	0.6802
1.2044	1420	0.6325
1.2129	1430	0.6928
1.2214	1440	0.6735
1.2299	1450	0.5701
1.2383	1460	0.6328
1.2468	1470	0.6378
1.2553	1480	0.6459
1.2638	1490	0.7120
1.2723	1500	0.5439
1.2807	1510	0.7851
1.2892	1520	0.6216
1.2977	1530	0.6349
1.3062	1540	0.5756
1.3147	1550	0.6687
1.3232	1560	0.5545
1.3316	1570	0.6519
1.3401	1580	0.6732
1.3486	1590	0.5427
1.3571	1600	0.5249
1.3656	1610	0.6435
1.3740	1620	0.6197
1.3825	1630	0.6038
1.3910	1640	0.6327
1.3995	1650	0.6852
1.4080	1660	0.6085
1.4165	1670	0.5865
1.4249	1680	0.4982
1.4334	1690	0.6043
1.4419	1700	0.6524
1.4504	1710	0.5401
1.4589	1720	0.5536
1.4673	1730	0.5866
1.4758	1740	0.6423
1.4843	1750	0.6148
1.4928	1760	0.6974
1.5013	1770	0.5371
1.5098	1780	0.6354
1.5182	1790	0.5086
1.5267	1800	0.6768
1.5352	1810	0.5401
1.5437	1820	0.6621
1.5522	1830	0.5422
1.5606	1840	0.5071
1.5691	1850	0.5319
1.5776	1860	0.5842
1.5861	1870	0.6710
1.5946	1880	0.6577
1.6031	1890	0.5490
1.6115	1900	0.4426
1.6200	1910	0.4777
1.6285	1920	0.6077
1.6370	1930	0.5324
1.6455	1940	0.6458
1.6539	1950	0.7076
1.6624	1960	0.5682
1.6709	1970	0.5779
1.6794	1980	0.6830
1.6879	1990	0.5693
1.6964	2000	0.5619
1.7048	2010	0.5912
1.7133	2020	0.5306
1.7218	2030	0.6202
1.7303	2040	0.5903
1.7388	2050	0.5880
1.7472	2060	0.5909
1.7557	2070	0.5551
1.7642	2080	0.5758
1.7727	2090	0.4812
1.7812	2100	0.6097
1.7897	2110	0.7131
1.7981	2120	0.6190
1.8066	2130	0.4986
1.8151	2140	0.6212
1.8236	2150	0.6456
1.8321	2160	0.5915
1.8405	2170	0.5032
1.8490	2180	0.5983
1.8575	2190	0.4840
1.8660	2200	0.6276
1.8745	2210	0.5345
1.8830	2220	0.5878
1.8914	2230	0.5753
1.8999	2240	0.6156
1.9084	2250	0.6027
1.9169	2260	0.5228
1.9254	2270	0.4246
1.9338	2280	0.5840
1.9423	2290	0.5531
1.9508	2300	0.4773
1.9593	2310	0.6268
1.9678	2320	0.6145
1.9763	2330	0.5289
1.9847	2340	0.6227
1.9932	2350	0.5717

Training Time

• Training: 1.9 minutes

Framework Versions

• Python: 3.12.13
• Sentence Transformers: 5.4.1
• Transformers: 5.0.0
• PyTorch: 2.10.0+cu128
• Accelerate: 1.13.0
• Datasets: 4.0.0
• Tokenizers: 0.22.2

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}