OloriBern/musique-hybrid-2000

CrossEncoder based on BAAI/bge-reranker-v2-m3

This is a Cross Encoder model finetuned from BAAI/bge-reranker-v2-m3 using the sentence-transformers library. It computes scores for pairs of texts, which can be used for text reranking and semantic search.

Model Details

Model Description

• Model Type: Cross Encoder
• Base model: BAAI/bge-reranker-v2-m3
• Maximum Sequence Length: 1024 tokens
• Number of Output Labels: 1 label

Model Sources

• Documentation: Sentence Transformers Documentation
• Documentation: Cross Encoder Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Cross Encoders on Hugging Face

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import CrossEncoder

# Download from the 🤗 Hub
model = CrossEncoder("cross_encoder_model_id")
# Get scores for pairs of texts
pairs = [
    ['What did the largest employer in the birthplace of Sonny Berman do about the Title IX complaint?', "New Haven, Connecticut. New Haven's economy originally was based in manufacturing, but the postwar period brought rapid industrial decline; the entire Northeast was affected, and medium-sized cities with large working-class populations, like New Haven, were hit particularly hard. Simultaneously, the growth and expansion of Yale University further affected the economic shift. Today, over half (56%) of the city's economy is now made up of services, in particular education and health care; Yale is the city's largest employer, followed by Yale – New Haven Hospital. Other large employers include St. Raphael Hospital, Smilow Cancer Hospital, Southern Connecticut State University, Assa Abloy Manufacturing, the Knights of Columbus headquarters, Higher One, Alexion Pharmaceuticals, Covidien and United Illuminating. Yale and Yale-New Haven are also among the largest employers in the state, and provide more $100,000+-salaried positions than any other employer in Connecticut.[citation needed]"],
    ['The city hockey club where Michael Langrish was born is named what?', 'Michael Langrish. Langrish was born in Southampton, Hampshire, the son of Douglas Frank Langrish and Brenda Florence Passingham. He was educated at King Edward VI School, Southampton, and Birmingham University, where he received a Bachelor of Social Sciences degree in 1967 and a Postgraduate Certificate in Education a year later. He was further educated at Fitzwilliam College, Cambridge, where he graduated with Bachelor of Arts and Master of Arts degrees in theology in 1973, and at Ridley Hall, Cambridge.'],
    ['When did the mint in the location of the headquarters of the publishers of Introduction to Christianity stop making circulating coins?', 'San Francisco Mint. The San Francisco Mint is a branch of the United States Mint and was opened in 1854 to serve the gold mines of the California Gold Rush. It quickly outgrew its first building and moved into a new one in 1874. This building, the Old United States Mint, also known affectionately as The Granite Lady, is one of the few that survived the great 1906 San Francisco earthquake. It served until 1937, when the present facility was opened.'],
    ['What county shares a border with the county Fiddlers Green, California, is located in?', 'Pinellas County, Florida. Pinellas County is a county located in the state of Florida. As of the 2010 census, the population was 916,542. The county is part of the Tampa -- St. Petersburg -- Clearwater, Florida Metropolitan Statistical Area. Clearwater is the county seat, and St. Petersburg is the largest city.'],
    ['Since what time has the company against whom the lawsuit of Apple vs Microsoft and HP was decided remained profitable?', "Macintosh. By March 2011, the market share of OS X in North America had increased to slightly over 14%. Whether the size of the Mac's market share and installed base is relevant, and to whom, is a hotly debated issue. Industry pundits have often called attention to the Mac's relatively small market share to predict Apple's impending doom, particularly in the early and mid-1990s when the company's future seemed bleakest. Others argue that market share is the wrong way to judge the Mac's success. Apple has positioned the Mac as a higher-end personal computer, and so it may be misleading to compare it to a budget PC. Because the overall market for personal computers has grown rapidly, the Mac's increasing sales numbers are effectively swamped by the industry's expanding sales volume as a whole. Apple's small market share, then, gives the impression that fewer people are using Macs than did ten years ago, when exactly the opposite is true. Soaring sales of the iPhone and iPad mean that the portion of Apple's profits represented by the Macintosh has declined in 2010, dropping to 24% from 46% two years earlier. Others try to de-emphasize market share, citing that it is rarely brought up in other industries. Regardless of the Mac's market share, Apple has remained profitable since Steve Jobs' return and the company's subsequent reorganization. Notably, a report published in the first quarter of 2008 found that Apple had a 14% market share in the personal computer market in the US, including 66% of all computers over $1,000. Market research indicates that Apple draws its customer base from a higher-income demographic than the mainstream personal computer market."],
]
scores = model.predict(pairs)
print(scores.shape)
# (5,)

# Or rank different texts based on similarity to a single text
ranks = model.rank(
    'What did the largest employer in the birthplace of Sonny Berman do about the Title IX complaint?',
    [
        "New Haven, Connecticut. New Haven's economy originally was based in manufacturing, but the postwar period brought rapid industrial decline; the entire Northeast was affected, and medium-sized cities with large working-class populations, like New Haven, were hit particularly hard. Simultaneously, the growth and expansion of Yale University further affected the economic shift. Today, over half (56%) of the city's economy is now made up of services, in particular education and health care; Yale is the city's largest employer, followed by Yale – New Haven Hospital. Other large employers include St. Raphael Hospital, Smilow Cancer Hospital, Southern Connecticut State University, Assa Abloy Manufacturing, the Knights of Columbus headquarters, Higher One, Alexion Pharmaceuticals, Covidien and United Illuminating. Yale and Yale-New Haven are also among the largest employers in the state, and provide more $100,000+-salaried positions than any other employer in Connecticut.[citation needed]",
        'Michael Langrish. Langrish was born in Southampton, Hampshire, the son of Douglas Frank Langrish and Brenda Florence Passingham. He was educated at King Edward VI School, Southampton, and Birmingham University, where he received a Bachelor of Social Sciences degree in 1967 and a Postgraduate Certificate in Education a year later. He was further educated at Fitzwilliam College, Cambridge, where he graduated with Bachelor of Arts and Master of Arts degrees in theology in 1973, and at Ridley Hall, Cambridge.',
        'San Francisco Mint. The San Francisco Mint is a branch of the United States Mint and was opened in 1854 to serve the gold mines of the California Gold Rush. It quickly outgrew its first building and moved into a new one in 1874. This building, the Old United States Mint, also known affectionately as The Granite Lady, is one of the few that survived the great 1906 San Francisco earthquake. It served until 1937, when the present facility was opened.',
        'Pinellas County, Florida. Pinellas County is a county located in the state of Florida. As of the 2010 census, the population was 916,542. The county is part of the Tampa -- St. Petersburg -- Clearwater, Florida Metropolitan Statistical Area. Clearwater is the county seat, and St. Petersburg is the largest city.',
        "Macintosh. By March 2011, the market share of OS X in North America had increased to slightly over 14%. Whether the size of the Mac's market share and installed base is relevant, and to whom, is a hotly debated issue. Industry pundits have often called attention to the Mac's relatively small market share to predict Apple's impending doom, particularly in the early and mid-1990s when the company's future seemed bleakest. Others argue that market share is the wrong way to judge the Mac's success. Apple has positioned the Mac as a higher-end personal computer, and so it may be misleading to compare it to a budget PC. Because the overall market for personal computers has grown rapidly, the Mac's increasing sales numbers are effectively swamped by the industry's expanding sales volume as a whole. Apple's small market share, then, gives the impression that fewer people are using Macs than did ten years ago, when exactly the opposite is true. Soaring sales of the iPhone and iPad mean that the portion of Apple's profits represented by the Macintosh has declined in 2010, dropping to 24% from 46% two years earlier. Others try to de-emphasize market share, citing that it is rarely brought up in other industries. Regardless of the Mac's market share, Apple has remained profitable since Steve Jobs' return and the company's subsequent reorganization. Notably, a report published in the first quarter of 2008 found that Apple had a 14% market share in the personal computer market in the US, including 66% of all computers over $1,000. Market research indicates that Apple draws its customer base from a higher-income demographic than the mainstream personal computer market.",
    ]
)
# [{'corpus_id': ..., 'score': ...}, {'corpus_id': ..., 'score': ...}, ...]

Evaluation

Metrics

Cross Encoder Binary Classification

• Datasets: validation and train_subset
• Evaluated with CEBinaryClassificationEvaluator

Metric	validation	train_subset
accuracy	0.7849	0.834
accuracy_threshold	0.7344	0.3486
f1	0.799	0.8477
f1_threshold	0.2754	0.3486
precision	0.7176	0.7993
recall	0.9012	0.9023
average_precision	0.8437	0.8815

Training Details

Training Dataset

Unnamed Dataset

• Size: 9,366 training samples
• Columns: sentence_0, sentence_1, and label
• Approximate statistics based on the first 1000 samples:

  	sentence_0	sentence_1	label
  type	string	string	float
  details	min: 38 characters mean: 96.15 characters max: 242 characters	min: 115 characters mean: 595.83 characters max: 1837 characters	min: 0.0 mean: 0.51 max: 1.0

• Samples:

  sentence_0	sentence_1	label
  What did the largest employer in the birthplace of Sonny Berman do about the Title IX complaint?	New Haven, Connecticut. New Haven's economy originally was based in manufacturing, but the postwar period brought rapid industrial decline; the entire Northeast was affected, and medium-sized cities with large working-class populations, like New Haven, were hit particularly hard. Simultaneously, the growth and expansion of Yale University further affected the economic shift. Today, over half (56%) of the city's economy is now made up of services, in particular education and health care; Yale is the city's largest employer, followed by Yale – New Haven Hospital. Other large employers include St. Raphael Hospital, Smilow Cancer Hospital, Southern Connecticut State University, Assa Abloy Manufacturing, the Knights of Columbus headquarters, Higher One, Alexion Pharmaceuticals, Covidien and United Illuminating. Yale and Yale-New Haven are also among the largest employers in the state, and provide more $100,000+-salaried positions than any other employer in Connecticut.[citation needed]	1.0
  The city hockey club where Michael Langrish was born is named what?	Michael Langrish. Langrish was born in Southampton, Hampshire, the son of Douglas Frank Langrish and Brenda Florence Passingham. He was educated at King Edward VI School, Southampton, and Birmingham University, where he received a Bachelor of Social Sciences degree in 1967 and a Postgraduate Certificate in Education a year later. He was further educated at Fitzwilliam College, Cambridge, where he graduated with Bachelor of Arts and Master of Arts degrees in theology in 1973, and at Ridley Hall, Cambridge.	1.0
  When did the mint in the location of the headquarters of the publishers of Introduction to Christianity stop making circulating coins?	San Francisco Mint. The San Francisco Mint is a branch of the United States Mint and was opened in 1854 to serve the gold mines of the California Gold Rush. It quickly outgrew its first building and moved into a new one in 1874. This building, the Old United States Mint, also known affectionately as The Granite Lady, is one of the few that survived the great 1906 San Francisco earthquake. It served until 1937, when the present facility was opened.	1.0

• Loss: BinaryCrossEntropyLoss with these parameters:

  {
      "activation_fn": "torch.nn.modules.linear.Identity",
      "pos_weight": null
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• per_device_train_batch_size: 2
• per_device_eval_batch_size: 2

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: steps
• prediction_loss_only: True
• per_device_train_batch_size: 2
• per_device_eval_batch_size: 2
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 5e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1
• num_train_epochs: 3
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.0
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: False
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: False
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• dispatch_batches: None
• split_batches: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• eval_use_gather_object: False
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

Epoch	Step	Training Loss	validation_average_precision	train_subset_average_precision
0.1067	250	-	0.7717	0.7739
0.2135	500	0.7773	0.7809	0.7917
0.3202	750	-	0.8008	0.8120
0.4270	1000	0.5897	0.8101	0.8180
0.5337	1250	-	0.8106	0.8361
0.6405	1500	0.5731	0.8196	0.8375
0.7472	1750	-	0.8223	0.8488
0.8540	2000	0.5599	0.8238	0.8567
0.9607	2250	-	0.8243	0.8588
1.0	2342	-	0.8176	-
1.0675	2500	0.5293	0.8271	0.8665
1.1742	2750	-	0.8168	0.8694
1.2810	3000	0.5225	0.8359	0.8743
1.3877	3250	-	0.8350	0.8755
1.4944	3500	0.5156	0.8327	0.8769
1.6012	3750	-	0.8390	0.8792
1.7079	4000	0.5181	0.8346	0.8823
1.8147	4250	-	0.8437	0.8815

Framework Versions

• Python: 3.11.13
• Sentence Transformers: 5.2.2
• Transformers: 4.44.2
• PyTorch: 2.10.0+cu128
• Accelerate: 1.12.0
• Datasets: 4.0.0
• Tokenizers: 0.19.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}