jameswatanabegoogle2024/adl-hw3-reranker

CrossEncoder based on cross-encoder/ms-marco-MiniLM-L12-v2

This is a Cross Encoder model finetuned from cross-encoder/ms-marco-MiniLM-L12-v2 using the sentence-transformers library. It computes scores for pairs of texts, which can be used for text reranking and semantic search.

Model Details

Model Description

• Model Type: Cross Encoder
• Base model: cross-encoder/ms-marco-MiniLM-L12-v2
• Maximum Sequence Length: 512 tokens
• Number of Output Labels: 1 label

Model Sources

• Documentation: Sentence Transformers Documentation
• Documentation: Cross Encoder Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Cross Encoders on Hugging Face

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import CrossEncoder

# Download from the 🤗 Hub
model = CrossEncoder("cross_encoder_model_id")
# Get scores for pairs of texts
pairs = [
    ['Besides the album Fight Like a Girl, are there any other interesting aspects about this article?', 'Fight Like a Girl (Bomshel song) "Fight Like a Girl" is a song recorded by American country music duo Bomshel. It was released in February 2009 as the second single and title track from their debut album "Fight Like a Girl", which was released on October 20, 2009 via Curb Records. Co-written by the duo\'s two members, Kristy Osmunson and Kelley Shepard, along with Bob Regan. The song received mixed reviews from critics who found the lyrical content cliché and indifferent to other similar songs. " Fight Like a Girl" peaked at number 30 on the "Billboard" Hot Country Songs chart. The accompanying music video for the song was directed by Eric Welch and features Bombshel performing with their guitars in front of different backgrounds. The song is a mid-tempo mostly accompanied by electric guitar. The song\'s lyrics tell of a young girl who is facing problems in her life, such as being teased on the playground at school. Her mother gives her advice by telling her to "fight like a girl." Leeann Ward of Country Universe gave the song a C rating, calling the lyrics "potentially cliched." C.M. Wilcox of The 9513, despite saying that "the singing isn\'t completely unlikeable and neither is the song," gave it a thumbs-down and compared its theme to Brooks & Dunn\'s "Cowgirls Don\'t Cry. " Bobby Peacock described the song more positively in his review of the album, saying that it had a more commercial sound than the rest of the album, but adding that it was "an appropriate introduction to the rebooted Bomshel." A music video, directed by Eric Welch, was released for the song on May 20, 2009.'],
    ['Did Ricky Martin try again for Menudo?', 'Charlie Masso Carlos Javier Rivera Masso (born June 13, 1969 in San Juan, Puerto Rico) is a Puerto Rican singer and actor. Masso is a former member of Menudo, joining the band at the height of Menudo\'s worldwide fame. He replaced Rene Farrait and quickly became a fan favorite with his participation on the albums of "Por Amor" and "Una Aventura Llamada Menudo", which was the soundtrack of Menudo\'s second movie. Charlie recorded 14 albums with Menudo, two were nominated for a Grammy Award for Best Latin Pop Album in 1984 and 85. Masso began his life as a Menudo the very moment that one of Menudo\'s members of the group\'s "golden era," Farrait, left the group. However, Masso\'s presence helped extend the golden era an additional three years, and when he left the group in 1987, he was the last member of the group\'s golden years to leave. Masso was a member of Menudo during their most commercially successful period in the States, which included the group\'s highest-charting U.S. single, "Hold Me" (#62 on the Billboard Hot 100.) Masso\'s bandmates included Xavier Serbia, Johnny Lozada, Ricky Melendez, Miguel Cancel, Ray Reyes, Roy Rosselló, Robby Rosa and Ricky Martin, among others. He returned to Menudo to complete a tour in Brazil, their 1987 Summer in the Streets tour, after Robby Rosa quit the group, to introduce new member Ruben Gomez and for their last tour in the Philippines when Ralphy Rodriguez was pulled out of the group by his parents late that year.'],
    ['Other than family, are there any other interesting aspects about the article on Vincent Price?', 'Vincent Price filmography This is the filmography of Vincent Price (May 27, 1911 – October 25, 1993), which includes appearances in theatre and television. Price made his theatre debut in the Gate Theatre\'s production of "Chicago" (1935), followed by work on Broadway. Under contract to Universal, Price traveled to Hollywood, making his screen debut in "Service de Luxe" (1938). By the 1960s, Vincent Price was working almost exclusively in the horror genre and teen film genres. Price\'s final film was "Edward Scissorhands" (1990) and the TV movie "The Heart of Justice" (1992) was his last screen appearance. Price died in Los Angeles, California on October 25, 1993.'],
    ['who is the leader of The Aquabats?', 'Chronicled in both live-action and animated segments, The Aquabats! Super Show! is centered around the adventures of The Aquabats, a group of superhero rock musicians who travel the countryside on a self-appointed mission to fight evil and "destroy boredom", protecting the world from the villains and creatures who threaten to destroy it while aiming to become a famous rock and roll band in their own right. The Aquabats consist of singer The MC Bat Commander (Christian Jacobs), the swaggering leader of the group; bassist Crash McLarson (Chad Larson), who can grow up to 100 feet in size; drummer Ricky Fitness (Richard Falomir), who has the power of super speed; guitarist EagleBones Falconhawk (Ian Fowles), who\'s armed with a laser-shooting electric guitar; and keyboardist Jimmy the Robot (James R. Briggs, Jr.), an android. Despite their superhuman strengths and abilities, The Aquabats are quite bumbling, disorganized, and sometimes cowardly when faced with danger; this has in fact led them to be labeled "the world\'s most inept superheroes". The band lives and travels by way of their "Battletram", a modified recreational vehicle which, despite its small exterior, has an implausibly massive interior (similar to the TARDIS from Doctor Who or The Big Bologna from The Kids From C.A.P.E.R. ), which contains, among many things, a science lab, a command center, and a living room. The Aquabats\' origin story was left intentionally vague throughout the series, a choice Jacobs explains was done for the sake of the viewer\'s imagination, as kids are more accepting of the inherent absurdity of the premise than adults tend to be: "\'There\'s five guys.'],
    ["Where was Frank Zappa's father from?", 'Zappa Plays Zappa  Zappa Plays Zappa (previously momentarily renamed as Dweezil Zappa Plays Frank Zappa) is an American tribute act led by Dweezil Zappa, the eldest son of late American composer and musician Frank Zappa, devoted to performing the music of Frank Zappa. The band debuted in 2006 with shows in Europe, Canada, and the United States during May and June (the tour was also known as "Zappa Plays Zappa: Tour de Frank"\'). The shows presented a collection of Frank Zappa\'s rock-oriented compositions from the 1960s to the late 1970s. Apart from Dweezil Zappa on lead guitar, many of the band members previously played with Frank Zappa. Among those, Napoleon Murphy Brock (sax, flute, and vocals) was an integral part of the band, while drummer/vocalist Terry Bozzio and guitarist Steve Vai performed as guests in parts of the shows. At several shows the live band played along with audio and video recordings of Frank Zappa himself, notably portions of "Chunga\'s Revenge", "Dumb All Over", "Cosmik Debris", and "Muffin Man". After a break, the band played again in the U.S. during the fall of 2006, including a show in New York on October 31. This revived Frank Zappa\'s tradition of playing Halloween shows in New York. A DVD documenting the 2006 tour was released in early 2008. In July and August 2007, the band played a North American tour, with a core lineup similar to that of the 2006 band. The band then played in Europe during September and October before returning to the US, starting with another Halloween show in New York. Special guest on the tour was vocalist and guitarist Ray White, a Zappa stalwart performer in the 1970s and early 1980s.'],
]
scores = model.predict(pairs)
print(scores.shape)
# (5,)

# Or rank different texts based on similarity to a single text
ranks = model.rank(
    'Besides the album Fight Like a Girl, are there any other interesting aspects about this article?',
    [
        'Fight Like a Girl (Bomshel song) "Fight Like a Girl" is a song recorded by American country music duo Bomshel. It was released in February 2009 as the second single and title track from their debut album "Fight Like a Girl", which was released on October 20, 2009 via Curb Records. Co-written by the duo\'s two members, Kristy Osmunson and Kelley Shepard, along with Bob Regan. The song received mixed reviews from critics who found the lyrical content cliché and indifferent to other similar songs. " Fight Like a Girl" peaked at number 30 on the "Billboard" Hot Country Songs chart. The accompanying music video for the song was directed by Eric Welch and features Bombshel performing with their guitars in front of different backgrounds. The song is a mid-tempo mostly accompanied by electric guitar. The song\'s lyrics tell of a young girl who is facing problems in her life, such as being teased on the playground at school. Her mother gives her advice by telling her to "fight like a girl." Leeann Ward of Country Universe gave the song a C rating, calling the lyrics "potentially cliched." C.M. Wilcox of The 9513, despite saying that "the singing isn\'t completely unlikeable and neither is the song," gave it a thumbs-down and compared its theme to Brooks & Dunn\'s "Cowgirls Don\'t Cry. " Bobby Peacock described the song more positively in his review of the album, saying that it had a more commercial sound than the rest of the album, but adding that it was "an appropriate introduction to the rebooted Bomshel." A music video, directed by Eric Welch, was released for the song on May 20, 2009.',
        'Charlie Masso Carlos Javier Rivera Masso (born June 13, 1969 in San Juan, Puerto Rico) is a Puerto Rican singer and actor. Masso is a former member of Menudo, joining the band at the height of Menudo\'s worldwide fame. He replaced Rene Farrait and quickly became a fan favorite with his participation on the albums of "Por Amor" and "Una Aventura Llamada Menudo", which was the soundtrack of Menudo\'s second movie. Charlie recorded 14 albums with Menudo, two were nominated for a Grammy Award for Best Latin Pop Album in 1984 and 85. Masso began his life as a Menudo the very moment that one of Menudo\'s members of the group\'s "golden era," Farrait, left the group. However, Masso\'s presence helped extend the golden era an additional three years, and when he left the group in 1987, he was the last member of the group\'s golden years to leave. Masso was a member of Menudo during their most commercially successful period in the States, which included the group\'s highest-charting U.S. single, "Hold Me" (#62 on the Billboard Hot 100.) Masso\'s bandmates included Xavier Serbia, Johnny Lozada, Ricky Melendez, Miguel Cancel, Ray Reyes, Roy Rosselló, Robby Rosa and Ricky Martin, among others. He returned to Menudo to complete a tour in Brazil, their 1987 Summer in the Streets tour, after Robby Rosa quit the group, to introduce new member Ruben Gomez and for their last tour in the Philippines when Ralphy Rodriguez was pulled out of the group by his parents late that year.',
        'Vincent Price filmography This is the filmography of Vincent Price (May 27, 1911 – October 25, 1993), which includes appearances in theatre and television. Price made his theatre debut in the Gate Theatre\'s production of "Chicago" (1935), followed by work on Broadway. Under contract to Universal, Price traveled to Hollywood, making his screen debut in "Service de Luxe" (1938). By the 1960s, Vincent Price was working almost exclusively in the horror genre and teen film genres. Price\'s final film was "Edward Scissorhands" (1990) and the TV movie "The Heart of Justice" (1992) was his last screen appearance. Price died in Los Angeles, California on October 25, 1993.',
        'Chronicled in both live-action and animated segments, The Aquabats! Super Show! is centered around the adventures of The Aquabats, a group of superhero rock musicians who travel the countryside on a self-appointed mission to fight evil and "destroy boredom", protecting the world from the villains and creatures who threaten to destroy it while aiming to become a famous rock and roll band in their own right. The Aquabats consist of singer The MC Bat Commander (Christian Jacobs), the swaggering leader of the group; bassist Crash McLarson (Chad Larson), who can grow up to 100 feet in size; drummer Ricky Fitness (Richard Falomir), who has the power of super speed; guitarist EagleBones Falconhawk (Ian Fowles), who\'s armed with a laser-shooting electric guitar; and keyboardist Jimmy the Robot (James R. Briggs, Jr.), an android. Despite their superhuman strengths and abilities, The Aquabats are quite bumbling, disorganized, and sometimes cowardly when faced with danger; this has in fact led them to be labeled "the world\'s most inept superheroes". The band lives and travels by way of their "Battletram", a modified recreational vehicle which, despite its small exterior, has an implausibly massive interior (similar to the TARDIS from Doctor Who or The Big Bologna from The Kids From C.A.P.E.R. ), which contains, among many things, a science lab, a command center, and a living room. The Aquabats\' origin story was left intentionally vague throughout the series, a choice Jacobs explains was done for the sake of the viewer\'s imagination, as kids are more accepting of the inherent absurdity of the premise than adults tend to be: "\'There\'s five guys.',
        'Zappa Plays Zappa  Zappa Plays Zappa (previously momentarily renamed as Dweezil Zappa Plays Frank Zappa) is an American tribute act led by Dweezil Zappa, the eldest son of late American composer and musician Frank Zappa, devoted to performing the music of Frank Zappa. The band debuted in 2006 with shows in Europe, Canada, and the United States during May and June (the tour was also known as "Zappa Plays Zappa: Tour de Frank"\'). The shows presented a collection of Frank Zappa\'s rock-oriented compositions from the 1960s to the late 1970s. Apart from Dweezil Zappa on lead guitar, many of the band members previously played with Frank Zappa. Among those, Napoleon Murphy Brock (sax, flute, and vocals) was an integral part of the band, while drummer/vocalist Terry Bozzio and guitarist Steve Vai performed as guests in parts of the shows. At several shows the live band played along with audio and video recordings of Frank Zappa himself, notably portions of "Chunga\'s Revenge", "Dumb All Over", "Cosmik Debris", and "Muffin Man". After a break, the band played again in the U.S. during the fall of 2006, including a show in New York on October 31. This revived Frank Zappa\'s tradition of playing Halloween shows in New York. A DVD documenting the 2006 tour was released in early 2008. In July and August 2007, the band played a North American tour, with a core lineup similar to that of the 2006 band. The band then played in Europe during September and October before returning to the US, starting with another Halloween show in New York. Special guest on the tour was vocalist and guitarist Ray White, a Zappa stalwart performer in the 1970s and early 1980s.',
    ]
)
# [{'corpus_id': ..., 'score': ...}, {'corpus_id': ..., 'score': ...}, ...]

Evaluation

Metrics

Cross Encoder Reranking

• Dataset: dev
• Evaluated with CERerankingEvaluator with these parameters:

  {
      "at_k": 10
  }
  

Metric	Value
map	0.995
mrr@10	0.995
ndcg@10	0.9972

Training Details

Training Dataset

Unnamed Dataset

• Size: 157,630 training samples
• Columns: sentence_0, sentence_1, and label
• Approximate statistics based on the first 1000 samples:

  	sentence_0	sentence_1	label
  type	string	string	float
  details	min: 18 characters mean: 60.22 characters max: 246 characters	min: 39 characters mean: 1582.31 characters max: 2325 characters	min: 0.0 mean: 0.18 max: 1.0

• Samples:

  sentence_0	sentence_1	label
  Besides the album Fight Like a Girl, are there any other interesting aspects about this article?	Fight Like a Girl (Bomshel song) "Fight Like a Girl" is a song recorded by American country music duo Bomshel. It was released in February 2009 as the second single and title track from their debut album "Fight Like a Girl", which was released on October 20, 2009 via Curb Records. Co-written by the duo's two members, Kristy Osmunson and Kelley Shepard, along with Bob Regan. The song received mixed reviews from critics who found the lyrical content cliché and indifferent to other similar songs. " Fight Like a Girl" peaked at number 30 on the "Billboard" Hot Country Songs chart. The accompanying music video for the song was directed by Eric Welch and features Bombshel performing with their guitars in front of different backgrounds. The song is a mid-tempo mostly accompanied by electric guitar. The song's lyrics tell of a young girl who is facing problems in her life, such as being teased on the playground at school. Her mother gives her advice by telling her to "fight like a girl." Leean...	0.0
  Did Ricky Martin try again for Menudo?	Charlie Masso Carlos Javier Rivera Masso (born June 13, 1969 in San Juan, Puerto Rico) is a Puerto Rican singer and actor. Masso is a former member of Menudo, joining the band at the height of Menudo's worldwide fame. He replaced Rene Farrait and quickly became a fan favorite with his participation on the albums of "Por Amor" and "Una Aventura Llamada Menudo", which was the soundtrack of Menudo's second movie. Charlie recorded 14 albums with Menudo, two were nominated for a Grammy Award for Best Latin Pop Album in 1984 and 85. Masso began his life as a Menudo the very moment that one of Menudo's members of the group's "golden era," Farrait, left the group. However, Masso's presence helped extend the golden era an additional three years, and when he left the group in 1987, he was the last member of the group's golden years to leave. Masso was a member of Menudo during their most commercially successful period in the States, which included the group's highest-charting U.S. single, "Hold ...	0.0
  Other than family, are there any other interesting aspects about the article on Vincent Price?	Vincent Price filmography This is the filmography of Vincent Price (May 27, 1911 – October 25, 1993), which includes appearances in theatre and television. Price made his theatre debut in the Gate Theatre's production of "Chicago" (1935), followed by work on Broadway. Under contract to Universal, Price traveled to Hollywood, making his screen debut in "Service de Luxe" (1938). By the 1960s, Vincent Price was working almost exclusively in the horror genre and teen film genres. Price's final film was "Edward Scissorhands" (1990) and the TV movie "The Heart of Justice" (1992) was his last screen appearance. Price died in Los Angeles, California on October 25, 1993.	0.0

• Loss: BinaryCrossEntropyLoss with these parameters:

  {
      "activation_fn": "torch.nn.modules.linear.Identity",
      "pos_weight": null
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• per_device_train_batch_size: 64
• per_device_eval_batch_size: 64

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: steps
• prediction_loss_only: True
• per_device_train_batch_size: 64
• per_device_eval_batch_size: 64
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 5e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1
• num_train_epochs: 3
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.0
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: False
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• hub_revision: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• liger_kernel_config: None
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

Epoch	Step	Training Loss	dev_ndcg@10
0.2030	500	0.4906	-
0.2497	615	-	0.9657
0.4060	1000	0.1561	-
0.4994	1230	-	0.9874
0.6090	1500	0.1198	-
0.7491	1845	-	0.9917
0.8120	2000	0.1023	-
0.9988	2460	-	0.9899
1.0	2463	-	0.9899
1.0150	2500	0.0871	-
1.2180	3000	0.0725	-
1.2485	3075	-	0.9917
1.4210	3500	0.0727	-
1.4982	3690	-	0.9935
1.6240	4000	0.0679	-
1.7479	4305	-	0.9935
1.8270	4500	0.0635	-
1.9976	4920	-	0.9954
2.0	4926	-	0.9954
2.0300	5000	0.0619	-
2.2330	5500	0.0513	-
2.2473	5535	-	0.9954
2.4361	6000	0.0509	-
2.4970	6150	-	0.9972

Framework Versions

• Python: 3.12.13
• Sentence Transformers: 5.1.0
• Transformers: 4.56.1
• PyTorch: 2.6.0+cu124
• Accelerate: 1.10.1
• Datasets: 4.0.0
• Tokenizers: 0.22.2

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}