ahmedHamdi/IR-fr-en-gemma_masked

SentenceTransformer based on google/embeddinggemma-300m

This is a sentence-transformers model finetuned from google/embeddinggemma-300m. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: google/embeddinggemma-300m
• Maximum Sequence Length: 256 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Cosine Similarity

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 256, 'do_lower_case': False, 'architecture': 'Gemma3TextModel'})
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("ahmedHamdi/IR-fr-en-gemma_masked")
# Run inference
sentences = [
    "Tuscany, 1984. Mario and GPE find themselves stuck at a level crossing, and the two friends confide in each other about their lives. GPE, among other things, is worried about his sister PERSON, who has fallen into depression because of her relationship with an American. As the wait drags on, the two men decide to take a detour and end up stranded in the middle of the countryside. They take refuge in an inn for the night and wake up the next morning in a Tuscan village called PERSON, sometime between 1400 and 1500. Initially believing it to be a big joke, the two men find themselves having to confront the reality of the era. They find shelter with PERSON, the brother of the PERSON's guest who died the day before. They begin working at the ORG owned by PERSON's mother, PERSON, and while GPE adapts immediately to his new lifestyle, PERSON refuses to adjust. Soon, PERSON, the daughter of a wealthy family, with whom he begins a relationship. Meanwhile, PERSON is arrested, and GPE tries to write a letter to ORG to secure his release. GPE makes no secret of his jealousy of ORG relationship and convinces his friend to set off for GPE to join PERSON.\nDuring their journey to GPE, they encounter a beautiful ORG, PERSON, who frightens them by shooting an arrow at their chariot. The story then divides into two parts, depending on whether it is presented in the television or film version.",
    'Tuscan countryside. The janitor PERSON and the teacher GPE are stopped at a level crossing, waiting for the train to pass. The two are friends and trust each other. Among other things, GPE is worried about his sister PERSON, who fell into depression for the failure of her relationship with an American boy. The wait continues and decide to travel a road in the fields. After a while they remain in cream with the car in the middle of the countryside. You do evening, it rains. The two find accommodation in an inn for the night, in a room that already hosts another person. The next morning as soon as they wake up they see, having fun, the guest urinating from the window, but their laughter are immediately truncated by the hiss of a spear that kills him. Mario and GPE see fonding people in the black cloak on horseback, rush to the ground floor and find other people, dressed in a very strange way. Increducts make themselves be said by a man where they find themselves and discover that they are fried, an imaginary Tuscan village, in 1492. considering it first a terrible joke, they must resign themselves to the harsh reality making themselves hosting from PERSON, the brother of the killed man, ORG, who tells them of a terrible feud with such a ORG, who is exterminating his family. Once in the village they know PERSON, a mother of PERSON and the late ORG and begin to work in their slaughter shop. In the ORG context of the village, the most disparate episodes happen. GPE immediately seems comfortable, while PERSON does not want to settle down; Soon, however, during a religious function, PERSON makes the acquaintance of Pia, a girl of a rich family, with whom he begins to see himself looking out from the surrounding wall of her house. In the meantime, PERSON is arrested and GPE and PERSON write a letter to ORG in vain to obtain his liberation. GPE (who continues to tell PERSON to ask PERSON if he has an friend to introduce him) does not hide a certain annoyance for the meetings between PERSON and the young PERSON and complains that he always remains and only him to work in the butcher shop. Driven by his political-intellectual ardor, he convinces his friend to travel to GPE, in order to reach PERSON and dissuade him from leaving for the ORG and discovering GPE, so that, in the future, his sister cannot meet the American boy who left her. In an unspecified place the two come across a beautiful ORG, PERSON, which intimidates them by throwing an arrow against their wagon. At this point the story differs depending on the version, standard or extended.',
    "World-famous symphony conductor Sir PERSON returns from a visit to his native GPE and discovers that his rich brother-in-law, August Henshler, having misunderstood PERSON's casual instruction to watch over his much younger wife PERSON while he was away, hired a private detective named ORG to follow her. PERSON is livid, and when August hands him the detective's report he tears it to pieces. Despite himself, the idea of infidelity on ORG's part hounds PERSON's mind, and he acquires ORG's address from August, ostensibly so he can destroy any copies of the report which ORG may have kept on file. ORG summarizes the report for PERSON: his wife was spied late at night wearing only a nightgown as she went to the hotel room of PERSON's secretary, PERSON, a man closer in age to her own, where she stayed for thirty-eight minutes. Distressed by the news, PERSON quarrels with PERSON before proceeding to his concert, where he conducts three pieces of classical music, envisioning scenarios appropriate to each one: a perfect crime scenario in which he murders his wife with a straight razor and frames PERSON (to the ORG to PERSON's Semiramide), accepting the situation and giving PERSON a generous check and his blessing (to the Prelude to ORG's Tannhäuser), and a game of Russian roulette with a blubbering PERSON and PERSON, that ends in PERSON's death with the first trigger pull (to GPE's Francesca da Rimini). After the concert, PERSON tries to stage his murder fantasy, but makes a mess of their apartment while looking for the recording device which he fantasized using to lure PERSON to the crime scene, then accidentally records at the wrong speed. Daphne returns home and questions her husband about his ill temperament, but devotedly bandages his thumb when he cuts it testing the sharpness of his razor. He next tries the forgiveness fantasy, but spills ink all over his checkbook. Russian roulette also fizzles out when he cannot remember where he left the bullets. Without revealing his suspicions, he asks if she has ever been to PERSON's room. She admits it, saying she was searching for her sister, PERSON, August's wife, suspecting she was having an affair with PERSON; she became trapped there when she saw ORG spying on the room, fearful that if he saw her walk out she would be suspected of having an affair. Alfred begs PERSON's forgiveness for his irrational behavior, which she gladly gives, ascribing it to the creative temperament of a great artist.",
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[ 1.0000,  0.5453, -0.0518],
#         [ 0.5453,  1.0000,  0.0562],
#         [-0.0518,  0.0562,  1.0000]])

Training Details

Training Dataset

Unnamed Dataset

• Size: 16,276 training samples
• Columns: sentence_0 and sentence_1
• Approximate statistics based on the first 1000 samples:

  	sentence_0	sentence_1
  type	string	string
  details	min: 8 tokens mean: 122.12 tokens max: 256 tokens	min: 13 tokens mean: 226.96 tokens max: 256 tokens

• Samples:

  sentence_0	sentence_1
  General Overview The story takes place in GPE, a fantasy world inhabited by humans and dragons. PERSON, a teenager searching for his identity, flees his castle and wanders the countryside after killing his father. He will meet LOC, the young ORG, and a fearsome sorcerer. Arren gradually discovers how to overcome his fears and find his own voice.	As a war galley sails through a storm, two dragons fight above the clouds, ending in one's death. In the kingdom of GPE, the royal wizard ORG proclaims the dragons to be a sign of a loss of balance in the world. The king is dealing with disease sweeping across his kingdom, and the disappearance of his son, PERSON. In the middle of the night, ORG kills his father, steals his father's sword, and flees the castle. Arren travels through the desert and is rescued from wolves by the archmage Sparrowhawk. Together they travel to the city of LOC. When ORG explores the town alone, he rescues a young girl named ORG from slavers, but is later captured by the same slave master, GPE. His sword is dumped in the sea. Sparrowhawk rescues Arren from the slave caravan and takes him to a farm run by ORG's oldest and greatest friend PERSON, who lives with ORG. ORG's intervention against ORG's slave caravan angers Lord Cob, a powerful warlock and the ruler of Hort Town, who wants the archmage brought to th...
  Inspired by the true story of ORG (PERSON), Fighter in the PERSON tells the story of how a Korean immigrant became a legend in GPE by creating a new martial art: Taekkyon.	At the tail-end of World War II, PERSON is a young Korean man who longs to be able to fly fighter planes. Stowing away to GPE in order to join their air force, Bae-dal's first experience of the country is when a con-man tries to steal his money. Bae-dal discovers that the man is a fellow Korean called ORG (PERSON), who has survived the harsh treatment of Koreans in GPE by turning to petty crime. With their different motives: Bae-dal driven by desire for action and Chun-bae needing to escape from some gangsters, the two Koreans stow away in a truck to the air force training camp. The commander in charge of the camp is a pompous imperialist called PERSON (PERSON). Having mistreated the two Koreans, he is amused by Bae-dal's fighting spirit and says that if Bae-dal can beat him with his inferior foreign fighting style (Taekkyon), he will release them. The two men fight with PERSON easily defeating Bae-dal, but an American attack on the airforce base allows Bae-dal and PERSON to escape. La...
  Ten-year-old PERSON is thrilled when his grandfather moves into his house. He's less than thrilled when he has to give up his bedroom and move into the attic. With the help of his friends, PERSON decides to declare war on his grandfather, who is much smarter than he looks...	After accidentally stealing from a grocery store due to having trouble with the self-checkouts and causing a scene with the store manager, recently widowed PERSON (PERSON) is visited by his daughter PERSON (PERSON) who wants him to move in with her family. PERSON does not want to leave his house because he built it himself. PERSON convinces PERSON to move in with her and gives him her son PERSON's (PERSON) bedroom. PERSON is not happy about giving his room to his grandfather and being moved to the attic. PERSON is welcomed by PERSON's husband PERSON (PERSON) and two daughters, PERSON (PERSON) and PERSON (Poppy Gagnon). During his first day, PERSON spends most of his time in his new room, sitting in his chair and looking at the sky while still thinking about his late wife. PERSON then tells his friends PERSON (PERSON), PERSON (PERSON) and PERSON (PERSON) about his grandfather moving in with his family and living in his room. After a miserable first night in his new room, PERSON decides ...

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "gather_across_devices": false
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 4
• per_device_eval_batch_size: 4
• multi_dataset_batch_sampler: round_robin

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: no
• prediction_loss_only: True
• per_device_train_batch_size: 4
• per_device_eval_batch_size: 4
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 5e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1
• num_train_epochs: 3
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: None
• warmup_ratio: 0.0
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• bf16: False
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch_fused
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• project: huggingface
• trackio_space_id: trackio
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• hub_revision: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: no
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• liger_kernel_config: None
• eval_use_gather_object: False
• average_tokens_across_devices: True
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: round_robin
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

Epoch	Step	Training Loss
0.1229	500	0.0396
0.2458	1000	0.0699
0.3686	1500	0.1463
0.4915	2000	0.1551
0.6144	2500	0.1705
0.7373	3000	0.1281
0.8602	3500	0.1261
0.9830	4000	0.1362
1.1059	4500	0.09
1.2288	5000	0.0709
1.3517	5500	0.1023
1.4746	6000	0.0672
1.5974	6500	0.0847
1.7203	7000	0.0734
1.8432	7500	0.0687
1.9661	8000	0.0602
2.0890	8500	0.0431
2.2118	9000	0.0298
2.3347	9500	0.0317
2.4576	10000	0.0218
2.5805	10500	0.0178
2.7034	11000	0.0211
2.8262	11500	0.0171
2.9491	12000	0.0227

Framework Versions

• Python: 3.9.18
• Sentence Transformers: 5.1.2
• Transformers: 4.57.6
• PyTorch: 2.8.0+cu128
• Accelerate: 1.10.1
• Datasets: 4.5.0
• Tokenizers: 0.22.2

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}