--- language: [] library_name: sentence-transformers tags: - sentence-transformers - sentence-similarity - feature-extraction - generated_from_trainer - dataset_size:557850 - loss:MatryoshkaLoss - loss:MultipleNegativesRankingLoss - mteb model-index: - name: AbderrahmanSkiredj1/arabic_text_embedding_sts_arabertv02_arabicnlitriplet results: - dataset: config: ar-ar name: MTEB STS17 (ar-ar) revision: faeb762787bd10488a50c8b5be4a3b82e411949c split: test type: mteb/sts17-crosslingual-sts metrics: - type: cosine_pearson value: 84.4552565985224 - type: cosine_spearman value: 84.95895784502326 - type: euclidean_pearson value: 82.78929391924628 - type: euclidean_spearman value: 84.30172209203968 - type: main_score value: 84.95895784502326 - type: manhattan_pearson value: 82.8787314198733 - type: manhattan_spearman value: 84.31971710672602 task: type: STS datasets: [] widget: - source_sentence: رجل قائم على قدم واحدة بالقرب من شاطئ نظيف sentences: - رجل يقدم عرضاً - هناك رجل بالخارج قرب الشاطئ - رجل يجلس على أريكه - source_sentence: الرجل يرتدي قميصاً أزرق. sentences: - رجل يرتدي قميصاً أزرق بجانب الطريق مع شاحنة زرقاء وسيارة حمراء مع الماء في الخلفية. - كتاب جواهر المعاني - رجل يرتدي قميص أسود يعزف على الجيتار. - source_sentence: يجلس شاب ذو شعر أشقر على الحائط يقرأ جريدة بينما تمر امرأة وفتاة شابة. sentences: - ذكر شاب ينظر إلى جريدة بينما تمر إمرأتان بجانبه - رجل يستلقي على وجهه على مقعد في الحديقة. - الشاب نائم بينما الأم تقود ابنتها إلى الحديقة pipeline_tag: sentence-similarity --- # SentenceTransformer This is a [sentence-transformers](https://www.SBERT.net) model trained on the Omartificial-Intelligence-Space/arabic-n_li-triplet dataset. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more. ## Model Details ### Model Description - **Model Type:** Sentence Transformer - **Maximum Sequence Length:** 512 tokens - **Output Dimensionality:** 768 tokens - **Similarity Function:** Cosine Similarity - **Training Dataset:** - Omartificial-Intelligence-Space/arabic-n_li-triplet ### Model Sources - **Documentation:** [Sentence Transformers Documentation](https://sbert.net) - **Repository:** [Sentence Transformers on GitHub](https://github.com/UKPLab/sentence-transformers) - **Hugging Face:** [Sentence Transformers on Hugging Face](https://huggingface.co/models?library=sentence-transformers) ### Full Model Architecture ``` SentenceTransformer( (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: BertModel (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True}) ) ``` ## Usage ### Direct Usage (Sentence Transformers) First install the Sentence Transformers library: ```bash pip install -U sentence-transformers ``` Then you can load this model and run inference. ```python from sentence_transformers import SentenceTransformer # Download from the 🤗 Hub model = SentenceTransformer("AbderrahmanSkiredj1/arabic_text_embedding_sts_arabertv02_arabicnlitriplet") # Run inference sentences = [ 'يجلس شاب ذو شعر أشقر على الحائط يقرأ جريدة بينما تمر امرأة وفتاة شابة.', 'ذكر شاب ينظر إلى جريدة بينما تمر إمرأتان بجانبه', 'الشاب نائم بينما الأم تقود ابنتها إلى الحديقة', ] embeddings = model.encode(sentences) print(embeddings.shape) # [3, 768] # Get the similarity scores for the embeddings similarities = model.similarity(embeddings, embeddings) print(similarities.shape) # [3, 3] ``` ## Training Details ### Training Dataset #### Omartificial-Intelligence-Space/arabic-n_li-triplet * Dataset: Omartificial-Intelligence-Space/arabic-n_li-triplet * Size: 557,850 training samples * Columns: anchor, positive, and negative * Approximate statistics based on the first 1000 samples: | | anchor | positive | negative | |:--------|:---------------------------------------------------------------------------------|:----------------------------------------------------------------------------------|:----------------------------------------------------------------------------------| | type | string | string | string | | details | | | | * Samples: | anchor | positive | negative | |:------------------------------------------------------------|:--------------------------------------------|:------------------------------------| | شخص على حصان يقفز فوق طائرة معطلة | شخص في الهواء الطلق، على حصان. | شخص في مطعم، يطلب عجة. | | أطفال يبتسمون و يلوحون للكاميرا | هناك أطفال حاضرون | الاطفال يتجهمون | | صبي يقفز على لوح التزلج في منتصف الجسر الأحمر. | الفتى يقوم بخدعة التزلج | الصبي يتزلج على الرصيف | * Loss: [MatryoshkaLoss](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#matryoshkaloss) with these parameters: ```json { "loss": "MultipleNegativesRankingLoss", "matryoshka_dims": [ 768, 512, 256, 128, 64 ], "matryoshka_weights": [ 1, 1, 1, 1, 1 ], "n_dims_per_step": -1 } ``` ### Evaluation Dataset #### Omartificial-Intelligence-Space/arabic-n_li-triplet * Dataset: Omartificial-Intelligence-Space/arabic-n_li-triplet * Size: 6,584 evaluation samples * Columns: anchor, positive, and negative * Approximate statistics based on the first 1000 samples: | | anchor | positive | negative | |:--------|:----------------------------------------------------------------------------------|:---------------------------------------------------------------------------------|:---------------------------------------------------------------------------------| | type | string | string | string | | details | | | | * Samples: | anchor | positive | negative | |:-----------------------------------------------------------------------------------------------------------------------------------------------------|:-------------------------------------------------------|:---------------------------------------------------| | امرأتان يتعانقان بينما يحملان حزمة | إمرأتان يحملان حزمة | الرجال يتشاجرون خارج مطعم | | طفلين صغيرين يرتديان قميصاً أزرق، أحدهما يرتدي الرقم 9 والآخر يرتدي الرقم 2 يقفان على خطوات خشبية في الحمام ويغسلان أيديهما في المغسلة. | طفلين يرتديان قميصاً مرقماً يغسلون أيديهم | طفلين يرتديان سترة يذهبان إلى المدرسة | | رجل يبيع الدونات لعميل خلال معرض عالمي أقيم في مدينة أنجليس | رجل يبيع الدونات لعميل | امرأة تشرب قهوتها في مقهى صغير | * Loss: [MatryoshkaLoss](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#matryoshkaloss) with these parameters: ```json { "loss": "MultipleNegativesRankingLoss", "matryoshka_dims": [ 768, 512, 256, 128, 64 ], "matryoshka_weights": [ 1, 1, 1, 1, 1 ], "n_dims_per_step": -1 } ``` ### Training Hyperparameters #### Non-Default Hyperparameters - `per_device_train_batch_size`: 64 - `per_device_eval_batch_size`: 64 - `learning_rate`: 1e-06 - `num_train_epochs`: 10 - `warmup_ratio`: 0.1 - `fp16`: True - `batch_sampler`: no_duplicates #### All Hyperparameters
Click to expand - `overwrite_output_dir`: False - `do_predict`: False - `prediction_loss_only`: True - `per_device_train_batch_size`: 64 - `per_device_eval_batch_size`: 64 - `per_gpu_train_batch_size`: None - `per_gpu_eval_batch_size`: None - `gradient_accumulation_steps`: 1 - `eval_accumulation_steps`: None - `learning_rate`: 1e-06 - `weight_decay`: 0 - `adam_beta1`: 0.9 - `adam_beta2`: 0.999 - `adam_epsilon`: 1e-08 - `max_grad_norm`: 1.0 - `num_train_epochs`: 10 - `max_steps`: -1 - `lr_scheduler_type`: linear - `lr_scheduler_kwargs`: {} - `warmup_ratio`: 0.1 - `warmup_steps`: 0 - `log_level`: passive - `log_level_replica`: warning - `log_on_each_node`: True - `logging_nan_inf_filter`: True - `save_safetensors`: True - `save_on_each_node`: False - `save_only_model`: False - `no_cuda`: False - `use_cpu`: False - `use_mps_device`: False - `seed`: 42 - `data_seed`: None - `jit_mode_eval`: False - `use_ipex`: False - `bf16`: False - `fp16`: True - `fp16_opt_level`: O1 - `half_precision_backend`: auto - `bf16_full_eval`: False - `fp16_full_eval`: False - `tf32`: None - `local_rank`: 0 - `ddp_backend`: None - `tpu_num_cores`: None - `tpu_metrics_debug`: False - `debug`: [] - `dataloader_drop_last`: False - `dataloader_num_workers`: 0 - `dataloader_prefetch_factor`: None - `past_index`: -1 - `disable_tqdm`: False - `remove_unused_columns`: True - `label_names`: None - `load_best_model_at_end`: False - `ignore_data_skip`: False - `fsdp`: [] - `fsdp_min_num_params`: 0 - `fsdp_config`: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False} - `fsdp_transformer_layer_cls_to_wrap`: None - `accelerator_config`: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True} - `deepspeed`: None - `label_smoothing_factor`: 0.0 - `optim`: adamw_torch - `optim_args`: None - `adafactor`: False - `group_by_length`: False - `length_column_name`: length - `ddp_find_unused_parameters`: None - `ddp_bucket_cap_mb`: None - `ddp_broadcast_buffers`: False - `dataloader_pin_memory`: True - `dataloader_persistent_workers`: False - `skip_memory_metrics`: True - `use_legacy_prediction_loop`: False - `push_to_hub`: False - `resume_from_checkpoint`: None - `hub_model_id`: None - `hub_strategy`: every_save - `hub_private_repo`: False - `hub_always_push`: False - `gradient_checkpointing`: False - `gradient_checkpointing_kwargs`: None - `include_inputs_for_metrics`: False - `fp16_backend`: auto - `push_to_hub_model_id`: None - `push_to_hub_organization`: None - `mp_parameters`: - `auto_find_batch_size`: False - `full_determinism`: False - `torchdynamo`: None - `ray_scope`: last - `ddp_timeout`: 1800 - `torch_compile`: False - `torch_compile_backend`: None - `torch_compile_mode`: None - `dispatch_batches`: None - `split_batches`: None - `include_tokens_per_second`: False - `include_num_input_tokens_seen`: False - `neftune_noise_alpha`: None - `optim_target_modules`: None - `batch_sampler`: no_duplicates - `multi_dataset_batch_sampler`: proportional
### Training Logs | Epoch | Step | Training Loss | |:------:|:----:|:-------------:| | 0.0183 | 40 | 10.8122 | | 0.0367 | 80 | 10.8841 | | 0.0550 | 120 | 10.0068 | | 0.0734 | 160 | 9.925 | | 0.0917 | 200 | 9.2169 | | 0.1101 | 240 | 8.8431 | | 0.1284 | 280 | 8.6185 | | 0.1468 | 320 | 8.8141 | | 0.1651 | 360 | 8.0517 | | 0.1835 | 400 | 7.9315 | | 0.2018 | 440 | 7.7502 | | 0.2202 | 480 | 7.432 | | 0.2385 | 520 | 7.4885 | | 0.2569 | 560 | 6.8975 | | 0.2752 | 600 | 6.9895 | | 0.2936 | 640 | 6.5074 | | 0.3119 | 680 | 6.3892 | | 0.3303 | 720 | 6.6392 | | 0.3486 | 760 | 6.235 | | 0.3670 | 800 | 5.95 | | 0.3853 | 840 | 5.7161 | | 0.4037 | 880 | 5.6532 | | 0.4220 | 920 | 5.5386 | | 0.4404 | 960 | 5.7382 | | 0.4587 | 1000 | 5.3164 | | 0.4771 | 1040 | 5.1571 | | 0.4954 | 1080 | 4.9178 | | 0.5138 | 1120 | 5.0309 | | 0.5321 | 1160 | 5.0867 | | 0.5505 | 1200 | 5.1403 | ### Framework Versions - Python: 3.10.14 - Sentence Transformers: 3.0.1 - Transformers: 4.39.3 - PyTorch: 2.2.2+cu121 - Accelerate: 0.29.1 - Datasets: 2.18.0 - Tokenizers: 0.15.2 ## Citation ### BibTeX #### Sentence Transformers ```bibtex @inproceedings{reimers-2019-sentence-bert, title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks", author = "Reimers, Nils and Gurevych, Iryna", booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing", month = "11", year = "2019", publisher = "Association for Computational Linguistics", url = "https://arxiv.org/abs/1908.10084", } ``` #### MatryoshkaLoss ```bibtex @misc{kusupati2024matryoshka, title={Matryoshka Representation Learning}, author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi}, year={2024}, eprint={2205.13147}, archivePrefix={arXiv}, primaryClass={cs.LG} } ``` #### MultipleNegativesRankingLoss ```bibtex @misc{henderson2017efficient, title={Efficient Natural Language Response Suggestion for Smart Reply}, author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil}, year={2017}, eprint={1705.00652}, archivePrefix={arXiv}, primaryClass={cs.CL} } ```