--- base_model: sentence-transformers/all-mpnet-base-v2 library_name: sentence-transformers metrics: - cosine_accuracy - dot_accuracy - manhattan_accuracy - euclidean_accuracy - max_accuracy pipeline_tag: sentence-similarity tags: - sentence-transformers - sentence-similarity - feature-extraction - generated_from_trainer - dataset_size:6462 - loss:MultipleNegativesRankingLoss widget: - source_sentence: gain successful RDP authentication sentences: - Creates or Schedules a task. - Execute processes on other systems complete with full interactivity for console applications without having to manually install client software. - allows users to execute commands remotely on target systems using various methods including WMI, SMB, SSH, RDP, and PowerShell - source_sentence: collect and stage the informaiton in AD sentences: - Displays the directory structure of a path or of the disk in a drive graphically. - Get user name and group information along with the respective security identifiers (SID) claims privileges logon identifier (logon ID) for the current user on the local system. - retrieve stored passwords from various software and operating systems - source_sentence: Download files or binary for further usage sentences: - allows users to extract sensitive credential information from the Local Security Authority (LSA) on Windows systems. - Transfer data from or to a server using URLs. - Displays and modifies entries in the Address Resolution Protocol (ARP) cache. - source_sentence: collect and stage the informaiton in AD sentences: - Adds displays or modifies global groups in domains. - Gets the local security groups. - Displays the directory structure of a path or of the disk in a drive graphically. - source_sentence: Modify Registry of Current User Profile sentences: - Stops one or more running services. - Allows users to manage local and domain user accounts. - Saves a copy of specified subkeys, entries, and values of the registry in a specified file. model-index: - name: SentenceTransformer based on sentence-transformers/all-mpnet-base-v2 results: - task: type: triplet name: Triplet dataset: name: dev type: dev metrics: - type: cosine_accuracy value: 1.0 name: Cosine Accuracy - type: dot_accuracy value: 0.0 name: Dot Accuracy - type: manhattan_accuracy value: 1.0 name: Manhattan Accuracy - type: euclidean_accuracy value: 1.0 name: Euclidean Accuracy - type: max_accuracy value: 1.0 name: Max Accuracy - task: type: triplet name: Triplet dataset: name: test type: test metrics: - type: cosine_accuracy value: 1.0 name: Cosine Accuracy - type: dot_accuracy value: 0.0 name: Dot Accuracy - type: manhattan_accuracy value: 1.0 name: Manhattan Accuracy - type: euclidean_accuracy value: 1.0 name: Euclidean Accuracy - type: max_accuracy value: 1.0 name: Max Accuracy --- # SentenceTransformer based on sentence-transformers/all-mpnet-base-v2 This is a [sentence-transformers](https://www.SBERT.net) model finetuned from [sentence-transformers/all-mpnet-base-v2](https://huggingface.co/sentence-transformers/all-mpnet-base-v2). It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more. ## Model Details ### Model Description - **Model Type:** Sentence Transformer - **Base model:** [sentence-transformers/all-mpnet-base-v2](https://huggingface.co/sentence-transformers/all-mpnet-base-v2) - **Maximum Sequence Length:** 384 tokens - **Output Dimensionality:** 768 tokens - **Similarity Function:** Cosine Similarity ### Model Sources - **Documentation:** [Sentence Transformers Documentation](https://sbert.net) - **Repository:** [Sentence Transformers on GitHub](https://github.com/UKPLab/sentence-transformers) - **Hugging Face:** [Sentence Transformers on Hugging Face](https://huggingface.co/models?library=sentence-transformers) ### Full Model Architecture ``` SentenceTransformer( (0): Transformer({'max_seq_length': 384, 'do_lower_case': False}) with Transformer model: MPNetModel (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True}) (2): Normalize() ) ``` ## Usage ### Direct Usage (Sentence Transformers) First install the Sentence Transformers library: ```bash pip install -U sentence-transformers ``` Then you can load this model and run inference. ```python from sentence_transformers import SentenceTransformer # Download from the 🤗 Hub model = SentenceTransformer("brilan/procedure-tool-matching_3_epochs") # Run inference sentences = [ 'Modify Registry of Current User Profile', 'Saves a copy of specified subkeys, entries, and values of the registry in a specified file.', 'Stops one or more running services.', ] embeddings = model.encode(sentences) print(embeddings.shape) # [3, 768] # Get the similarity scores for the embeddings similarities = model.similarity(embeddings, embeddings) print(similarities.shape) # [3, 3] ``` ## Evaluation ### Metrics #### Triplet * Dataset: `dev` * Evaluated with [TripletEvaluator](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.TripletEvaluator) | Metric | Value | |:--------------------|:--------| | **cosine_accuracy** | **1.0** | | dot_accuracy | 0.0 | | manhattan_accuracy | 1.0 | | euclidean_accuracy | 1.0 | | max_accuracy | 1.0 | #### Triplet * Dataset: `test` * Evaluated with [TripletEvaluator](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.TripletEvaluator) | Metric | Value | |:--------------------|:--------| | **cosine_accuracy** | **1.0** | | dot_accuracy | 0.0 | | manhattan_accuracy | 1.0 | | euclidean_accuracy | 1.0 | | max_accuracy | 1.0 | ## Training Details ### Training Dataset #### Unnamed Dataset * Size: 6,462 training samples * Columns: anchor, positive, and negative * Approximate statistics based on the first 1000 samples: | | anchor | positive | negative | |:--------|:---------------------------------------------------------------------------------|:----------------------------------------------------------------------------------|:----------------------------------------------------------------------------------| | type | string | string | string | | details | | | | * Samples: | anchor | positive | negative | |:---------------------------------------------------------------------------------------|:--------------------------------------------------------------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | used compromised domain accounts to gain access to the target environment | allows users to execute commands remotely on target systems using various methods including WMI, SMB, SSH, RDP, and PowerShell | Displays information about user sessions on a Remote Desktop Session Host server. | | use default credentials to connect to IPC$ shares on remote machines | Execute commands on remote targets via Remote Desktop Protocol (RDP) without requiring a graphical user interface (GUI). | It provides functionality to view create modify and delete user accounts directly from the command prompt. | | gain access to the server via SSH | allow users to connect to RDP servers | allows administrators to manage and configure audit policies for the system and provides the ability to view, set, and modify the audit policies that control what events are logged by the Windows security auditing subsystem. | * Loss: [MultipleNegativesRankingLoss](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#multiplenegativesrankingloss) with these parameters: ```json { "scale": 20.0, "similarity_fct": "cos_sim" } ``` ### Evaluation Dataset #### Unnamed Dataset * Size: 2,770 evaluation samples * Columns: anchor, positive, and negative * Approximate statistics based on the first 1000 samples: | | anchor | positive | negative | |:--------|:---------------------------------------------------------------------------------|:----------------------------------------------------------------------------------|:----------------------------------------------------------------------------------| | type | string | string | string | | details | | | | * Samples: | anchor | positive | negative | |:-------------------------------------------------------------------|:--------------------------------------------------------------------------------------------------------------------------------------------------------------|:--------------------------------------------------------------------------------------------------------| | Disable Windows Services related to security products | stop running service | Creates lists and deletes stored user names and passwords or credentials. | | Get user information | Gets the local security groups. | Copy files from source to dest between local and remote machine skipping identical files. | | used pass the hash for lateral movement | Execute processes on other systems complete with full interactivity for console applications without having to manually install client software. | Extracts passwords keys,pin,codes,tickets from the memory of lsass | * Loss: [MultipleNegativesRankingLoss](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#multiplenegativesrankingloss) with these parameters: ```json { "scale": 20.0, "similarity_fct": "cos_sim" } ``` ### Training Hyperparameters #### Non-Default Hyperparameters - `eval_strategy`: steps - `per_device_train_batch_size`: 16 - `per_device_eval_batch_size`: 16 - `warmup_ratio`: 0.1 - `fp16`: True - `batch_sampler`: no_duplicates #### All Hyperparameters
Click to expand - `overwrite_output_dir`: False - `do_predict`: False - `eval_strategy`: steps - `prediction_loss_only`: True - `per_device_train_batch_size`: 16 - `per_device_eval_batch_size`: 16 - `per_gpu_train_batch_size`: None - `per_gpu_eval_batch_size`: None - `gradient_accumulation_steps`: 1 - `eval_accumulation_steps`: None - `torch_empty_cache_steps`: None - `learning_rate`: 5e-05 - `weight_decay`: 0.0 - `adam_beta1`: 0.9 - `adam_beta2`: 0.999 - `adam_epsilon`: 1e-08 - `max_grad_norm`: 1.0 - `num_train_epochs`: 3 - `max_steps`: -1 - `lr_scheduler_type`: linear - `lr_scheduler_kwargs`: {} - `warmup_ratio`: 0.1 - `warmup_steps`: 0 - `log_level`: passive - `log_level_replica`: warning - `log_on_each_node`: True - `logging_nan_inf_filter`: True - `save_safetensors`: True - `save_on_each_node`: False - `save_only_model`: False - `restore_callback_states_from_checkpoint`: False - `no_cuda`: False - `use_cpu`: False - `use_mps_device`: False - `seed`: 42 - `data_seed`: None - `jit_mode_eval`: False - `use_ipex`: False - `bf16`: False - `fp16`: True - `fp16_opt_level`: O1 - `half_precision_backend`: auto - `bf16_full_eval`: False - `fp16_full_eval`: False - `tf32`: None - `local_rank`: 0 - `ddp_backend`: None - `tpu_num_cores`: None - `tpu_metrics_debug`: False - `debug`: [] - `dataloader_drop_last`: False - `dataloader_num_workers`: 0 - `dataloader_prefetch_factor`: None - `past_index`: -1 - `disable_tqdm`: False - `remove_unused_columns`: True - `label_names`: None - `load_best_model_at_end`: False - `ignore_data_skip`: False - `fsdp`: [] - `fsdp_min_num_params`: 0 - `fsdp_config`: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False} - `fsdp_transformer_layer_cls_to_wrap`: None - `accelerator_config`: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None} - `deepspeed`: None - `label_smoothing_factor`: 0.0 - `optim`: adamw_torch - `optim_args`: None - `adafactor`: False - `group_by_length`: False - `length_column_name`: length - `ddp_find_unused_parameters`: None - `ddp_bucket_cap_mb`: None - `ddp_broadcast_buffers`: False - `dataloader_pin_memory`: True - `dataloader_persistent_workers`: False - `skip_memory_metrics`: True - `use_legacy_prediction_loop`: False - `push_to_hub`: False - `resume_from_checkpoint`: None - `hub_model_id`: None - `hub_strategy`: every_save - `hub_private_repo`: False - `hub_always_push`: False - `gradient_checkpointing`: False - `gradient_checkpointing_kwargs`: None - `include_inputs_for_metrics`: False - `eval_do_concat_batches`: True - `fp16_backend`: auto - `push_to_hub_model_id`: None - `push_to_hub_organization`: None - `mp_parameters`: - `auto_find_batch_size`: False - `full_determinism`: False - `torchdynamo`: None - `ray_scope`: last - `ddp_timeout`: 1800 - `torch_compile`: False - `torch_compile_backend`: None - `torch_compile_mode`: None - `dispatch_batches`: None - `split_batches`: None - `include_tokens_per_second`: False - `include_num_input_tokens_seen`: False - `neftune_noise_alpha`: None - `optim_target_modules`: None - `batch_eval_metrics`: False - `eval_on_start`: False - `eval_use_gather_object`: False - `batch_sampler`: no_duplicates - `multi_dataset_batch_sampler`: proportional
### Training Logs | Epoch | Step | Training Loss | loss | dev_cosine_accuracy | test_cosine_accuracy | |:------:|:----:|:-------------:|:------:|:-------------------:|:--------------------:| | 0 | 0 | - | - | 0.8596 | - | | 0.2475 | 100 | 2.0428 | 1.3753 | 0.9989 | - | | 0.4950 | 200 | 1.5299 | 1.2361 | 1.0 | - | | 0.7426 | 300 | 1.4871 | 1.1853 | 1.0 | - | | 0.9901 | 400 | 1.4612 | 1.1707 | 1.0 | - | | 1.2376 | 500 | 0.0287 | 1.2190 | 1.0 | - | | 1.1584 | 600 | 0.9192 | 1.1738 | 1.0 | - | | 1.4059 | 700 | 1.4131 | 1.1708 | 1.0 | - | | 1.6535 | 800 | 1.4254 | 1.1428 | 1.0 | - | | 1.9010 | 900 | 1.3977 | 1.1373 | 1.0 | - | | 2.1485 | 1000 | 0.5379 | 1.1419 | 1.0 | - | | 2.0693 | 1100 | 0.386 | 1.1306 | 1.0 | - | | 2.3168 | 1200 | 1.3708 | 1.1260 | 1.0 | - | | 2.3465 | 1212 | - | - | - | 1.0 | ### Framework Versions - Python: 3.11.5 - Sentence Transformers: 3.1.0 - Transformers: 4.44.2 - PyTorch: 2.4.1+cu121 - Accelerate: 1.0.0 - Datasets: 3.0.1 - Tokenizers: 0.19.1 ## Citation ### BibTeX #### Sentence Transformers ```bibtex @inproceedings{reimers-2019-sentence-bert, title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks", author = "Reimers, Nils and Gurevych, Iryna", booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing", month = "11", year = "2019", publisher = "Association for Computational Linguistics", url = "https://arxiv.org/abs/1908.10084", } ``` #### MultipleNegativesRankingLoss ```bibtex @misc{henderson2017efficient, title={Efficient Natural Language Response Suggestion for Smart Reply}, author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil}, year={2017}, eprint={1705.00652}, archivePrefix={arXiv}, primaryClass={cs.CL} } ```