Hugging Face's logo

adriansanz
/
ST-tramits-SB-001-5ep

Sentence Similarity
sentence-transformers
Safetensors
xlm-roberta
feature-extraction
Generated from Trainer
dataset_size:2372
loss:MatryoshkaLoss
loss:MultipleNegativesRankingLoss
Eval Results
text-embeddings-inference
Inference Endpoints
Model card Files Community
Edit model card

SentenceTransformer based on BAAI/bge-m3

This is a sentence-transformers model finetuned from BAAI/bge-m3 on the json dataset. It maps sentences & paragraphs to a 1024-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

  • Model Type: Sentence Transformer
  • Base model: BAAI/bge-m3
  • Maximum Sequence Length: 8192 tokens
  • Output Dimensionality: 1024 tokens
  • Similarity Function: Cosine Similarity
  • Training Dataset:
    • json

Model Sources

Full Model Architecture 

SentenceTransformer(
  (0): Transformer({'max_seq_length': 8192, 'do_lower_case': False}) with Transformer model: XLMRobertaModel 
  (1): Pooling({'word_embedding_dimension': 1024, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("adriansanz/ST-tramits-SB-001-5ep")
# Run inference
sentences = [
    'Descripció. Retorna en format JSON adequat',
    "Quin és el contingut de l'annex específic?",
    "Què passa amb l'habitatge?",
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 1024]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Information Retrieval

Metric Value
cosine_accuracy@1 0.3322
cosine_accuracy@3 0.5902
cosine_accuracy@5 0.6998
cosine_accuracy@10 0.8094
cosine_precision@1 0.3322
cosine_precision@3 0.1967
cosine_precision@5 0.14
cosine_precision@10 0.0809
cosine_recall@1 0.3322
cosine_recall@3 0.5902
cosine_recall@5 0.6998
cosine_recall@10 0.8094
cosine_ndcg@10 0.5626
cosine_mrr@10 0.4843
cosine_map@100 0.4924

Information Retrieval

Metric Value
cosine_accuracy@1 0.3406
cosine_accuracy@3 0.5767
cosine_accuracy@5 0.6981
cosine_accuracy@10 0.8162
cosine_precision@1 0.3406
cosine_precision@3 0.1922
cosine_precision@5 0.1396
cosine_precision@10 0.0816
cosine_recall@1 0.3406
cosine_recall@3 0.5767
cosine_recall@5 0.6981
cosine_recall@10 0.8162
cosine_ndcg@10 0.5661
cosine_mrr@10 0.4872
cosine_map@100 0.4952

Information Retrieval

Metric Value
cosine_accuracy@1 0.3305
cosine_accuracy@3 0.5801
cosine_accuracy@5 0.6948
cosine_accuracy@10 0.8162
cosine_precision@1 0.3305
cosine_precision@3 0.1934
cosine_precision@5 0.139
cosine_precision@10 0.0816
cosine_recall@1 0.3305
cosine_recall@3 0.5801
cosine_recall@5 0.6948
cosine_recall@10 0.8162
cosine_ndcg@10 0.563
cosine_mrr@10 0.483
cosine_map@100 0.4908

Information Retrieval

Metric Value
cosine_accuracy@1 0.3288
cosine_accuracy@3 0.5885
cosine_accuracy@5 0.7015
cosine_accuracy@10 0.8094
cosine_precision@1 0.3288
cosine_precision@3 0.1962
cosine_precision@5 0.1403
cosine_precision@10 0.0809
cosine_recall@1 0.3288
cosine_recall@3 0.5885
cosine_recall@5 0.7015
cosine_recall@10 0.8094
cosine_ndcg@10 0.5626
cosine_mrr@10 0.4842
cosine_map@100 0.492

Information Retrieval

Metric Value
cosine_accuracy@1 0.3474
cosine_accuracy@3 0.5818
cosine_accuracy@5 0.6998
cosine_accuracy@10 0.8061
cosine_precision@1 0.3474
cosine_precision@3 0.1939
cosine_precision@5 0.14
cosine_precision@10 0.0806
cosine_recall@1 0.3474
cosine_recall@3 0.5818
cosine_recall@5 0.6998
cosine_recall@10 0.8061
cosine_ndcg@10 0.5654
cosine_mrr@10 0.4894
cosine_map@100 0.4973

Information Retrieval

Metric Value
cosine_accuracy@1 0.2917
cosine_accuracy@3 0.5683
cosine_accuracy@5 0.6644
cosine_accuracy@10 0.7875
cosine_precision@1 0.2917
cosine_precision@3 0.1894
cosine_precision@5 0.1329
cosine_precision@10 0.0788
cosine_recall@1 0.2917
cosine_recall@3 0.5683
cosine_recall@5 0.6644
cosine_recall@10 0.7875
cosine_ndcg@10 0.532
cosine_mrr@10 0.4512
cosine_map@100 0.4595

Training Details

Training Dataset

json

  • Dataset: json
  • Size: 2,372 training samples
  • Columns: positive and anchor
  • Approximate statistics based on the first 1000 samples:
    positive anchor
    type string string
    details
    • min: 3 tokens
    • mean: 35.12 tokens
    • max: 166 tokens
    • min: 8 tokens
    • mean: 19.49 tokens
    • max: 47 tokens
  • Samples:
    positive anchor
    Comunicar la variació d'alguna de les següents dades del Padró Municipal d'Habitants: Nom, Cognoms, Data de naixement, DNI, Passaport, Número de permís de residència (NIE), Sexe, Municipi i/o província de naixement, Nacionalitat, Titulació acadèmica. Quin és l'objectiu del canvi de dades personals en el Padró Municipal d'Habitants?
    EN QUÈ CONSISTEIX: Tramitar la sol·licitud de matrimoni civil a l'Ajuntament. Què és el matrimoni civil a l'Ajuntament de Sant Boi de Llobregat?
    En domiciliar el pagament de tributs municipals en entitats bancàries. Quin és el benefici de domiciliar el pagament de tributs?
  • Loss: MatryoshkaLoss with these parameters:
    {
    "loss": "MultipleNegativesRankingLoss",
    "matryoshka_dims": [
        1024,
        768,
        512,
        256,
        128,
        64
    ],
    "matryoshka_weights": [
        1,
        1,
        1,
        1,
        1,
        1
    ],
    "n_dims_per_step": -1
}

Training Hyperparameters

Non-Default Hyperparameters

  • eval_strategy: epoch
  • per_device_train_batch_size: 16
  • per_device_eval_batch_size: 16
  • gradient_accumulation_steps: 16
  • learning_rate: 2e-05
  • num_train_epochs: 5
  • lr_scheduler_type: cosine
  • warmup_ratio: 0.2
  • bf16: True
  • tf32: True
  • load_best_model_at_end: True
  • optim: adamw_torch_fused
  • batch_sampler: no_duplicates

All Hyperparameters

Click to expand
  • overwrite_output_dir: False
  • do_predict: False
  • eval_strategy: epoch
  • prediction_loss_only: True
  • per_device_train_batch_size: 16
  • per_device_eval_batch_size: 16
  • per_gpu_train_batch_size: None
  • per_gpu_eval_batch_size: None
  • gradient_accumulation_steps: 16
  • eval_accumulation_steps: None
  • torch_empty_cache_steps: None
  • learning_rate: 2e-05
  • weight_decay: 0.0
  • adam_beta1: 0.9
  • adam_beta2: 0.999
  • adam_epsilon: 1e-08
  • max_grad_norm: 1.0
  • num_train_epochs: 5
  • max_steps: -1
  • lr_scheduler_type: cosine
  • lr_scheduler_kwargs: {}
  • warmup_ratio: 0.2
  • warmup_steps: 0
  • log_level: passive
  • log_level_replica: warning
  • log_on_each_node: True
  • logging_nan_inf_filter: True
  • save_safetensors: True
  • save_on_each_node: False
  • save_only_model: False
  • restore_callback_states_from_checkpoint: False
  • no_cuda: False
  • use_cpu: False
  • use_mps_device: False
  • seed: 42
  • data_seed: None
  • jit_mode_eval: False
  • use_ipex: False
  • bf16: True
  • fp16: False
  • fp16_opt_level: O1
  • half_precision_backend: auto
  • bf16_full_eval: False
  • fp16_full_eval: False
  • tf32: True
  • local_rank: 0
  • ddp_backend: None
  • tpu_num_cores: None
  • tpu_metrics_debug: False
  • debug: []
  • dataloader_drop_last: False
  • dataloader_num_workers: 0
  • dataloader_prefetch_factor: None
  • past_index: -1
  • disable_tqdm: False
  • remove_unused_columns: True
  • label_names: None
  • load_best_model_at_end: True
  • ignore_data_skip: False
  • fsdp: []
  • fsdp_min_num_params: 0
  • fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
  • fsdp_transformer_layer_cls_to_wrap: None
  • accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
  • deepspeed: None
  • label_smoothing_factor: 0.0
  • optim: adamw_torch_fused
  • optim_args: None
  • adafactor: False
  • group_by_length: False
  • length_column_name: length
  • ddp_find_unused_parameters: None
  • ddp_bucket_cap_mb: None
  • ddp_broadcast_buffers: False
  • dataloader_pin_memory: True
  • dataloader_persistent_workers: False
  • skip_memory_metrics: True
  • use_legacy_prediction_loop: False
  • push_to_hub: False
  • resume_from_checkpoint: None
  • hub_model_id: None
  • hub_strategy: every_save
  • hub_private_repo: False
  • hub_always_push: False
  • gradient_checkpointing: False
  • gradient_checkpointing_kwargs: None
  • include_inputs_for_metrics: False
  • eval_do_concat_batches: True
  • fp16_backend: auto
  • push_to_hub_model_id: None
  • push_to_hub_organization: None
  • mp_parameters:
  • auto_find_batch_size: False
  • full_determinism: False
  • torchdynamo: None
  • ray_scope: last
  • ddp_timeout: 1800
  • torch_compile: False
  • torch_compile_backend: None
  • torch_compile_mode: None
  • dispatch_batches: None
  • split_batches: None
  • include_tokens_per_second: False
  • include_num_input_tokens_seen: False
  • neftune_noise_alpha: None
  • optim_target_modules: None
  • batch_eval_metrics: False
  • eval_on_start: False
  • eval_use_gather_object: False
  • batch_sampler: no_duplicates
  • multi_dataset_batch_sampler: proportional

Training Logs

Epoch Step Training Loss dim_1024_cosine_map@100 dim_768_cosine_map@100 dim_512_cosine_map@100 dim_256_cosine_map@100 dim_128_cosine_map@100 dim_64_cosine_map@100
0.9664 9 - 0.4730 0.4766 0.4640 0.4612 0.4456 0.4083
1.0738 10 2.6023 - - - - - -
1.9329 18 - 0.4951 0.4966 0.4977 0.4773 0.4849 0.4501
2.1477 20 0.974 - - - - - -
2.8993 27 - 0.4891 0.4973 0.4941 0.4867 0.4925 0.4684
3.2215 30 0.408 - - - - - -
3.9732 37 - 0.4944 0.4998 0.4931 0.4991 0.4974 0.4616
4.2953 40 0.2718 - - - - - -
4.8322 45 - 0.4924 0.4952 0.4908 0.4920 0.4973 0.4595
  • The bold row denotes the saved checkpoint.

Framework Versions

  • Python: 3.10.12
  • Sentence Transformers: 3.2.0
  • Transformers: 4.44.2
  • PyTorch: 2.4.1+cu121
  • Accelerate: 1.1.0.dev0
  • Datasets: 3.0.1
  • Tokenizers: 0.19.1

Citation

BibTeX

Sentence Transformers 

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MatryoshkaLoss 

@misc{kusupati2024matryoshka,
    title={Matryoshka Representation Learning},
    author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
    year={2024},
    eprint={2205.13147},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

MultipleNegativesRankingLoss 

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}
Downloads last month
8
Safetensors
Model size
568M params
Tensor type
F32
·
Inference Examples
Sentence Similarity
This model does not have enough activity to be deployed to Inference API (serverless) yet. Increase its social visibility and check back later, or deploy to Inference Endpoints (dedicated) instead.

Model tree for adriansanz/ST-tramits-SB-001-5ep

Base model

BAAI/bge-m3
Finetuned
(129)
this model

Evaluation results

View on Papers With Code