Hugging Face's logo

stephantulkens
/
NIFE-gte-modernbert-base

Safetensors
English
Model card Files
xet
Community
1
stephantulkens commited on
Commit
09460c0
·
unverified ·
1 Parent(s): cde3ed4

update model to query

Browse files
Files changed (16) hide show
  1. document_0_SentenceTransformer/1_Pooling/config.json +0 -10
  2. document_0_SentenceTransformer/README.md +0 -254
  3. document_0_SentenceTransformer/config.json +0 -45
  4. document_0_SentenceTransformer/config_sentence_transformers.json +0 -14
  5. document_0_SentenceTransformer/model.safetensors +0 -3
  6. document_0_SentenceTransformer/modules.json +0 -14
  7. document_0_SentenceTransformer/sentence_bert_config.json +0 -4
  8. document_0_SentenceTransformer/special_tokens_map.json +0 -37
  9. document_0_SentenceTransformer/tokenizer.json +0 -0
  10. document_0_SentenceTransformer/tokenizer_config.json +0 -945
  11. query_0_SentenceTransformer/README.md +0 -147
  12. query_0_SentenceTransformer/config_sentence_transformers.json +0 -14
  13. query_0_SentenceTransformer/model.safetensors +0 -3
  14. query_0_SentenceTransformer/modules.json +0 -14
  15. query_0_SentenceTransformer/tokenizer.json +0 -0
  16. router_config.json +0 -18
document_0_SentenceTransformer/1_Pooling/config.json DELETED
@@ -1,10 +0,0 @@
1
- {
2
- "word_embedding_dimension": 768,
3
- "pooling_mode_cls_token": true,
4
- "pooling_mode_mean_tokens": false,
5
- "pooling_mode_max_tokens": false,
6
- "pooling_mode_mean_sqrt_len_tokens": false,
7
- "pooling_mode_weightedmean_tokens": false,
8
- "pooling_mode_lasttoken": false,
9
- "include_prompt": true
10
- }
 
 
 
 
 
 
 
 
 
 
 
document_0_SentenceTransformer/README.md DELETED
@@ -1,254 +0,0 @@
1
- ---
2
- license: apache-2.0
3
- language:
4
- - en
5
- base_model:
6
- - answerdotai/ModernBERT-base
7
- base_model_relation: finetune
8
- pipeline_tag: sentence-similarity
9
- library_name: transformers
10
- tags:
11
- - sentence-transformers
12
- - mteb
13
- - embedding
14
- - transformers.js
15
- - text-embeddings-inference
16
- ---
17
-
18
- # gte-modernbert-base
19
-
20
- We are excited to introduce the `gte-modernbert` series of models, which are built upon the latest modernBERT pre-trained encoder-only foundation models. The `gte-modernbert` series models include both text embedding models and rerank models.
21
-
22
- The `gte-modernbert` models demonstrates competitive performance in several text embedding and text retrieval evaluation tasks when compared to similar-scale models from the current open-source community. This includes assessments such as MTEB, LoCO, and COIR evaluation.
23
-
24
- ## Model Overview
25
-
26
- - Developed by: Tongyi Lab, Alibaba Group
27
- - Model Type: Text Embedding
28
- - Primary Language: English
29
- - Model Size: 149M
30
- - Max Input Length: 8192 tokens
31
- - Output Dimension: 768
32
-
33
- ### Model list
34
-
35
-
36
- | Models | Language | Model Type | Model Size | Max Seq. Length | Dimension | MTEB-en | BEIR | LoCo | CoIR |
37
- |:--------------------------------------------------------------------------------------:|:--------:|:----------------------:|:----------:|:---------------:|:---------:|:-------:|:----:|:----:|:----:|
38
- | [`gte-modernbert-base`](https://huggingface.co/Alibaba-NLP/gte-modernbert-base) | English | text embedding | 149M | 8192 | 768 | 64.38 | 55.33 | 87.57 | 79.31 |
39
- | [`gte-reranker-modernbert-base`](https://huggingface.co/Alibaba-NLP/gte-reranker-modernbert-base) | English | text reranker | 149M | 8192 | - | - | 56.19 | 90.68 | 79.99 |
40
-
41
- ## Usage
42
-
43
- > [!TIP]
44
- > For `transformers` and `sentence-transformers`, if your GPU supports it, the efficient Flash Attention 2 will be used automatically if you have `flash_attn` installed. It is not mandatory.
45
- >
46
- > ```bash
47
- > pip install flash_attn
48
- > ```
49
-
50
- Use with `transformers`
51
-
52
- ```python
53
- # Requires transformers>=4.48.0
54
-
55
- import torch.nn.functional as F
56
- from transformers import AutoModel, AutoTokenizer
57
-
58
- input_texts = [
59
- "what is the capital of China?",
60
- "how to implement quick sort in python?",
61
- "Beijing",
62
- "sorting algorithms"
63
- ]
64
-
65
- model_path = "Alibaba-NLP/gte-modernbert-base"
66
- tokenizer = AutoTokenizer.from_pretrained(model_path)
67
- model = AutoModel.from_pretrained(model_path)
68
-
69
- # Tokenize the input texts
70
- batch_dict = tokenizer(input_texts, max_length=8192, padding=True, truncation=True, return_tensors='pt')
71
-
72
- outputs = model(**batch_dict)
73
- embeddings = outputs.last_hidden_state[:, 0]
74
-
75
- # (Optionally) normalize embeddings
76
- embeddings = F.normalize(embeddings, p=2, dim=1)
77
- scores = (embeddings[:1] @ embeddings[1:].T) * 100
78
- print(scores.tolist())
79
- # [[42.89073944091797, 71.30911254882812, 33.664554595947266]]
80
- ```
81
-
82
- Use with `sentence-transformers`:
83
-
84
- ```python
85
- # Requires transformers>=4.48.0
86
- from sentence_transformers import SentenceTransformer
87
- from sentence_transformers.util import cos_sim
88
-
89
- input_texts = [
90
- "what is the capital of China?",
91
- "how to implement quick sort in python?",
92
- "Beijing",
93
- "sorting algorithms"
94
- ]
95
-
96
- model = SentenceTransformer("Alibaba-NLP/gte-modernbert-base")
97
- embeddings = model.encode(input_texts)
98
- print(embeddings.shape)
99
- # (4, 768)
100
-
101
- similarities = cos_sim(embeddings[0], embeddings[1:])
102
- print(similarities)
103
- # tensor([[0.4289, 0.7131, 0.3366]])
104
- ```
105
-
106
- Use with `transformers.js`:
107
-
108
- ```js
109
- // npm i @huggingface/transformers
110
- import { pipeline, matmul } from "@huggingface/transformers";
111
-
112
- // Create a feature extraction pipeline
113
- const extractor = await pipeline(
114
- "feature-extraction",
115
- "Alibaba-NLP/gte-modernbert-base",
116
- { dtype: "fp32" }, // Supported options: "fp32", "fp16", "q8", "q4", "q4f16"
117
- );
118
-
119
- // Embed queries and documents
120
- const embeddings = await extractor(
121
- [
122
- "what is the capital of China?",
123
- "how to implement quick sort in python?",
124
- "Beijing",
125
- "sorting algorithms",
126
- ],
127
- { pooling: "cls", normalize: true },
128
- );
129
-
130
- // Compute similarity scores
131
- const similarities = (await matmul(embeddings.slice([0, 1]), embeddings.slice([1, null]).transpose(1, 0))).mul(100);
132
- console.log(similarities.tolist()); // [[42.89077377319336, 71.30916595458984, 33.66455841064453]]
133
- ```
134
-
135
- Additionally, you can also deploy `Alibaba-NLP/gte-modernbert-base` with [Text Embeddings Inference (TEI)](https://github.com/huggingface/text-embeddings-inference) as follows:
136
-
137
- - CPU
138
-
139
- ```bash
140
- docker run --platform linux/amd64 \
141
- -p 8080:80 \
142
- -v $PWD/data:/data \
143
- --pull always \
144
- ghcr.io/huggingface/text-embeddings-inference:cpu-1.7 \
145
- --model-id Alibaba-NLP/gte-modernbert-base
146
- ```
147
-
148
- - GPU
149
-
150
- ```bash
151
- docker run --gpus all \
152
- -p 8080:80 \
153
- -v $PWD/data:/data \
154
- --pull always \
155
- ghcr.io/huggingface/text-embeddings-inference:1.7 \
156
- --model-id Alibaba-NLP/gte-modernbert-base
157
- ```
158
-
159
- Then you can send requests to the deployed API via the OpenAI-compatible `v1/embeddings` route (more information about the [OpenAI Embeddings API](https://platform.openai.com/docs/api-reference/embeddings)):
160
-
161
- ```bash
162
- curl https://0.0.0.0:8080/v1/embeddings \
163
- -H "Content-Type: application/json" \
164
- -d '{
165
- "input": [
166
- "what is the capital of China?",
167
- "how to implement quick sort in python?",
168
- "Beijing",
169
- "sorting algorithms"
170
- ],
171
- "model": "Alibaba-NLP/gte-modernbert-base",
172
- "encoding_format": "float"
173
- }'
174
- ```
175
-
176
- ## Training Details
177
-
178
- The `gte-modernbert` series of models follows the training scheme of the previous [GTE models](https://huggingface.co/collections/Alibaba-NLP/gte-models-6680f0b13f885cb431e6d469), with the only difference being that the pre-training language model base has been replaced from [GTE-MLM](https://huggingface.co/Alibaba-NLP/gte-en-mlm-base) to [ModernBert](https://huggingface.co/answerdotai/ModernBERT-base). For more training details, please refer to our paper: [mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrieval](https://aclanthology.org/2024.emnlp-industry.103/)
179
-
180
- ## Evaluation
181
-
182
- ### MTEB
183
-
184
- The results of other models are retrieved from [MTEB leaderboard](https://huggingface.co/spaces/mteb/leaderboard). Given that all models in the `gte-modernbert` series have a size of less than 1B parameters, we focused exclusively on the results of models under 1B from the MTEB leaderboard.
185
-
186
- | Model Name | Param Size (M) | Dimension | Sequence Length | Average (56) | Class. (12) | Clust. (11) | Pair Class. (3) | Reran. (4) | Retr. (15) | STS (10) | Summ. (1) |
187
- |:------------------------------------------------------------------------------------------------:|:--------------:|:---------:|:---------------:|:------------:|:-----------:|:---:|:---:|:---:|:---:|:-----------:|:--------:|
188
- | [mxbai-embed-large-v1](https://huggingface.co/mixedbread-ai/mxbai-embed-large-v1) | 335 | 1024 | 512 | 64.68 | 75.64 | 46.71 | 87.2 | 60.11 | 54.39 | 85 | 32.71 |
189
- | [multilingual-e5-large-instruct](https://huggingface.co/intfloat/multilingual-e5-large-instruct) | 560 | 1024 | 514 | 64.41 | 77.56 | 47.1 | 86.19 | 58.58 | 52.47 | 84.78 | 30.39 |
190
- | [bge-large-en-v1.5](https://huggingface.co/BAAI/bge-large-en-v1.5) | 335 | 1024 | 512 | 64.23 | 75.97 | 46.08 | 87.12 | 60.03 | 54.29 | 83.11 | 31.61 |
191
- | [gte-base-en-v1.5](https://huggingface.co/Alibaba-NLP/gte-base-en-v1.5) | 137 | 768 | 8192 | 64.11 | 77.17 | 46.82 | 85.33 | 57.66 | 54.09 | 81.97 | 31.17 |
192
- | [bge-base-en-v1.5](https://huggingface.co/BAAI/bge-base-en-v1.5) | 109 | 768 | 512 | 63.55 | 75.53 | 45.77 | 86.55 | 58.86 | 53.25 | 82.4 | 31.07 |
193
- | [gte-large-en-v1.5](https://huggingface.co/Alibaba-NLP/gte-large-en-v1.5) | 409 | 1024 | 8192 | 65.39 | 77.75 | 47.95 | 84.63 | 58.50 | 57.91 | 81.43 | 30.91 |
194
- | [modernbert-embed-base](https://huggingface.co/nomic-ai/modernbert-embed-base) | 149 | 768 | 8192 | 62.62 | 74.31 | 44.98 | 83.96 | 56.42 | 52.89 | 81.78 | 31.39 |
195
- | [nomic-embed-text-v1.5](https://huggingface.co/nomic-ai/nomic-embed-text-v1.5) | | 768 | 8192 | 62.28 | 73.55 | 43.93 | 84.61 | 55.78 | 53.01| 81.94 | 30.4 |
196
- | [gte-multilingual-base](https://huggingface.co/Alibaba-NLP/gte-multilingual-base) | 305 | 768 | 8192 | 61.4 | 70.89 | 44.31 | 84.24 | 57.47 |51.08 | 82.11 | 30.58 |
197
- | [jina-embeddings-v3](https://huggingface.co/jinaai/jina-embeddings-v3) | 572 | 1024 | 8192 | 65.51 | 82.58 |45.21 |84.01 |58.13 |53.88 | 85.81 | 29.71 |
198
- | [**gte-modernbert-base**](https://huggingface.co/Alibaba-NLP/gte-modernbert-base) | 149 | 768 | 8192 | **64.38** | **76.99** | **46.47** | **85.93** | **59.24** | **55.33** | **81.57** | **30.68** |
199
-
200
-
201
- ### LoCo (Long Document Retrieval)(NDCG@10)
202
-
203
- | Model Name | Dimension | Sequence Length | Average (5) | QsmsumRetrieval | SummScreenRetrieval | QasperAbastractRetrieval | QasperTitleRetrieval | GovReportRetrieval |
204
- |:----:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|
205
- | [gte-qwen1.5-7b](https://huggingface.co/Alibaba-NLP/gte-qwen1.5-7b) | 4096 | 32768 | 87.57 | 49.37 | 93.10 | 99.67 | 97.54 | 98.21 |
206
- | [gte-large-v1.5](https://huggingface.co/Alibaba-NLP/gte-large-v1.5) |1024 | 8192 | 86.71 | 44.55 | 92.61 | 99.82 | 97.81 | 98.74 |
207
- | [gte-base-v1.5](https://huggingface.co/Alibaba-NLP/gte-base-v1.5) | 768 | 8192 | 87.44 | 49.91 | 91.78 | 99.82 | 97.13 | 98.58 |
208
- | [gte-modernbert-base](https://huggingface.co/Alibaba-NLP/gte-modernbert-base) | 768 | 8192 | 88.88 | 54.45 | 93.00 | 99.82 | 98.03 | 98.70 |
209
- | [gte-reranker-modernbert-base](https://huggingface.co/Alibaba-NLP/gte-reranker-modernbert-base) | - | 8192 | 90.68 | 70.86 | 94.06 | 99.73 | 99.11 | 89.67 |
210
-
211
- ### COIR (Code Retrieval Task)(NDCG@10)
212
-
213
- | Model Name | Dimension | Sequence Length | Average(20) | CodeSearchNet-ccr-go | CodeSearchNet-ccr-java | CodeSearchNet-ccr-javascript | CodeSearchNet-ccr-php | CodeSearchNet-ccr-python | CodeSearchNet-ccr-ruby | CodeSearchNet-go | CodeSearchNet-java | CodeSearchNet-javascript | CodeSearchNet-php | CodeSearchNet-python | CodeSearchNet-ruby | apps | codefeedback-mt | codefeedback-st | codetrans-contest | codetrans-dl | cosqa | stackoverflow-qa | synthetic-text2sql |
214
- |:----:|:---:|:---:|:---:|:---:| :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
215
- | [gte-modernbert-base](https://huggingface.co/Alibaba-NLP/gte-modernbert-base) | 768 | 8192 | 79.31 | 94.15 | 93.57 | 94.27 | 91.51 | 93.93 | 90.63 | 88.32 | 83.27 | 76.05 | 85.12 | 88.16 | 77.59 | 57.54 | 82.34 | 85.95 | 71.89 | 35.46 | 43.47 | 91.2 | 61.87 |
216
- | [gte-reranker-modernbert-base](https://huggingface.co/Alibaba-NLP/gte-reranker-modernbert-base) | - | 8192 | 79.99 | 96.43 | 96.88 | 98.32 | 91.81 | 97.7 | 91.96 | 88.81 | 79.71 | 76.27 | 89.39 | 98.37 | 84.11 | 47.57 | 83.37 | 88.91 | 49.66 | 36.36 | 44.37 | 89.58 | 64.21 |
217
-
218
- ### BEIR(NDCG@10)
219
-
220
- | Model Name | Dimension | Sequence Length | Average(15) | ArguAna | ClimateFEVER | CQADupstackAndroidRetrieval | DBPedia | FEVER | FiQA2018 | HotpotQA | MSMARCO | NFCorpus | NQ | QuoraRetrieval | SCIDOCS | SciFact | Touche2020 | TRECCOVID |
221
- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
222
- | [gte-modernbert-base](https://huggingface.co/Alibaba-NLP/gte-modernbert-base) | 768 | 8192 | 55.33 | 72.68 | 37.74 | 42.63 | 41.79 | 91.03 | 48.81 | 69.47 | 40.9 | 36.44 | 57.62 | 88.55 | 21.29 | 77.4 | 21.68 | 81.95 |
223
- | [gte-reranker-modernbert-base](https://huggingface.co/Alibaba-NLP/gte-reranker-modernbert-base) | - | 8192 | 56.73 | 69.03 | 37.79 | 44.68 | 47.23 | 94.54 | 49.81 | 78.16 | 45.38 | 30.69 | 64.57 | 87.77 | 20.60 | 73.57 | 27.36 | 79.89 |
224
-
225
-
226
-
227
- ## Hiring
228
-
229
- We have open positions for **Research Interns** and **Full-Time Researchers** to join our team at Tongyi Lab.
230
- We are seeking passionate individuals with expertise in representation learning, LLM-driven information retrieval, Retrieval-Augmented Generation (RAG), and agent-based systems.
231
- Our team is located in the vibrant cities of **Beijing** and **Hangzhou**.
232
- If you are driven by curiosity and eager to make a meaningful impact through your work, we would love to hear from you. Please submit your resume along with a brief introduction to <a href="mailto:dingkun.ldk@alibaba-inc.com">dingkun.ldk@alibaba-inc.com</a>.
233
-
234
-
235
- ## Citation
236
-
237
- If you find our paper or models helpful, feel free to give us a cite.
238
-
239
- ```
240
- @inproceedings{zhang2024mgte,
241
- title={mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrieval},
242
- author={Zhang, Xin and Zhang, Yanzhao and Long, Dingkun and Xie, Wen and Dai, Ziqi and Tang, Jialong and Lin, Huan and Yang, Baosong and Xie, Pengjun and Huang, Fei and others},
243
- booktitle={Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track},
244
- pages={1393--1412},
245
- year={2024}
246
- }
247
-
248
- @article{li2023towards,
249
- title={Towards general text embeddings with multi-stage contrastive learning},
250
- author={Li, Zehan and Zhang, Xin and Zhang, Yanzhao and Long, Dingkun and Xie, Pengjun and Zhang, Meishan},
251
- journal={arXiv preprint arXiv:2308.03281},
252
- year={2023}
253
- }
254
- ```
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
document_0_SentenceTransformer/config.json DELETED
@@ -1,45 +0,0 @@
1
- {
2
- "architectures": [
3
- "ModernBertModel"
4
- ],
5
- "attention_bias": false,
6
- "attention_dropout": 0.0,
7
- "bos_token_id": 50281,
8
- "classifier_activation": "gelu",
9
- "classifier_bias": false,
10
- "classifier_dropout": 0.0,
11
- "classifier_pooling": "mean",
12
- "cls_token_id": 50281,
13
- "decoder_bias": true,
14
- "deterministic_flash_attn": false,
15
- "dtype": "float32",
16
- "embedding_dropout": 0.0,
17
- "eos_token_id": 50282,
18
- "global_attn_every_n_layers": 3,
19
- "global_rope_theta": 160000.0,
20
- "gradient_checkpointing": false,
21
- "hidden_activation": "gelu",
22
- "hidden_size": 768,
23
- "initializer_cutoff_factor": 2.0,
24
- "initializer_range": 0.02,
25
- "intermediate_size": 1152,
26
- "layer_norm_eps": 1e-05,
27
- "local_attention": 128,
28
- "local_rope_theta": 10000.0,
29
- "max_position_embeddings": 8192,
30
- "mlp_bias": false,
31
- "mlp_dropout": 0.0,
32
- "model_type": "modernbert",
33
- "norm_bias": false,
34
- "norm_eps": 1e-05,
35
- "num_attention_heads": 12,
36
- "num_hidden_layers": 22,
37
- "pad_token_id": 50283,
38
- "position_embedding_type": "absolute",
39
- "repad_logits_with_grad": false,
40
- "sep_token_id": 50282,
41
- "sparse_pred_ignore_index": -100,
42
- "sparse_prediction": false,
43
- "transformers_version": "4.56.2",
44
- "vocab_size": 50368
45
- }
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
document_0_SentenceTransformer/config_sentence_transformers.json DELETED
@@ -1,14 +0,0 @@
1
- {
2
- "__version__": {
3
- "sentence_transformers": "5.1.1",
4
- "transformers": "4.56.2",
5
- "pytorch": "2.8.0"
6
- },
7
- "prompts": {
8
- "query": "",
9
- "document": ""
10
- },
11
- "default_prompt_name": null,
12
- "similarity_fn_name": "cosine",
13
- "model_type": "SentenceTransformer"
14
- }
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
document_0_SentenceTransformer/model.safetensors DELETED
@@ -1,3 +0,0 @@
1
- version https://git-lfs.github.com/spec/v1
2
- oid sha256:0f9247027e7d57e8b36440b5b3d10a785ded92c7c9f4a313ff7f54a549967290
3
- size 596070136
 
 
 
 
document_0_SentenceTransformer/modules.json DELETED
@@ -1,14 +0,0 @@
1
- [
2
- {
3
- "idx": 0,
4
- "name": "0",
5
- "path": "",
6
- "type": "sentence_transformers.models.Transformer"
7
- },
8
- {
9
- "idx": 1,
10
- "name": "1",
11
- "path": "1_Pooling",
12
- "type": "sentence_transformers.models.Pooling"
13
- }
14
- ]
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
document_0_SentenceTransformer/sentence_bert_config.json DELETED
@@ -1,4 +0,0 @@
1
- {
2
- "max_seq_length": 8192,
3
- "do_lower_case": false
4
- }
 
 
 
 
 
document_0_SentenceTransformer/special_tokens_map.json DELETED
@@ -1,37 +0,0 @@
1
- {
2
- "cls_token": {
3
- "content": "[CLS]",
4
- "lstrip": false,
5
- "normalized": false,
6
- "rstrip": false,
7
- "single_word": false
8
- },
9
- "mask_token": {
10
- "content": "[MASK]",
11
- "lstrip": true,
12
- "normalized": false,
13
- "rstrip": false,
14
- "single_word": false
15
- },
16
- "pad_token": {
17
- "content": "[PAD]",
18
- "lstrip": false,
19
- "normalized": false,
20
- "rstrip": false,
21
- "single_word": false
22
- },
23
- "sep_token": {
24
- "content": "[SEP]",
25
- "lstrip": false,
26
- "normalized": false,
27
- "rstrip": false,
28
- "single_word": false
29
- },
30
- "unk_token": {
31
- "content": "[UNK]",
32
- "lstrip": false,
33
- "normalized": false,
34
- "rstrip": false,
35
- "single_word": false
36
- }
37
- }
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
document_0_SentenceTransformer/tokenizer.json DELETED
The diff for this file is too large to render. See raw diff
 
document_0_SentenceTransformer/tokenizer_config.json DELETED
@@ -1,945 +0,0 @@
1
- {
2
- "added_tokens_decoder": {
3
- "0": {
4
- "content": "|||IP_ADDRESS|||",
5
- "lstrip": false,
6
- "normalized": true,
7
- "rstrip": false,
8
- "single_word": false,
9
- "special": false
10
- },
11
- "1": {
12
- "content": "<|padding|>",
13
- "lstrip": false,
14
- "normalized": false,
15
- "rstrip": false,
16
- "single_word": false,
17
- "special": true
18
- },
19
- "50254": {
20
- "content": " ",
21
- "lstrip": false,
22
- "normalized": true,
23
- "rstrip": false,
24
- "single_word": false,
25
- "special": false
26
- },
27
- "50255": {
28
- "content": " ",
29
- "lstrip": false,
30
- "normalized": true,
31
- "rstrip": false,
32
- "single_word": false,
33
- "special": false
34
- },
35
- "50256": {
36
- "content": " ",
37
- "lstrip": false,
38
- "normalized": true,
39
- "rstrip": false,
40
- "single_word": false,
41
- "special": false
42
- },
43
- "50257": {
44
- "content": " ",
45
- "lstrip": false,
46
- "normalized": true,
47
- "rstrip": false,
48
- "single_word": false,
49
- "special": false
50
- },
51
- "50258": {
52
- "content": " ",
53
- "lstrip": false,
54
- "normalized": true,
55
- "rstrip": false,
56
- "single_word": false,
57
- "special": false
58
- },
59
- "50259": {
60
- "content": " ",
61
- "lstrip": false,
62
- "normalized": true,
63
- "rstrip": false,
64
- "single_word": false,
65
- "special": false
66
- },
67
- "50260": {
68
- "content": " ",
69
- "lstrip": false,
70
- "normalized": true,
71
- "rstrip": false,
72
- "single_word": false,
73
- "special": false
74
- },
75
- "50261": {
76
- "content": " ",
77
- "lstrip": false,
78
- "normalized": true,
79
- "rstrip": false,
80
- "single_word": false,
81
- "special": false
82
- },
83
- "50262": {
84
- "content": " ",
85
- "lstrip": false,
86
- "normalized": true,
87
- "rstrip": false,
88
- "single_word": false,
89
- "special": false
90
- },
91
- "50263": {
92
- "content": " ",
93
- "lstrip": false,
94
- "normalized": true,
95
- "rstrip": false,
96
- "single_word": false,
97
- "special": false
98
- },
99
- "50264": {
100
- "content": " ",
101
- "lstrip": false,
102
- "normalized": true,
103
- "rstrip": false,
104
- "single_word": false,
105
- "special": false
106
- },
107
- "50265": {
108
- "content": " ",
109
- "lstrip": false,
110
- "normalized": true,
111
- "rstrip": false,
112
- "single_word": false,
113
- "special": false
114
- },
115
- "50266": {
116
- "content": " ",
117
- "lstrip": false,
118
- "normalized": true,
119
- "rstrip": false,
120
- "single_word": false,
121
- "special": false
122
- },
123
- "50267": {
124
- "content": " ",
125
- "lstrip": false,
126
- "normalized": true,
127
- "rstrip": false,
128
- "single_word": false,
129
- "special": false
130
- },
131
- "50268": {
132
- "content": " ",
133
- "lstrip": false,
134
- "normalized": true,
135
- "rstrip": false,
136
- "single_word": false,
137
- "special": false
138
- },
139
- "50269": {
140
- "content": " ",
141
- "lstrip": false,
142
- "normalized": true,
143
- "rstrip": false,
144
- "single_word": false,
145
- "special": false
146
- },
147
- "50270": {
148
- "content": " ",
149
- "lstrip": false,
150
- "normalized": true,
151
- "rstrip": false,
152
- "single_word": false,
153
- "special": false
154
- },
155
- "50271": {
156
- "content": " ",
157
- "lstrip": false,
158
- "normalized": true,
159
- "rstrip": false,
160
- "single_word": false,
161
- "special": false
162
- },
163
- "50272": {
164
- "content": " ",
165
- "lstrip": false,
166
- "normalized": true,
167
- "rstrip": false,
168
- "single_word": false,
169
- "special": false
170
- },
171
- "50273": {
172
- "content": " ",
173
- "lstrip": false,
174
- "normalized": true,
175
- "rstrip": false,
176
- "single_word": false,
177
- "special": false
178
- },
179
- "50274": {
180
- "content": " ",
181
- "lstrip": false,
182
- "normalized": true,
183
- "rstrip": false,
184
- "single_word": false,
185
- "special": false
186
- },
187
- "50275": {
188
- "content": " ",
189
- "lstrip": false,
190
- "normalized": true,
191
- "rstrip": false,
192
- "single_word": false,
193
- "special": false
194
- },
195
- "50276": {
196
- "content": " ",
197
- "lstrip": false,
198
- "normalized": true,
199
- "rstrip": false,
200
- "single_word": false,
201
- "special": false
202
- },
203
- "50277": {
204
- "content": "|||EMAIL_ADDRESS|||",
205
- "lstrip": false,
206
- "normalized": true,
207
- "rstrip": false,
208
- "single_word": false,
209
- "special": false
210
- },
211
- "50278": {
212
- "content": "|||PHONE_NUMBER|||",
213
- "lstrip": false,
214
- "normalized": true,
215
- "rstrip": false,
216
- "single_word": false,
217
- "special": false
218
- },
219
- "50279": {
220
- "content": "<|endoftext|>",
221
- "lstrip": false,
222
- "normalized": false,
223
- "rstrip": false,
224
- "single_word": false,
225
- "special": true
226
- },
227
- "50280": {
228
- "content": "[UNK]",
229
- "lstrip": false,
230
- "normalized": false,
231
- "rstrip": false,
232
- "single_word": false,
233
- "special": true
234
- },
235
- "50281": {
236
- "content": "[CLS]",
237
- "lstrip": false,
238
- "normalized": false,
239
- "rstrip": false,
240
- "single_word": false,
241
- "special": true
242
- },
243
- "50282": {
244
- "content": "[SEP]",
245
- "lstrip": false,
246
- "normalized": false,
247
- "rstrip": false,
248
- "single_word": false,
249
- "special": true
250
- },
251
- "50283": {
252
- "content": "[PAD]",
253
- "lstrip": false,
254
- "normalized": false,
255
- "rstrip": false,
256
- "single_word": false,
257
- "special": true
258
- },
259
- "50284": {
260
- "content": "[MASK]",
261
- "lstrip": true,
262
- "normalized": false,
263
- "rstrip": false,
264
- "single_word": false,
265
- "special": true
266
- },
267
- "50285": {
268
- "content": "[unused0]",
269
- "lstrip": false,
270
- "normalized": true,
271
- "rstrip": false,
272
- "single_word": false,
273
- "special": false
274
- },
275
- "50286": {
276
- "content": "[unused1]",
277
- "lstrip": false,
278
- "normalized": true,
279
- "rstrip": false,
280
- "single_word": false,
281
- "special": false
282
- },
283
- "50287": {
284
- "content": "[unused2]",
285
- "lstrip": false,
286
- "normalized": true,
287
- "rstrip": false,
288
- "single_word": false,
289
- "special": false
290
- },
291
- "50288": {
292
- "content": "[unused3]",
293
- "lstrip": false,
294
- "normalized": true,
295
- "rstrip": false,
296
- "single_word": false,
297
- "special": false
298
- },
299
- "50289": {
300
- "content": "[unused4]",
301
- "lstrip": false,
302
- "normalized": true,
303
- "rstrip": false,
304
- "single_word": false,
305
- "special": false
306
- },
307
- "50290": {
308
- "content": "[unused5]",
309
- "lstrip": false,
310
- "normalized": true,
311
- "rstrip": false,
312
- "single_word": false,
313
- "special": false
314
- },
315
- "50291": {
316
- "content": "[unused6]",
317
- "lstrip": false,
318
- "normalized": true,
319
- "rstrip": false,
320
- "single_word": false,
321
- "special": false
322
- },
323
- "50292": {
324
- "content": "[unused7]",
325
- "lstrip": false,
326
- "normalized": true,
327
- "rstrip": false,
328
- "single_word": false,
329
- "special": false
330
- },
331
- "50293": {
332
- "content": "[unused8]",
333
- "lstrip": false,
334
- "normalized": true,
335
- "rstrip": false,
336
- "single_word": false,
337
- "special": false
338
- },
339
- "50294": {
340
- "content": "[unused9]",
341
- "lstrip": false,
342
- "normalized": true,
343
- "rstrip": false,
344
- "single_word": false,
345
- "special": false
346
- },
347
- "50295": {
348
- "content": "[unused10]",
349
- "lstrip": false,
350
- "normalized": true,
351
- "rstrip": false,
352
- "single_word": false,
353
- "special": false
354
- },
355
- "50296": {
356
- "content": "[unused11]",
357
- "lstrip": false,
358
- "normalized": true,
359
- "rstrip": false,
360
- "single_word": false,
361
- "special": false
362
- },
363
- "50297": {
364
- "content": "[unused12]",
365
- "lstrip": false,
366
- "normalized": true,
367
- "rstrip": false,
368
- "single_word": false,
369
- "special": false
370
- },
371
- "50298": {
372
- "content": "[unused13]",
373
- "lstrip": false,
374
- "normalized": true,
375
- "rstrip": false,
376
- "single_word": false,
377
- "special": false
378
- },
379
- "50299": {
380
- "content": "[unused14]",
381
- "lstrip": false,
382
- "normalized": true,
383
- "rstrip": false,
384
- "single_word": false,
385
- "special": false
386
- },
387
- "50300": {
388
- "content": "[unused15]",
389
- "lstrip": false,
390
- "normalized": true,
391
- "rstrip": false,
392
- "single_word": false,
393
- "special": false
394
- },
395
- "50301": {
396
- "content": "[unused16]",
397
- "lstrip": false,
398
- "normalized": true,
399
- "rstrip": false,
400
- "single_word": false,
401
- "special": false
402
- },
403
- "50302": {
404
- "content": "[unused17]",
405
- "lstrip": false,
406
- "normalized": true,
407
- "rstrip": false,
408
- "single_word": false,
409
- "special": false
410
- },
411
- "50303": {
412
- "content": "[unused18]",
413
- "lstrip": false,
414
- "normalized": true,
415
- "rstrip": false,
416
- "single_word": false,
417
- "special": false
418
- },
419
- "50304": {
420
- "content": "[unused19]",
421
- "lstrip": false,
422
- "normalized": true,
423
- "rstrip": false,
424
- "single_word": false,
425
- "special": false
426
- },
427
- "50305": {
428
- "content": "[unused20]",
429
- "lstrip": false,
430
- "normalized": true,
431
- "rstrip": false,
432
- "single_word": false,
433
- "special": false
434
- },
435
- "50306": {
436
- "content": "[unused21]",
437
- "lstrip": false,
438
- "normalized": true,
439
- "rstrip": false,
440
- "single_word": false,
441
- "special": false
442
- },
443
- "50307": {
444
- "content": "[unused22]",
445
- "lstrip": false,
446
- "normalized": true,
447
- "rstrip": false,
448
- "single_word": false,
449
- "special": false
450
- },
451
- "50308": {
452
- "content": "[unused23]",
453
- "lstrip": false,
454
- "normalized": true,
455
- "rstrip": false,
456
- "single_word": false,
457
- "special": false
458
- },
459
- "50309": {
460
- "content": "[unused24]",
461
- "lstrip": false,
462
- "normalized": true,
463
- "rstrip": false,
464
- "single_word": false,
465
- "special": false
466
- },
467
- "50310": {
468
- "content": "[unused25]",
469
- "lstrip": false,
470
- "normalized": true,
471
- "rstrip": false,
472
- "single_word": false,
473
- "special": false
474
- },
475
- "50311": {
476
- "content": "[unused26]",
477
- "lstrip": false,
478
- "normalized": true,
479
- "rstrip": false,
480
- "single_word": false,
481
- "special": false
482
- },
483
- "50312": {
484
- "content": "[unused27]",
485
- "lstrip": false,
486
- "normalized": true,
487
- "rstrip": false,
488
- "single_word": false,
489
- "special": false
490
- },
491
- "50313": {
492
- "content": "[unused28]",
493
- "lstrip": false,
494
- "normalized": true,
495
- "rstrip": false,
496
- "single_word": false,
497
- "special": false
498
- },
499
- "50314": {
500
- "content": "[unused29]",
501
- "lstrip": false,
502
- "normalized": true,
503
- "rstrip": false,
504
- "single_word": false,
505
- "special": false
506
- },
507
- "50315": {
508
- "content": "[unused30]",
509
- "lstrip": false,
510
- "normalized": true,
511
- "rstrip": false,
512
- "single_word": false,
513
- "special": false
514
- },
515
- "50316": {
516
- "content": "[unused31]",
517
- "lstrip": false,
518
- "normalized": true,
519
- "rstrip": false,
520
- "single_word": false,
521
- "special": false
522
- },
523
- "50317": {
524
- "content": "[unused32]",
525
- "lstrip": false,
526
- "normalized": true,
527
- "rstrip": false,
528
- "single_word": false,
529
- "special": false
530
- },
531
- "50318": {
532
- "content": "[unused33]",
533
- "lstrip": false,
534
- "normalized": true,
535
- "rstrip": false,
536
- "single_word": false,
537
- "special": false
538
- },
539
- "50319": {
540
- "content": "[unused34]",
541
- "lstrip": false,
542
- "normalized": true,
543
- "rstrip": false,
544
- "single_word": false,
545
- "special": false
546
- },
547
- "50320": {
548
- "content": "[unused35]",
549
- "lstrip": false,
550
- "normalized": true,
551
- "rstrip": false,
552
- "single_word": false,
553
- "special": false
554
- },
555
- "50321": {
556
- "content": "[unused36]",
557
- "lstrip": false,
558
- "normalized": true,
559
- "rstrip": false,
560
- "single_word": false,
561
- "special": false
562
- },
563
- "50322": {
564
- "content": "[unused37]",
565
- "lstrip": false,
566
- "normalized": true,
567
- "rstrip": false,
568
- "single_word": false,
569
- "special": false
570
- },
571
- "50323": {
572
- "content": "[unused38]",
573
- "lstrip": false,
574
- "normalized": true,
575
- "rstrip": false,
576
- "single_word": false,
577
- "special": false
578
- },
579
- "50324": {
580
- "content": "[unused39]",
581
- "lstrip": false,
582
- "normalized": true,
583
- "rstrip": false,
584
- "single_word": false,
585
- "special": false
586
- },
587
- "50325": {
588
- "content": "[unused40]",
589
- "lstrip": false,
590
- "normalized": true,
591
- "rstrip": false,
592
- "single_word": false,
593
- "special": false
594
- },
595
- "50326": {
596
- "content": "[unused41]",
597
- "lstrip": false,
598
- "normalized": true,
599
- "rstrip": false,
600
- "single_word": false,
601
- "special": false
602
- },
603
- "50327": {
604
- "content": "[unused42]",
605
- "lstrip": false,
606
- "normalized": true,
607
- "rstrip": false,
608
- "single_word": false,
609
- "special": false
610
- },
611
- "50328": {
612
- "content": "[unused43]",
613
- "lstrip": false,
614
- "normalized": true,
615
- "rstrip": false,
616
- "single_word": false,
617
- "special": false
618
- },
619
- "50329": {
620
- "content": "[unused44]",
621
- "lstrip": false,
622
- "normalized": true,
623
- "rstrip": false,
624
- "single_word": false,
625
- "special": false
626
- },
627
- "50330": {
628
- "content": "[unused45]",
629
- "lstrip": false,
630
- "normalized": true,
631
- "rstrip": false,
632
- "single_word": false,
633
- "special": false
634
- },
635
- "50331": {
636
- "content": "[unused46]",
637
- "lstrip": false,
638
- "normalized": true,
639
- "rstrip": false,
640
- "single_word": false,
641
- "special": false
642
- },
643
- "50332": {
644
- "content": "[unused47]",
645
- "lstrip": false,
646
- "normalized": true,
647
- "rstrip": false,
648
- "single_word": false,
649
- "special": false
650
- },
651
- "50333": {
652
- "content": "[unused48]",
653
- "lstrip": false,
654
- "normalized": true,
655
- "rstrip": false,
656
- "single_word": false,
657
- "special": false
658
- },
659
- "50334": {
660
- "content": "[unused49]",
661
- "lstrip": false,
662
- "normalized": true,
663
- "rstrip": false,
664
- "single_word": false,
665
- "special": false
666
- },
667
- "50335": {
668
- "content": "[unused50]",
669
- "lstrip": false,
670
- "normalized": true,
671
- "rstrip": false,
672
- "single_word": false,
673
- "special": false
674
- },
675
- "50336": {
676
- "content": "[unused51]",
677
- "lstrip": false,
678
- "normalized": true,
679
- "rstrip": false,
680
- "single_word": false,
681
- "special": false
682
- },
683
- "50337": {
684
- "content": "[unused52]",
685
- "lstrip": false,
686
- "normalized": true,
687
- "rstrip": false,
688
- "single_word": false,
689
- "special": false
690
- },
691
- "50338": {
692
- "content": "[unused53]",
693
- "lstrip": false,
694
- "normalized": true,
695
- "rstrip": false,
696
- "single_word": false,
697
- "special": false
698
- },
699
- "50339": {
700
- "content": "[unused54]",
701
- "lstrip": false,
702
- "normalized": true,
703
- "rstrip": false,
704
- "single_word": false,
705
- "special": false
706
- },
707
- "50340": {
708
- "content": "[unused55]",
709
- "lstrip": false,
710
- "normalized": true,
711
- "rstrip": false,
712
- "single_word": false,
713
- "special": false
714
- },
715
- "50341": {
716
- "content": "[unused56]",
717
- "lstrip": false,
718
- "normalized": true,
719
- "rstrip": false,
720
- "single_word": false,
721
- "special": false
722
- },
723
- "50342": {
724
- "content": "[unused57]",
725
- "lstrip": false,
726
- "normalized": true,
727
- "rstrip": false,
728
- "single_word": false,
729
- "special": false
730
- },
731
- "50343": {
732
- "content": "[unused58]",
733
- "lstrip": false,
734
- "normalized": true,
735
- "rstrip": false,
736
- "single_word": false,
737
- "special": false
738
- },
739
- "50344": {
740
- "content": "[unused59]",
741
- "lstrip": false,
742
- "normalized": true,
743
- "rstrip": false,
744
- "single_word": false,
745
- "special": false
746
- },
747
- "50345": {
748
- "content": "[unused60]",
749
- "lstrip": false,
750
- "normalized": true,
751
- "rstrip": false,
752
- "single_word": false,
753
- "special": false
754
- },
755
- "50346": {
756
- "content": "[unused61]",
757
- "lstrip": false,
758
- "normalized": true,
759
- "rstrip": false,
760
- "single_word": false,
761
- "special": false
762
- },
763
- "50347": {
764
- "content": "[unused62]",
765
- "lstrip": false,
766
- "normalized": true,
767
- "rstrip": false,
768
- "single_word": false,
769
- "special": false
770
- },
771
- "50348": {
772
- "content": "[unused63]",
773
- "lstrip": false,
774
- "normalized": true,
775
- "rstrip": false,
776
- "single_word": false,
777
- "special": false
778
- },
779
- "50349": {
780
- "content": "[unused64]",
781
- "lstrip": false,
782
- "normalized": true,
783
- "rstrip": false,
784
- "single_word": false,
785
- "special": false
786
- },
787
- "50350": {
788
- "content": "[unused65]",
789
- "lstrip": false,
790
- "normalized": true,
791
- "rstrip": false,
792
- "single_word": false,
793
- "special": false
794
- },
795
- "50351": {
796
- "content": "[unused66]",
797
- "lstrip": false,
798
- "normalized": true,
799
- "rstrip": false,
800
- "single_word": false,
801
- "special": false
802
- },
803
- "50352": {
804
- "content": "[unused67]",
805
- "lstrip": false,
806
- "normalized": true,
807
- "rstrip": false,
808
- "single_word": false,
809
- "special": false
810
- },
811
- "50353": {
812
- "content": "[unused68]",
813
- "lstrip": false,
814
- "normalized": true,
815
- "rstrip": false,
816
- "single_word": false,
817
- "special": false
818
- },
819
- "50354": {
820
- "content": "[unused69]",
821
- "lstrip": false,
822
- "normalized": true,
823
- "rstrip": false,
824
- "single_word": false,
825
- "special": false
826
- },
827
- "50355": {
828
- "content": "[unused70]",
829
- "lstrip": false,
830
- "normalized": true,
831
- "rstrip": false,
832
- "single_word": false,
833
- "special": false
834
- },
835
- "50356": {
836
- "content": "[unused71]",
837
- "lstrip": false,
838
- "normalized": true,
839
- "rstrip": false,
840
- "single_word": false,
841
- "special": false
842
- },
843
- "50357": {
844
- "content": "[unused72]",
845
- "lstrip": false,
846
- "normalized": true,
847
- "rstrip": false,
848
- "single_word": false,
849
- "special": false
850
- },
851
- "50358": {
852
- "content": "[unused73]",
853
- "lstrip": false,
854
- "normalized": true,
855
- "rstrip": false,
856
- "single_word": false,
857
- "special": false
858
- },
859
- "50359": {
860
- "content": "[unused74]",
861
- "lstrip": false,
862
- "normalized": true,
863
- "rstrip": false,
864
- "single_word": false,
865
- "special": false
866
- },
867
- "50360": {
868
- "content": "[unused75]",
869
- "lstrip": false,
870
- "normalized": true,
871
- "rstrip": false,
872
- "single_word": false,
873
- "special": false
874
- },
875
- "50361": {
876
- "content": "[unused76]",
877
- "lstrip": false,
878
- "normalized": true,
879
- "rstrip": false,
880
- "single_word": false,
881
- "special": false
882
- },
883
- "50362": {
884
- "content": "[unused77]",
885
- "lstrip": false,
886
- "normalized": true,
887
- "rstrip": false,
888
- "single_word": false,
889
- "special": false
890
- },
891
- "50363": {
892
- "content": "[unused78]",
893
- "lstrip": false,
894
- "normalized": true,
895
- "rstrip": false,
896
- "single_word": false,
897
- "special": false
898
- },
899
- "50364": {
900
- "content": "[unused79]",
901
- "lstrip": false,
902
- "normalized": true,
903
- "rstrip": false,
904
- "single_word": false,
905
- "special": false
906
- },
907
- "50365": {
908
- "content": "[unused80]",
909
- "lstrip": false,
910
- "normalized": true,
911
- "rstrip": false,
912
- "single_word": false,
913
- "special": false
914
- },
915
- "50366": {
916
- "content": "[unused81]",
917
- "lstrip": false,
918
- "normalized": true,
919
- "rstrip": false,
920
- "single_word": false,
921
- "special": false
922
- },
923
- "50367": {
924
- "content": "[unused82]",
925
- "lstrip": false,
926
- "normalized": true,
927
- "rstrip": false,
928
- "single_word": false,
929
- "special": false
930
- }
931
- },
932
- "clean_up_tokenization_spaces": true,
933
- "cls_token": "[CLS]",
934
- "extra_special_tokens": {},
935
- "mask_token": "[MASK]",
936
- "model_input_names": [
937
- "input_ids",
938
- "attention_mask"
939
- ],
940
- "model_max_length": 1000000000000000019884624838656,
941
- "pad_token": "[PAD]",
942
- "sep_token": "[SEP]",
943
- "tokenizer_class": "PreTrainedTokenizerFast",
944
- "unk_token": "[UNK]"
945
- }
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
query_0_SentenceTransformer/README.md DELETED
@@ -1,147 +0,0 @@
1
- ---
2
- tags:
3
- - sentence-transformers
4
- - sentence-similarity
5
- - feature-extraction
6
- - dense
7
- - generated_from_trainer
8
- base_model: Alibaba-NLP/gte-modernbert-base
9
- pipeline_tag: sentence-similarity
10
- library_name: sentence-transformers
11
- ---
12
-
13
- # SentenceTransformer based on Alibaba-NLP/gte-modernbert-base
14
-
15
- This is a [sentence-transformers](https://www.SBERT.net) model finetuned from [Alibaba-NLP/gte-modernbert-base](https://huggingface.co/Alibaba-NLP/gte-modernbert-base). It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.
16
-
17
- ## Model Details
18
-
19
- ### Model Description
20
- - **Model Type:** Sentence Transformer
21
- - **Base model:** [Alibaba-NLP/gte-modernbert-base](https://huggingface.co/Alibaba-NLP/gte-modernbert-base)
22
- - **Maximum Sequence Length:** inf tokens
23
- - **Output Dimensionality:** 768 dimensions
24
- - **Similarity Function:** Cosine Similarity
25
- <!-- - **Training Dataset:** Unknown -->
26
- <!-- - **Language:** Unknown -->
27
- <!-- - **License:** Unknown -->
28
-
29
- ### Model Sources
30
-
31
- - **Documentation:** [Sentence Transformers Documentation](https://sbert.net)
32
- - **Repository:** [Sentence Transformers on GitHub](https://github.com/UKPLab/sentence-transformers)
33
- - **Hugging Face:** [Sentence Transformers on Hugging Face](https://huggingface.co/models?library=sentence-transformers)
34
-
35
- ### Full Model Architecture
36
-
37
- ```
38
- SentenceTransformer(
39
- (0): StaticEmbedding(
40
- (embedding): EmbeddingBag(100003, 768, mode='mean')
41
- )
42
- (1): Normalize()
43
- )
44
- ```
45
-
46
- ## Usage
47
-
48
- ### Direct Usage (Sentence Transformers)
49
-
50
- First install the Sentence Transformers library:
51
-
52
- ```bash
53
- pip install -U sentence-transformers
54
- ```
55
-
56
- Then you can load this model and run inference.
57
- ```python
58
- from sentence_transformers import SentenceTransformer
59
-
60
- # Download from the 🤗 Hub
61
- model = SentenceTransformer("stephantulkens/NIFE-gte-modernbert-base")
62
- # Run inference
63
- sentences = [
64
- 'The weather is lovely today.',
65
- "It's so sunny outside!",
66
- 'He drove to the stadium.',
67
- ]
68
- embeddings = model.encode(sentences)
69
- print(embeddings.shape)
70
- # [3, 768]
71
-
72
- # Get the similarity scores for the embeddings
73
- similarities = model.similarity(embeddings, embeddings)
74
- print(similarities)
75
- # tensor([[1.0000, 0.4225, 0.2490],
76
- # [0.4225, 1.0000, 0.3060],
77
- # [0.2490, 0.3060, 1.0000]])
78
- ```
79
-
80
- <!--
81
- ### Direct Usage (Transformers)
82
-
83
- <details><summary>Click to see the direct usage in Transformers</summary>
84
-
85
- </details>
86
- -->
87
-
88
- <!--
89
- ### Downstream Usage (Sentence Transformers)
90
-
91
- You can finetune this model on your own dataset.
92
-
93
- <details><summary>Click to expand</summary>
94
-
95
- </details>
96
- -->
97
-
98
- <!--
99
- ### Out-of-Scope Use
100
-
101
- *List how the model may foreseeably be misused and address what users ought not to do with the model.*
102
- -->
103
-
104
- <!--
105
- ## Bias, Risks and Limitations
106
-
107
- *What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.*
108
- -->
109
-
110
- <!--
111
- ### Recommendations
112
-
113
- *What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.*
114
- -->
115
-
116
- ## Training Details
117
-
118
- ### Framework Versions
119
- - Python: 3.12.8
120
- - Sentence Transformers: 5.1.1
121
- - Transformers: 4.56.2
122
- - PyTorch: 2.8.0
123
- - Accelerate: 1.10.1
124
- - Datasets: 4.1.1
125
- - Tokenizers: 0.22.1
126
-
127
- ## Citation
128
-
129
- ### BibTeX
130
-
131
- <!--
132
- ## Glossary
133
-
134
- *Clearly define terms in order to be accessible across audiences.*
135
- -->
136
-
137
- <!--
138
- ## Model Card Authors
139
-
140
- *Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.*
141
- -->
142
-
143
- <!--
144
- ## Model Card Contact
145
-
146
- *Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.*
147
- -->
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
query_0_SentenceTransformer/config_sentence_transformers.json DELETED
@@ -1,14 +0,0 @@
1
- {
2
- "model_type": "SentenceTransformer",
3
- "__version__": {
4
- "sentence_transformers": "5.1.1",
5
- "transformers": "4.56.2",
6
- "pytorch": "2.8.0"
7
- },
8
- "prompts": {
9
- "query": "",
10
- "document": ""
11
- },
12
- "default_prompt_name": null,
13
- "similarity_fn_name": "cosine"
14
- }
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
query_0_SentenceTransformer/model.safetensors DELETED
@@ -1,3 +0,0 @@
1
- version https://git-lfs.github.com/spec/v1
2
- oid sha256:bd3aabaef0433c9b51e538dfdbbc8da720dbe79c96650beae4ce0f68001ac81f
3
- size 307209312
 
 
 
 
query_0_SentenceTransformer/modules.json DELETED
@@ -1,14 +0,0 @@
1
- [
2
- {
3
- "idx": 0,
4
- "name": "0",
5
- "path": "",
6
- "type": "sentence_transformers.models.StaticEmbedding"
7
- },
8
- {
9
- "idx": 1,
10
- "name": "1",
11
- "path": "1_Normalize",
12
- "type": "sentence_transformers.models.Normalize"
13
- }
14
- ]
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
query_0_SentenceTransformer/tokenizer.json DELETED
The diff for this file is too large to render. See raw diff
 
router_config.json DELETED
@@ -1,18 +0,0 @@
1
- {
2
- "types": {
3
- "query_0_SentenceTransformer": "sentence_transformers.SentenceTransformer.SentenceTransformer",
4
- "document_0_SentenceTransformer": "sentence_transformers.SentenceTransformer.SentenceTransformer"
5
- },
6
- "structure": {
7
- "query": [
8
- "query_0_SentenceTransformer"
9
- ],
10
- "document": [
11
- "document_0_SentenceTransformer"
12
- ]
13
- },
14
- "parameters": {
15
- "default_route": "document",
16
- "allow_empty_key": true
17
- }
18
- }