DanielSc4 commited on
Commit
c57e387
·
verified ·
1 Parent(s): 986a6ea

Upload xlmr-large-classifier-pinocchio_it_tra2-eng model files

Browse files
.gitattributes CHANGED
@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
 
 
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ tokenizer.json filter=lfs diff=lfs merge=lfs -text
README.md ADDED
@@ -0,0 +1,56 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ language:
3
+ - eng
4
+ license: apache-2.0
5
+ tags:
6
+ - text-classification
7
+ pipeline_tag: text-classification
8
+ ---
9
+
10
+
11
+ # xlmr-large-classifier-pinocchio_it_tra2-eng - MT/HT Classifier
12
+
13
+ This model is a fine-tuned version of [`FacebookAI/xlm-roberta-large`](https://huggingface.co/FacebookAI/xlm-roberta-large) for distinguishing between Machine Translated (MT) and Human Translated (HT) text
14
+ (or HT1 and HT2 if using two different human translators).
15
+
16
+
17
+
18
+ Training data:
19
+ * Train: 1490, for each label: 745
20
+ * Validation: 164, for each label: 82
21
+ * Test: 214, for each label: 107
22
+
23
+
24
+ Results on the held-out test set:
25
+ * Accuracy: 0.5093
26
+ * F1-Score: 0.3787
27
+ * Precision: 0.5161
28
+ * Recall: 0.2991
29
+
30
+ ## label mapping
31
+ Label MT: 0
32
+
33
+ Label PE: 1 (this is the human translator)
34
+
35
+ ## Info
36
+ Upload date: 2025-04-24 00:00
37
+
38
+ ## Usage
39
+ ```python
40
+ from transformers import AutoTokenizer, AutoModelForSequenceClassification
41
+
42
+ tokenizer = AutoTokenizer.from_pretrained("DanielSc4/xlmr-large-classifier-pinocchio_it_tra2-eng")
43
+ model = AutoModelForSequenceClassification.from_pretrained("DanielSc4/xlmr-large-classifier-pinocchio_it_tra2-eng")
44
+
45
+ device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
46
+ inp = tokenizer('This is a test', return_tensors='pt').to(device)
47
+ model = model.to(device)
48
+
49
+ out = model(**inp)
50
+
51
+ logits = out.logits
52
+ probs = logits.softmax(dim=-1)
53
+ pred = probs.argmax(dim=-1).item()
54
+ print("Predicted class: " + str(pred)) # 0 for MT, 1 for PE
55
+ ```
56
+
config.json ADDED
@@ -0,0 +1,28 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "architectures": [
3
+ "XLMRobertaForSequenceClassification"
4
+ ],
5
+ "attention_probs_dropout_prob": 0.1,
6
+ "bos_token_id": 0,
7
+ "classifier_dropout": null,
8
+ "eos_token_id": 2,
9
+ "hidden_act": "gelu",
10
+ "hidden_dropout_prob": 0.1,
11
+ "hidden_size": 1024,
12
+ "initializer_range": 0.02,
13
+ "intermediate_size": 4096,
14
+ "layer_norm_eps": 1e-05,
15
+ "max_position_embeddings": 514,
16
+ "model_type": "xlm-roberta",
17
+ "num_attention_heads": 16,
18
+ "num_hidden_layers": 24,
19
+ "output_past": true,
20
+ "pad_token_id": 1,
21
+ "position_embedding_type": "absolute",
22
+ "problem_type": "single_label_classification",
23
+ "torch_dtype": "float32",
24
+ "transformers_version": "4.50.1",
25
+ "type_vocab_size": 1,
26
+ "use_cache": true,
27
+ "vocab_size": 250002
28
+ }
model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:57c3c261cc3f88d8b3fe5631a629198b0752d93f8939218bf9ea64ed4608ff95
3
+ size 2239618672
sentencepiece.bpe.model ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:cfc8146abe2a0488e9e2a0c56de7952f7c11ab059eca145a0a727afce0db2865
3
+ size 5069051
special_tokens_map.json ADDED
@@ -0,0 +1,15 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "bos_token": "<s>",
3
+ "cls_token": "<s>",
4
+ "eos_token": "</s>",
5
+ "mask_token": {
6
+ "content": "<mask>",
7
+ "lstrip": true,
8
+ "normalized": false,
9
+ "rstrip": false,
10
+ "single_word": false
11
+ },
12
+ "pad_token": "<pad>",
13
+ "sep_token": "</s>",
14
+ "unk_token": "<unk>"
15
+ }
test.json ADDED
@@ -0,0 +1,858 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ [
2
+ {
3
+ "text": "Then the judge, referring Pinocchio to the giandarmes, said to them:",
4
+ "label": 0
5
+ },
6
+ {
7
+ "text": "The judge then, pointing to Pinocchio, said to them:",
8
+ "label": 1
9
+ },
10
+ {
11
+ "text": "\"If the others get out of prison, I want to get out too,\" Pinocchio said to the jailer.",
12
+ "label": 0
13
+ },
14
+ {
15
+ "text": "\"If the others are to be let out of prison, I will go also,\" said Pinocchio to the jailor.",
16
+ "label": 1
17
+ },
18
+ {
19
+ "text": "\"If all cats looked like you, lucky the mice! ...",
20
+ "label": 0
21
+ },
22
+ {
23
+ "text": "\"If all cats resembled you, how fortunate the mice would be!\"",
24
+ "label": 1
25
+ },
26
+ {
27
+ "text": "To all these questions, asked hastily and without catching his breath, the Snail replied with his usual phlegm:",
28
+ "label": 0
29
+ },
30
+ {
31
+ "text": "To all these rapid, breathless questions the Snail replied in her usual phlegmatic manner:",
32
+ "label": 1
33
+ },
34
+ {
35
+ "text": "\"So much the worse for you! repeated the Cat.",
36
+ "label": 0
37
+ },
38
+ {
39
+ "text": "\"So much the worse for you!\" repeated the Cat.",
40
+ "label": 1
41
+ },
42
+ {
43
+ "text": "Chapter VII",
44
+ "label": 0
45
+ },
46
+ {
47
+ "text": "CHAPTER VII",
48
+ "label": 1
49
+ },
50
+ {
51
+ "text": "\"Are not you ashamed? Instead of hanging out on the street, go find yourself some work, and learn how to earn a living!\"",
52
+ "label": 0
53
+ },
54
+ {
55
+ "text": "\"Are you not ashamed to beg? Instead of idling about the roads, go and look for a little work and learn to earn your bread.\"",
56
+ "label": 1
57
+ },
58
+ {
59
+ "text": "\"O beautiful little girl with blue hair, \"cried Pinocchio \"open me for charity. Have pity on a poor boy chased by assassins ...\"",
60
+ "label": 0
61
+ },
62
+ {
63
+ "text": "\"Oh! beautiful Child with blue hair,\" cried Pinocchio, \"open the door for pity's sake! Have compassion on a poor boy pursued by assas .",
64
+ "label": 1
65
+ },
66
+ {
67
+ "text": "\"Is it sweet or bitter?",
68
+ "label": 0
69
+ },
70
+ {
71
+ "text": "\"Is it sweet or bitter?\"",
72
+ "label": 1
73
+ },
74
+ {
75
+ "text": "Father and son immediately turned towards the ceiling, and saw the Talking Cricket over a rafter.",
76
+ "label": 0
77
+ },
78
+ {
79
+ "text": "The father and son looked immediately up to the ceiling, and there on a beam they saw the Talking-cricket.",
80
+ "label": 1
81
+ },
82
+ {
83
+ "text": "Pinocchio was very afraid of thunder and lightning: if not that hunger was stronger than fear: which is why he approached the door of the house, and took his career, in a hundred leaps he reached the town, with his tongue out. and out of breath, like a hunting dog.",
84
+ "label": 0
85
+ },
86
+ {
87
+ "text": "Pinocchio had a great fear of thunder, but hunger was stronger than fear. He therefore closed the house door and made a rush for the village, which he reached in a hundred bounds, with his tongue hanging out and panting for breath, like a dog after game.",
88
+ "label": 1
89
+ },
90
+ {
91
+ "text": "Seeing that the door would not open, the Little Man threw it open with a violent kick: and entering the room, he said with his usual little laugh to Pinocchio and Candlewick:",
92
+ "label": 0
93
+ },
94
+ {
95
+ "text": "FINDING that the door remained shut the little man burst it open with a violent kick, and coming into the room he said to Pinocchio and Candlewick with his usual little laugh:",
96
+ "label": 1
97
+ },
98
+ {
99
+ "text": "\"Do you want to double your gold coins?",
100
+ "label": 0
101
+ },
102
+ {
103
+ "text": "\"Would you like to double your money?\"",
104
+ "label": 1
105
+ },
106
+ {
107
+ "text": "\"I'm going to live in a country ... which is the most beautiful country in this world: a real bonanza! ...",
108
+ "label": 0
109
+ },
110
+ {
111
+ "text": "\"I am going to live in a country . . . the most delightful country in the world: a real land of Cocagne! . . .\"",
112
+ "label": 1
113
+ },
114
+ {
115
+ "text": "And they left.",
116
+ "label": 0
117
+ },
118
+ {
119
+ "text": "And they walked off.",
120
+ "label": 1
121
+ },
122
+ {
123
+ "text": "Having said that, he slipped down the side street and began to run on his heels. The more he ran, the more distinct he heard the sound of the fifes and the thuds of the bass drum: pì-pì-pì, pì-pì-pì, pì-pì-pì, zum, zum, zum, zum.",
124
+ "label": 0
125
+ },
126
+ {
127
+ "text": "\"To-day I will go and hear the fifes, and tomorrow I will go to school,\" finally decided the young scapegrace, shrugging his shoulders. The more he ran the nearer came the sounds of the fifes and the beating of the big drum: fi-fi-fi, zum, zum, zum, zum.",
128
+ "label": 1
129
+ },
130
+ {
131
+ "text": "\"What would that be?",
132
+ "label": 0
133
+ },
134
+ {
135
+ "text": "\"And in what did it consist?\"",
136
+ "label": 1
137
+ },
138
+ {
139
+ "text": "Later in the evening he practiced reading and writing. He had bought a large book in the nearby town for a few cents, which lacked the title page and the index, and with that he was reading. As for writing, he used a tempered pen to use; and having neither inkwell nor ink, he dipped it in a small bottle filled with blackberry and cherry juice.",
140
+ "label": 0
141
+ },
142
+ {
143
+ "text": "By his industry, ingenuity, and his anxiety to work and to overcome difficulties, he not only succeeded in maintaining his father, who continued infirm, in comfort, but he also contrived to put aside forty pence to buy himself a new coat.",
144
+ "label": 1
145
+ },
146
+ {
147
+ "text": "\"I didn't throw it at you!",
148
+ "label": 0
149
+ },
150
+ {
151
+ "text": "\"I did not hit you with it! . . .\"",
152
+ "label": 1
153
+ },
154
+ {
155
+ "text": "In fact they went, and knocked on the door.",
156
+ "label": 0
157
+ },
158
+ {
159
+ "text": "They went and knocked.",
160
+ "label": 1
161
+ },
162
+ {
163
+ "text": "The most singular thing was this: that the wool of the little goat, instead of being white, or black, or palled of several colors, like that of the other goats, was instead all blue, but of such a dazzling blue, that it was very reminiscent of hair of the beautiful little girl.",
164
+ "label": 0
165
+ },
166
+ {
167
+ "text": "But the most singular thing was this. The little goat's hair, instead of being white or black, or a mixture of two colours as is usual with other goats, was blue, and of a very vivid blue, greatly resembling the hair of the beautiful Child.",
168
+ "label": 1
169
+ },
170
+ {
171
+ "text": "\"And I never obey.",
172
+ "label": 0
173
+ },
174
+ {
175
+ "text": ". . .\" \"And I never obey.\"",
176
+ "label": 1
177
+ },
178
+ {
179
+ "text": "At these last words, Pinocchio jumped up in a rage and took a wooden hammer from the counter, threw it at the Talking Cricket.",
180
+ "label": 0
181
+ },
182
+ {
183
+ "text": "At these last words Pinocchio jumped up in a rage, and snatching a wooden hammer from the bench he threw it at the Talking-cricket. SNATCHED UP A WOODEN HAMMER.",
184
+ "label": 1
185
+ },
186
+ {
187
+ "text": "As soon as he got back to the use of the word, he began to say, trembling and stammering with fright:",
188
+ "label": 0
189
+ },
190
+ {
191
+ "text": "As soon as he had recovered the use of his speech, he began to say, stuttering and trembling with fear:",
192
+ "label": 1
193
+ },
194
+ {
195
+ "text": "The Marmottina raised her right paw in front: and after having felt Pinocchio's pulse, she said to him with a sigh:",
196
+ "label": 0
197
+ },
198
+ {
199
+ "text": "The little Marmot raised her right fore-paw; and after having felt Pinocchio's pulse she said to him, sighing:",
200
+ "label": 1
201
+ },
202
+ {
203
+ "text": "\"I'm waiting for the coffin to come and take me away.\"",
204
+ "label": 0
205
+ },
206
+ {
207
+ "text": "\"I am waiting for the bier to come to carry me away.\"",
208
+ "label": 1
209
+ },
210
+ {
211
+ "text": "\"We are a poor father and a poor son, without bread and without a roof,\" replied the puppet.",
212
+ "label": 0
213
+ },
214
+ {
215
+ "text": "\"We are a poor father and son without bread and without a roof,\" answered the puppet.",
216
+ "label": 1
217
+ },
218
+ {
219
+ "text": "That sea monster was neither more nor less that gigantic Dog-fish mentioned several times in this story, and which for its massacres and for its insatiable voracity, was nicknamed «the Attila of fish and fishermen».",
220
+ "label": 0
221
+ },
222
+ {
223
+ "text": "This sea-monster was neither more nor less than that gigantic Dog-fish who has been mentioned many times in this story, and who, for his slaughter and for his insatiable voracity, had been named the \"Attila of fish and fishermen.\"",
224
+ "label": 1
225
+ },
226
+ {
227
+ "text": "\"So what must I do to please you?",
228
+ "label": 0
229
+ },
230
+ {
231
+ "text": "\"Then what must I do to please you?\"",
232
+ "label": 1
233
+ },
234
+ {
235
+ "text": "\"Tell me and hurry up.",
236
+ "label": 0
237
+ },
238
+ {
239
+ "text": "\"Tell it me, then, and be quick about it.\"",
240
+ "label": 1
241
+ },
242
+ {
243
+ "text": "In the meantime the fighting became more and more ferocious, when suddenly a large Crab, which had come out of the water and had slowly climbed up to the beach, shouted with a cold trombone voice:",
244
+ "label": 0
245
+ },
246
+ {
247
+ "text": "The battle meantime had become fiercer than ever, when a big crab, who had come out of the water and had climbed slowly up on to the shore, called out in a hoarse voice that sounded like a trumpet with a bad cold:",
248
+ "label": 1
249
+ },
250
+ {
251
+ "text": "\"I have thought of making myself a beautiful wooden puppet: but a marvelous puppet, who can dance, fence and jump through hoops. With this puppet I want to travel the world, to get a loaf of bread and a glass of wine: what do you think?",
252
+ "label": 0
253
+ },
254
+ {
255
+ "text": "\"I thought I would make a beautiful wooden puppet; but a wonderful puppet that should know how to dance, to fence, and to leap like an acrobat. With this puppet I would travel about the world to earn a piece of bread and a glass of wine. What do you think of it?\"",
256
+ "label": 1
257
+ },
258
+ {
259
+ "text": "Then he resumed in his usual little voice:",
260
+ "label": 0
261
+ },
262
+ {
263
+ "text": "He began again in the same soft voice:",
264
+ "label": 1
265
+ },
266
+ {
267
+ "text": "\"I am sorry \"said the Owl \"of having to contradict the Raven, my illustrious friend and colleague: for me, however, the puppet is always alive; but if unfortunately he was not alive, then it would be a sign that he is really dead.",
268
+ "label": 0
269
+ },
270
+ {
271
+ "text": "\"I regret,\" said the Owl, \"to be obliged to contradict the Crow, my illustrious friend and colleague; but in my opinion the puppet is still alive: but if unfortunately he should not be alive, then it would be a sign that he is dead indeed!\"",
272
+ "label": 1
273
+ },
274
+ {
275
+ "text": "\"Do not you see it? I cry! said Pinocchio, raising his head towards that voice and rubbing his eyes with the sleeve of his jacket.",
276
+ "label": 0
277
+ },
278
+ {
279
+ "text": "\"Don't you see? I am crying!\" said Pinocchio, raising his head towards the voice and rubbing his eyes with his jacket.",
280
+ "label": 1
281
+ },
282
+ {
283
+ "text": "\"That is?",
284
+ "label": 0
285
+ },
286
+ {
287
+ "text": "\"In what way?\"",
288
+ "label": 1
289
+ },
290
+ {
291
+ "text": "\"Eat me? but do you want to understand that I am not a fish? Or does he not hear that I speak, and do I think like you?",
292
+ "label": 0
293
+ },
294
+ {
295
+ "text": "\"Eat me! but will you understand that I am not a fish? Do you hear that I talk and reason as you do?\"",
296
+ "label": 1
297
+ },
298
+ {
299
+ "text": "The poor puppet, beside himself with great contentment, thanked the Fox and the Cat a thousand times, and promised them a beautiful gift.",
300
+ "label": 0
301
+ },
302
+ {
303
+ "text": "The poor puppet, beside himself with joy, thanked the Fox and the Cat a thousand times, and promised them a beautiful present.",
304
+ "label": 1
305
+ },
306
+ {
307
+ "text": "Pinocchio got on: and the chariot began to move: but while the donkeys were galloping and the chariot ran over the cobblestones of the main road, the puppet thought he heard a soft and barely intelligible voice, which said to him:",
308
+ "label": 0
309
+ },
310
+ {
311
+ "text": "Pinocchio mounted, and the coach started. Whilst the donkeys were galloping and the coach was rattling over the stones of the high road, the puppet thought that he heard a low voice that was scarcely intelligible saying to him:",
312
+ "label": 1
313
+ },
314
+ {
315
+ "text": "\"He seems to me a great good son, full of desire to study, obedient, fond of his father and his family ...\"",
316
+ "label": 0
317
+ },
318
+ {
319
+ "text": "\"He seems to me to be a very good boy, anxious to learn, and obedient and affectionate to his father and family. . . .\"",
320
+ "label": 1
321
+ },
322
+ {
323
+ "text": "Chapter XVI",
324
+ "label": 0
325
+ },
326
+ {
327
+ "text": "CHAPTER XVI",
328
+ "label": 1
329
+ },
330
+ {
331
+ "text": "\"Why are they lost? Give me your hand, little father, and be careful not to slip! ...",
332
+ "label": 0
333
+ },
334
+ {
335
+ "text": "Follow me, dear papa, and in a short time we shall be in safety.\"",
336
+ "label": 1
337
+ },
338
+ {
339
+ "text": "\"You will say well,\" added Pinocchio, \"but I will never eat a fruit that is not peeled.\" I can't suffer the skins.\"",
340
+ "label": 0
341
+ },
342
+ {
343
+ "text": "\"You are no doubt right,\" interrupted Pinocchio, \"but I will never eat fruit that has not been peeled. I cannot bear rind.\"",
344
+ "label": 1
345
+ },
346
+ {
347
+ "text": "\"Save me, Alidoro! If you don't save me, I'm fried! ...\"",
348
+ "label": 0
349
+ },
350
+ {
351
+ "text": "\"Save me, Alidoro! If you do not save me I shall be fried! . . .\"",
352
+ "label": 1
353
+ },
354
+ {
355
+ "text": "\"I laugh, because I tickled myself under my wings when I spollinarmi.\"",
356
+ "label": 0
357
+ },
358
+ {
359
+ "text": "\"I am laughing because in pruning my feathers I tickled myself under my wings.\"",
360
+ "label": 1
361
+ },
362
+ {
363
+ "text": "Encouraged by this first victory, he forcibly freed himself from the killers' fingernails, and having jumped over the hedge of the road, he began to flee across the countryside. And the murderers running after him, like two dogs after a hare: and the one who had lost a leg ran with one leg, and we never knew how he did it.",
364
+ "label": 0
365
+ },
366
+ {
367
+ "text": "Encouraged by this first victory he used his nails to such purpose that he succeeded in liberating himself from his assailants, and jumping the hedge by the roadside he began to fly across country. The assassins ran after him like two dogs chasing a hare: and the one who had lost a paw ran on one leg, and no one ever knew how he managed it.",
368
+ "label": 1
369
+ },
370
+ {
371
+ "text": "Then the dog who, when he was really hungry, was not accustomed to letting flies land on his nose, snarled at the fisherman, showing him his terrible fangs.",
372
+ "label": 0
373
+ },
374
+ {
375
+ "text": "But the dog, who, when he was really hungry, would not stand trifling, turned upon him, growling and showing his terrible tusks.",
376
+ "label": 1
377
+ },
378
+ {
379
+ "text": "Then Geppetto said to the puppet:",
380
+ "label": 0
381
+ },
382
+ {
383
+ "text": "Geppetto then said to the puppet:",
384
+ "label": 1
385
+ },
386
+ {
387
+ "text": "\"Tell me, Grillo, and who are you?",
388
+ "label": 0
389
+ },
390
+ {
391
+ "text": "\"Tell me, Cricket, who may you be?\"",
392
+ "label": 1
393
+ },
394
+ {
395
+ "text": "After the nose made his mouth.",
396
+ "label": 0
397
+ },
398
+ {
399
+ "text": "The mouth was not even completed when it began to laugh and deride him.",
400
+ "label": 1
401
+ },
402
+ {
403
+ "text": "Meanwhile the stable was closed and Pinocchio was left alone: and because he had not eaten for many hours, he began to yawn with great appetite. And, yawning, he opened a mouth that looked like an oven.",
404
+ "label": 0
405
+ },
406
+ {
407
+ "text": "The stable was then shut and Pinocchio was left alone. He had not eaten for many hours, and he began to yawn from hunger. And when he yawned he opened a mouth that seemed as wide as an oven.",
408
+ "label": 1
409
+ },
410
+ {
411
+ "text": "Then Pinocchio, offering his arm to Geppetto, who barely had breath to stand up, said to him:",
412
+ "label": 0
413
+ },
414
+ {
415
+ "text": "Pinocchio then offering his arm to Geppetto, who had scarcely breath to stand, said to him:",
416
+ "label": 1
417
+ },
418
+ {
419
+ "text": "\"Bring me that puppet here, which you will find attached to the nail. It looks to me like a puppet made of very dry wood, and I'm sure that if you throw it on the fire, it will give me a beautiful flame to the roast.\"",
420
+ "label": 0
421
+ },
422
+ {
423
+ "text": "\"Bring that puppet here: you will find him hanging on a nail. It seems to me that he is made of very dry wood, and I am sure that if he was thrown on the fire he would make a beautiful blaze for the roast.\"",
424
+ "label": 1
425
+ },
426
+ {
427
+ "text": "Among these books, there was a volume bound in thick cardboard, with the spine and points of parchment. It was a Treatise on Arithmetic. I let you imagine if it was a lot of weight!",
428
+ "label": 0
429
+ },
430
+ {
431
+ "text": "Amongst the books there was one bound in strong cardboard with the back and points of parchment. It was a Treatise on Arithmetic. I leave you to imagine if it was big or not!",
432
+ "label": 1
433
+ },
434
+ {
435
+ "text": "\"What else is bothering you?",
436
+ "label": 0
437
+ },
438
+ {
439
+ "text": "\"What is the matter now?\"",
440
+ "label": 1
441
+ },
442
+ {
443
+ "text": "\"No: he is now alive, and has already returned to his home.",
444
+ "label": 0
445
+ },
446
+ {
447
+ "text": "\"No, he is alive, and has returned to his home.\"",
448
+ "label": 1
449
+ },
450
+ {
451
+ "text": "\"Hunger, my boy, is not a good reason to be able to appropriate stuff that is not ours ...",
452
+ "label": 0
453
+ },
454
+ {
455
+ "text": "\"Hunger, my boy, is not a good reason for appropriating what does not belong to us. . . .\"",
456
+ "label": 1
457
+ },
458
+ {
459
+ "text": "He wanted to cry, he wanted to despair, he wanted to throw away the tray and what was inside; but instead, whether it was the great pain or the great languidity of the stomach, the fact is that he fell unconscious.",
460
+ "label": 0
461
+ },
462
+ {
463
+ "text": "He wanted to cry. In his desperation he tried to throw away the tray and all that was on it; but instead, either from grief or exhaustion, he fainted away.",
464
+ "label": 1
465
+ },
466
+ {
467
+ "text": "\"Where will we meet again? ...",
468
+ "label": 0
469
+ },
470
+ {
471
+ "text": "\"Where shall we meet again? . . ..\"",
472
+ "label": 1
473
+ },
474
+ {
475
+ "text": "\"How far is the Field of Miracles from here?",
476
+ "label": 0
477
+ },
478
+ {
479
+ "text": "\"How far off is the Field of miracles?\"",
480
+ "label": 1
481
+ },
482
+ {
483
+ "text": "\"Do not give yourselves to despair, master. There are many donkeys in this world!",
484
+ "label": 0
485
+ },
486
+ {
487
+ "text": "\"Don't despair, master. There are such a number of little donkeys in the world!\"",
488
+ "label": 1
489
+ },
490
+ {
491
+ "text": "Finally bored of waiting, they turned to Pinocchio and laughed at him:",
492
+ "label": 0
493
+ },
494
+ {
495
+ "text": "Losing patience they turned to Pinocchio and said in a bantering tone:",
496
+ "label": 1
497
+ },
498
+ {
499
+ "text": "And he could already imagine that they had been drowned, when instead, turning to look, he noticed that both of them were running after him, still wrapped up in their sacks, and dripping water like two broken baskets.",
500
+ "label": 0
501
+ },
502
+ {
503
+ "text": "And he felt convinced that they were drowned, when, turning to look, he perceived that on the contrary they were both running after him, still enveloped in their sacks, with the water dripping from them as if they had been two hollow baskets.",
504
+ "label": 1
505
+ },
506
+ {
507
+ "text": "When he recovered, he found himself stretched out on a sofa, and the Fairy was beside him.",
508
+ "label": 0
509
+ },
510
+ {
511
+ "text": "When he came to himself he found that he was lying on a sofa, and the Fairy was beside him.",
512
+ "label": 1
513
+ },
514
+ {
515
+ "text": "\"I'll tell you, my dear little readers: the surprise was that when Pinocchio woke up, he was naturally made to scratch his head; and in scratching his head he noticed ...",
516
+ "label": 0
517
+ },
518
+ {
519
+ "text": "I will tell you, my dear little readers. The surprise was that Pinocchio when he awoke scratched his head; and in scratching his head he discovered.",
520
+ "label": 1
521
+ },
522
+ {
523
+ "text": "Gradually his eyes clouded over; and although he felt death approaching, he still hoped that at any moment some pitiful soul would come to help him. But when, wait wait, he saw that no one was appearing, absolutely no one, then his poor father came to mind ... and he stammered almost dying:",
524
+ "label": 0
525
+ },
526
+ {
527
+ "text": "Little by little his eyes began to grow dim, but although he felt that death was near he still continued to hope that some charitable person would come to his assistance before it was too late. But when, after waiting and waiting, he found that no one came, absolutely no one, then he remembered his poor father, and thinking he was dying .",
528
+ "label": 1
529
+ },
530
+ {
531
+ "text": "\"Both. And you?",
532
+ "label": 0
533
+ },
534
+ {
535
+ "text": "\"Both of them. And you?\"",
536
+ "label": 1
537
+ },
538
+ {
539
+ "text": "\"Wait there, I'll be back right away, \"replied the old man, thinking he had something to do with one of those nagging bad boys who enjoy ringing house bells at night, to harass decent people, who sleep peacefully.",
540
+ "label": 0
541
+ },
542
+ {
543
+ "text": "\"Wait there, I will be back directly,\" said the little old man, thinking he had to do with one of those rascally boys who amuse themselves at night by ringing the house bells to rouse respectable people who are sleeping quietly.",
544
+ "label": 1
545
+ },
546
+ {
547
+ "text": "\"Patience! I'll explain better \"added the Parrot. \"Know then that, while you were in the city, the Fox and the Cat returned to this field: they took the buried gold coins, and then they fled like the wind. And now whoever reaches them is good!\"",
548
+ "label": 0
549
+ },
550
+ {
551
+ "text": "\"Have patience! I will explain myself better,\" rejoined the Parrot. \"You must know, then, that whilst you were in the town the Fox and the Cat returned to the field: they took the buried money and then fled like the wind. And now he that catches them will be clever.\"",
552
+ "label": 1
553
+ },
554
+ {
555
+ "text": "Then he took him by the scruff, and, as he led him back, said to him, shaking his head menacingly:",
556
+ "label": 0
557
+ },
558
+ {
559
+ "text": "He then took him by the collar, and as he was leading him away he said to him, shaking his head threateningly:",
560
+ "label": 1
561
+ },
562
+ {
563
+ "text": "\"Why not? But first I want to see yours, dear Pinocchio.",
564
+ "label": 0
565
+ },
566
+ {
567
+ "text": "\"Why not? But first, my dear Pinocchio, I should like to see yours.\"",
568
+ "label": 1
569
+ },
570
+ {
571
+ "text": "\"I'm Lu ... ci ... gno ... lo ...\"",
572
+ "label": 0
573
+ },
574
+ {
575
+ "text": "\"I am . . . Can . . . dle . . . wick . . .\"",
576
+ "label": 1
577
+ },
578
+ {
579
+ "text": "\"I want to move on.",
580
+ "label": 0
581
+ },
582
+ {
583
+ "text": "\"I am determined to go on.\"",
584
+ "label": 1
585
+ },
586
+ {
587
+ "text": "Pinocchio, with that boulder around his neck, immediately went to the bottom: and the buyer, always holding the rope tightly in his hand, sat down on a rock, waiting for the donkey to have plenty of time to drown, to then skin it and remove it. Skin.",
588
+ "label": 0
589
+ },
590
+ {
591
+ "text": "Pinocchio, weighed down by the stone, went at once to the bottom; and his owner, keeping tight hold of the cord, sat down quietly on a piece of rock to wait until the little donkey was drowned, intending then to skin him.",
592
+ "label": 1
593
+ },
594
+ {
595
+ "text": "\"Because that pillow that I have over there on my feet bothers me.\"",
596
+ "label": 0
597
+ },
598
+ {
599
+ "text": "\"Because that pillow that is down there on my feet bothers me.\"",
600
+ "label": 1
601
+ },
602
+ {
603
+ "text": "\"Not a single penny,\" replied the charcoal burner, \"but I'll give you four, as long as you help me haul these two coal carts home.\"",
604
+ "label": 0
605
+ },
606
+ {
607
+ "text": "\"You shall have not only a halfpenny,\" said the man, \"but I will give you twopence, provided that you help me to drag home these two carts of charcoal.\"",
608
+ "label": 1
609
+ },
610
+ {
611
+ "text": "Pinocchio discovers the thieves, and as a reward for having been faithful he is set free.",
612
+ "label": 0
613
+ },
614
+ {
615
+ "text": "Pinocchio Discovers the Robbers Pinocchio discovers the robbers, and as a reward for his fidelity is set at liberty.",
616
+ "label": 1
617
+ },
618
+ {
619
+ "text": "Pinocchio, at this antiphon, threw himself on the ground, and did not want to walk any more. Meanwhile, the curious and loitering began to stop around there and make a small group.",
620
+ "label": 0
621
+ },
622
+ {
623
+ "text": "At this announcement Pinocchio threw himself on the ground and would not take another step. In the meanwhile a crowd of idlers and inquisitive people began to assemble and to make a ring round them.",
624
+ "label": 1
625
+ },
626
+ {
627
+ "text": "At this third lie, his nose lengthened in such an extraordinary way that poor Pinocchio could no longer turn around. If he turned from here, he tapped his nose on the bed or in the window panes, if he turned away, he hit it on the walls or on the bedroom door, if he raised his head a little more, he ran the risk of sticking it in the eye to the Fairy.",
628
+ "label": 0
629
+ },
630
+ {
631
+ "text": "At this third lie his nose grew to such an extraordinary length that poor Pinocchio could not move in any direction. If he turned to one side he struck his nose against the bed or the window-panes, if he turned to the other he struck it against the walls or the door, if he raised his head a little he ran the risk of sticking it into one of the Fairy's eyes.",
632
+ "label": 1
633
+ },
634
+ {
635
+ "text": "The first to dance in the boiling oil were the poor hake: then it was the turn of the ragnotti, then the mullets, then the sole and anchovies, and then it was the turn of Pinocchio. Who, seeing himself so close to death (and what an ugly death!) Was seized by so much tremor and so much fear, that he no longer had neither voice nor breath to recommend himself.",
636
+ "label": 0
637
+ },
638
+ {
639
+ "text": "The first to dance in the boiling oil were the poor whiting; the crabs followed, then the sardines, then the soles, then the anchovies, and at last it was Pinocchio's turn. Seeing himself so near death, and such a horrible death, he was so frightened, and trembled so violently, that he had neither voice nor breath left for further entreaties.",
640
+ "label": 1
641
+ },
642
+ {
643
+ "text": "\"Who Pinocchio?",
644
+ "label": 0
645
+ },
646
+ {
647
+ "text": "\"And who is Pinocchio?\"",
648
+ "label": 1
649
+ },
650
+ {
651
+ "text": "\"This is the real time to escape,\" he whispered then, turning to his father. \"The Dog-fish sleeps like a log: the sea is calm and you can see each other as during the day. Come then, father, behind me, and soon we will be safe.\"",
652
+ "label": 0
653
+ },
654
+ {
655
+ "text": "\"This is the moment to escape,\" he whispered, turning to his father; \"the Dog-fish is sleeping like a dormouse, the sea is calm, and it is as light as day. Follow me, dear papa, and in a short time we shall be in safety.\"",
656
+ "label": 1
657
+ },
658
+ {
659
+ "text": "\"Geppetto.",
660
+ "label": 0
661
+ },
662
+ {
663
+ "text": "\"Geppetto.\"",
664
+ "label": 1
665
+ },
666
+ {
667
+ "text": "In the meantime the boys, who had now finished throwing all their books, looked at the puppet's bundle of books a short distance away, and took possession of it in no time at all.",
668
+ "label": 0
669
+ },
670
+ {
671
+ "text": "Just then the boys, who had no more books of their own to throw, spied at a little distance the satchel that belonged to Pinocchio, and took possession of it in less time than it takes to tell.",
672
+ "label": 1
673
+ },
674
+ {
675
+ "text": "\"Poor gonzo! You wanted to do it your way, but you will regret it!\"",
676
+ "label": 0
677
+ },
678
+ {
679
+ "text": "\"Poor fool! you would follow your own way, but you will repent it!\"",
680
+ "label": 1
681
+ },
682
+ {
683
+ "text": "Chapter X",
684
+ "label": 0
685
+ },
686
+ {
687
+ "text": "THE BOOK WAS SOLD THERE AND THEN. CHAPTER X",
688
+ "label": 1
689
+ },
690
+ {
691
+ "text": "AND AMAZING EXERCISES",
692
+ "label": 0
693
+ },
694
+ {
695
+ "text": "On that evening, as you may imagine, an hour before the play was to begin the theatre was crammed.",
696
+ "label": 1
697
+ },
698
+ {
699
+ "text": "\"Raise me a curiosity, my dear Candlewick: have you ever suffered from ear disease?",
700
+ "label": 0
701
+ },
702
+ {
703
+ "text": "\"Satisfy my curiosity, my dear Candlewick: have you ever suffered from disease of the ears?\"",
704
+ "label": 1
705
+ },
706
+ {
707
+ "text": "I'll leave it to you to think if poor Pinocchio's heart began to beat faster! Doubling in strength and energy he swam towards the white rock: and he was already halfway, when suddenly a horrible head of a sea monster came out of the water, with its mouth wide open like a chasm, and three rows of fangs, which would have been scary even to see them painted.",
708
+ "label": 0
709
+ },
710
+ {
711
+ "text": "I leave you to imagine how rapidly poor Pinocchio's heart began to beat. He swam with redoubled strength and energy towards the white rock; and he was already half-way when he saw, rising up out of the water and coming to meet him, the horrible head of a sea-monster. His wide-open cavernous mouth and his three rows of enormous teeth would have been terrifying to look at even in a picture.",
712
+ "label": 1
713
+ },
714
+ {
715
+ "text": "\"Patience! shouted Geppetto all of a sudden, getting to his feet; and putting on her old whip tunic, all patched and patched, she ran out of the house.",
716
+ "label": 0
717
+ },
718
+ {
719
+ "text": "\"Well, patience!\" exclaimed Geppetto, all at once rising to his feet, and putting on his old fustian coat, all patched and darned, he ran out of the house.",
720
+ "label": 1
721
+ },
722
+ {
723
+ "text": "\"And where do you go?",
724
+ "label": 0
725
+ },
726
+ {
727
+ "text": "\"And where are you going?\"",
728
+ "label": 1
729
+ },
730
+ {
731
+ "text": "\"Stay here a little longer and you'll see us.",
732
+ "label": 0
733
+ },
734
+ {
735
+ "text": "\"Stay here a little longer and you will see us.\"",
736
+ "label": 1
737
+ },
738
+ {
739
+ "text": "\"I don't care ...",
740
+ "label": 0
741
+ },
742
+ {
743
+ "text": ". . .\" \"I don't care. . . .\"",
744
+ "label": 1
745
+ },
746
+ {
747
+ "text": "\"Because the guys who do not heed the advice of those who know more than them, always face some misfortune.",
748
+ "label": 0
749
+ },
750
+ {
751
+ "text": "\"Because boys who do not listen to the advice of those who know more than they do always meet with some misfortune or other.\"",
752
+ "label": 1
753
+ },
754
+ {
755
+ "text": "On hearing himself called Excellency, the puppeteer immediately made the round mouthpiece, and all of a sudden he became more human and more manageable, he said to Pinocchio:",
756
+ "label": 0
757
+ },
758
+ {
759
+ "text": "Upon hearing himself called Excellence the showman began to smile, and became at once kinder and more tractable. Turning to Pinocchio he asked:",
760
+ "label": 1
761
+ },
762
+ {
763
+ "text": "\"Died! \"repeated the other.",
764
+ "label": 0
765
+ },
766
+ {
767
+ "text": "\"Dead!\" repeated the other.",
768
+ "label": 1
769
+ },
770
+ {
771
+ "text": "\"That poor devil has been robbed of four gold coins: take him then, and put him immediately in prison.\"",
772
+ "label": 0
773
+ },
774
+ {
775
+ "text": "\"That poor devil has been robbed of four gold pieces; take him up, and put him immediately into prison.\"",
776
+ "label": 1
777
+ },
778
+ {
779
+ "text": "Then the puppet spread out his clothes in the sun to dry them, and began to look here and there if by chance he could see a small boat with a little man inside on that immense stretch of water. But after having looked closely, he saw nothing in front of him but sky, sea and some ship's sails, but so far away that he looked like a fly.",
780
+ "label": 0
781
+ },
782
+ {
783
+ "text": "The puppet put his clothes in the sun to dry, and began to look in every direction in hopes of seeing on the vast expanse of water a little boat with a little man in it. But although he looked and looked, he could see nothing but the sky, and the sea, and the sail of some ship, but so far away that it seemed no bigger than a fly.",
784
+ "label": 1
785
+ },
786
+ {
787
+ "text": "\"Far, far, far!",
788
+ "label": 0
789
+ },
790
+ {
791
+ "text": "\"Very far, very far, very far away.\"",
792
+ "label": 1
793
+ },
794
+ {
795
+ "text": "will be presented for the first time",
796
+ "label": 0
797
+ },
798
+ {
799
+ "text": "On that evening, as you may imagine, an hour before the play was to begin the theatre was crammed.",
800
+ "label": 1
801
+ },
802
+ {
803
+ "text": "\"And why should I redo your feet? Maybe to see you run away from your house again?",
804
+ "label": 0
805
+ },
806
+ {
807
+ "text": "\"Why should I make you new feet? To enable you, perhaps, to escape again from home?\"",
808
+ "label": 1
809
+ },
810
+ {
811
+ "text": "\"Why do I pity you?",
812
+ "label": 0
813
+ },
814
+ {
815
+ "text": "\"Why do you pity me?\"",
816
+ "label": 1
817
+ },
818
+ {
819
+ "text": "\"I've thought of everything. Your father has already been warned: and before night falls, he will be here.",
820
+ "label": 0
821
+ },
822
+ {
823
+ "text": "\"I have thought of everything. I have already let your father know, and he will be here to-night.\"",
824
+ "label": 1
825
+ },
826
+ {
827
+ "text": "\"Not me: I want to go to school.",
828
+ "label": 0
829
+ },
830
+ {
831
+ "text": "\"No; I am going to school.\"",
832
+ "label": 1
833
+ },
834
+ {
835
+ "text": "\"I made it to teach him a lesson. So once again he will learn not to put his mouth in the speeches of others.\"",
836
+ "label": 0
837
+ },
838
+ {
839
+ "text": "\"I did it to give him a lesson. He will learn another time not to meddle in other people's conversation.\"",
840
+ "label": 1
841
+ },
842
+ {
843
+ "text": "And there he had to stay four months: four very long months: and he would have stayed even longer if he hadn't had a very lucky chance. Because it is necessary to know that the young Emperor who reigned in the city of Acchiappa-citrulli, having won a beautiful victory against his enemies, ordered large public celebrations, lights, fireworks, races of barberi and velocipedes, and as a sign of greater exultation, he also wanted the prisons to be opened and all the marauders sent out.",
844
+ "label": 0
845
+ },
846
+ {
847
+ "text": "And there he remained for four months—four long months—and he would have remained longer still if a fortunate chance had not released him. For I must tell you that the young Emperor who reigned over the town of \"Trap for blockheads,\" having won a splendid victory over his enemies, ordered great public rejoicings. There were illuminations, fire-works, horse races, and velocipede races, and as a further sign of triumph he commanded that the prisons should be opened and all the prisoners liberated.",
848
+ "label": 1
849
+ },
850
+ {
851
+ "text": "And as he walked with a hurried step, his heart was beating fast and making him tick, tock, tick, tock, like a hall clock, when it really runs. And meanwhile he was thinking to himself:",
852
+ "label": 0
853
+ },
854
+ {
855
+ "text": "And as he walked along with hurried steps his heart beat fast tic, tac, tic, tac, like a drawing-room clock when it is really going well. Meanwhile he was thinking to himself:",
856
+ "label": 1
857
+ }
858
+ ]
tokenizer.json ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:93189c5d9a15db043017cfd920e00cf72fe9a4220bd74b460b635f6aa85a61a2
3
+ size 17082999
tokenizer_config.json ADDED
@@ -0,0 +1,55 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "added_tokens_decoder": {
3
+ "0": {
4
+ "content": "<s>",
5
+ "lstrip": false,
6
+ "normalized": false,
7
+ "rstrip": false,
8
+ "single_word": false,
9
+ "special": true
10
+ },
11
+ "1": {
12
+ "content": "<pad>",
13
+ "lstrip": false,
14
+ "normalized": false,
15
+ "rstrip": false,
16
+ "single_word": false,
17
+ "special": true
18
+ },
19
+ "2": {
20
+ "content": "</s>",
21
+ "lstrip": false,
22
+ "normalized": false,
23
+ "rstrip": false,
24
+ "single_word": false,
25
+ "special": true
26
+ },
27
+ "3": {
28
+ "content": "<unk>",
29
+ "lstrip": false,
30
+ "normalized": false,
31
+ "rstrip": false,
32
+ "single_word": false,
33
+ "special": true
34
+ },
35
+ "250001": {
36
+ "content": "<mask>",
37
+ "lstrip": true,
38
+ "normalized": false,
39
+ "rstrip": false,
40
+ "single_word": false,
41
+ "special": true
42
+ }
43
+ },
44
+ "bos_token": "<s>",
45
+ "clean_up_tokenization_spaces": false,
46
+ "cls_token": "<s>",
47
+ "eos_token": "</s>",
48
+ "extra_special_tokens": {},
49
+ "mask_token": "<mask>",
50
+ "model_max_length": 512,
51
+ "pad_token": "<pad>",
52
+ "sep_token": "</s>",
53
+ "tokenizer_class": "XLMRobertaTokenizer",
54
+ "unk_token": "<unk>"
55
+ }
training_args.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:7146b376922d917ea85454e1ba5a670b8b5254d71aa5c223d8728479c2097a59
3
+ size 5432