Hugging Face's logo

fxmarty
/
20220711-h09m49s39_example_conll2003

Token Classification
TensorBoard
distilbert
Model card Files Metrics Community
fxmarty commited on
Commit
0d8c94a
•
1 Parent(s): dcf18e8

add experience

Browse files
Files changed (7) hide show
  1. README.md +63 -0
  2. runs.json +372 -0
  3. tensorboard/1657532983.959344/events.out.tfevents.1657532983.ip-10-2-87-18.ec2.internal.1.1 +3 -0
  4. tensorboard/1657532983.9607587/events.out.tfevents.1657532983.ip-10-2-87-18.ec2.internal.1.2 +3 -0
  5. tensorboard/1657532983.9619267/events.out.tfevents.1657532983.ip-10-2-87-18.ec2.internal.1.3 +3 -0
  6. tensorboard/1657532983.9629915/events.out.tfevents.1657532983.ip-10-2-87-18.ec2.internal.1.4 +3 -0
  7. tensorboard/events.out.tfevents.1657532983.ip-10-2-87-18.ec2.internal.1.0 +3 -0
README.md ADDED
@@ -0,0 +1,63 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ pipeline_tag: token-classification
3
+ datasets:
4
+ - conll2003
5
+ metrics:
6
+ - precision
7
+ - recall
8
+ - f1
9
+ - accuracy
10
+ tags:
11
+ - distilbert
12
+ ---
13
+
14
+ **task**: `token-classification`
15
+ **Backend:** `sagemaker-training`
16
+ **Backend args:** `{'instance_type': 'ml.m5.2xlarge', 'supported_instructions': 'avx512'}`
17
+ **Number of evaluation samples:** `10`
18
+
19
+ Fixed parameters:
20
+ * **model_name_or_path**: `elastic/distilbert-base-uncased-finetuned-conll03-english`
21
+ * **dataset**:
22
+ * **path**: `conll2003`
23
+ * **eval_split**: `validation`
24
+ * **data_keys**: `{'primary': 'tokens'}`
25
+ * **ref_keys**: `['ner_tags']`
26
+ * **calibration_split**: `train`
27
+ * **node_exclusion**: `[]`
28
+ * **per_channel**: `False`
29
+ * **calibration**:
30
+ * **method**: `minmax`
31
+ * **num_calibration_samples**: `100`
32
+ * **framework**: `onnxruntime`
33
+ * **framework_args**:
34
+ * **opset**: `11`
35
+ * **optimization_level**: `1`
36
+ * **aware_training**: `False`
37
+
38
+ Benchmarked parameters:
39
+ * **quantization_approach**: `dynamic`, `static`
40
+ * **operators_to_quantize**: `['Add', 'MatMul']`, `['Add']`
41
+
42
+ # Evaluation
43
+ ## Non-time metrics
44
+ | quantization_approach | operators_to_quantize | | precision (original) | precision (optimized) | | recall (original) | recall (optimized) | | f1 (original) | f1 (optimized) | | accuracy (original) | accuracy (optimized) |
45
+ | :-------------------: | :-------------------: | :-: | :------------------: | :-------------------: | :-: | :---------------: | :----------------: | :-: | :-----------: | :------------: | :-: | :-----------------: | :------------------: |
46
+ | `dynamic` | `['Add', 'MatMul']` | \| | 0.970 | 0.969 | \| | 0.970 | 0.939 | \| | 0.970 | 0.954 | \| | 0.993 | 0.990 |
47
+ | `dynamic` | `['Add']` | \| | 0.970 | 0.970 | \| | 0.970 | 0.970 | \| | 0.970 | 0.970 | \| | 0.993 | 0.993 |
48
+ | `static` | `['Add', 'MatMul']` | \| | 0.970 | 0.104 | \| | 0.970 | 0.212 | \| | 0.970 | 0.140 | \| | 0.993 | 0.691 |
49
+ | `static` | `['Add']` | \| | 0.970 | 0.037 | \| | 0.970 | 0.121 | \| | 0.970 | 0.057 | \| | 0.993 | 0.110 |
50
+
51
+ ## Time metrics
52
+ Time benchmarks were run for 3 seconds per config.
53
+
54
+
55
+ Below, time metrics for batch size = 1, input length = 64.
56
+
57
+ | quantization_approach | operators_to_quantize | | latency_mean (original, ms) | latency_mean (optimized, ms) | | throughput (original, /s) | throughput (optimized, /s) |
58
+ | :-------------------: | :-------------------: | :-: | :-------------------------: | :--------------------------: | :-: | :-----------------------: | :------------------------: |
59
+ | `dynamic` | `['Add', 'MatMul']` | \| | 60.12 | 18.13 | \| | 16.67 | 55.33 |
60
+ | `dynamic` | `['Add']` | \| | 59.49 | 29.12 | \| | 17.00 | 34.67 |
61
+ | `static` | `['Add', 'MatMul']` | \| | 58.89 | 24.30 | \| | 17.00 | 41.33 |
62
+ | `static` | `['Add']` | \| | 43.19 | 38.12 | \| | 23.33 | 26.33 |
63
+
runs.json ADDED
@@ -0,0 +1,372 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ [
2
+ {
3
+ "model_name_or_path": "elastic/distilbert-base-uncased-finetuned-conll03-english",
4
+ "task": "token-classification",
5
+ "dataset": {
6
+ "path": "conll2003",
7
+ "eval_split": "validation",
8
+ "data_keys": {
9
+ "primary": "tokens",
10
+ "secondary": null
11
+ },
12
+ "ref_keys": [
13
+ "ner_tags"
14
+ ],
15
+ "name": null,
16
+ "calibration_split": "train"
17
+ },
18
+ "quantization_approach": "dynamic",
19
+ "operators_to_quantize": [
20
+ "Add",
21
+ "MatMul"
22
+ ],
23
+ "node_exclusion": [],
24
+ "aware_training": false,
25
+ "per_channel": false,
26
+ "calibration": {
27
+ "method": "minmax",
28
+ "num_calibration_samples": 100,
29
+ "calibration_histogram_percentile": null,
30
+ "calibration_moving_average": null,
31
+ "calibration_moving_average_constant": null
32
+ },
33
+ "framework": "onnxruntime",
34
+ "framework_args": {
35
+ "opset": 11,
36
+ "optimization_level": 1
37
+ },
38
+ "hardware": "Architecture: x86_64\nCPU op-mode(s): 32-bit, 64-bit\nByte Order: Little Endian\nAddress sizes: 46 bits physical, 48 bits virtual\nCPU(s): 8\nOn-line CPU(s) list: 0-7\nThread(s) per core: 2\nCore(s) per socket: 4\nSocket(s): 1\nNUMA node(s): 1\nVendor ID: GenuineIntel\nCPU family: 6\nModel: 85\nModel name: Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz\nStepping: 7\nCPU MHz: 3098.713\nBogoMIPS: 4999.99\nHypervisor vendor: KVM\nVirtualization type: full\nL1d cache: 128 KiB\nL1i cache: 128 KiB\nL2 cache: 4 MiB\nL3 cache: 35.8 MiB\nNUMA node0 CPU(s): 0-7\nVulnerability Itlb multihit: KVM: Vulnerable\nVulnerability L1tf: Mitigation; PTE Inversion\nVulnerability Mds: Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown\nVulnerability Meltdown: Mitigation; PTI\nVulnerability Spec store bypass: Vulnerable\nVulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization\nVulnerability Spectre v2: Mitigation; Retpolines, STIBP disabled, RSB filling\nVulnerability Srbds: Not affected\nVulnerability Tsx async abort: Not affected\nFlags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single pti fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid mpx avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves ida arat pku ospke\n",
39
+ "versions": {
40
+ "transformers": "4.20.1",
41
+ "optimum": "1.2.3.dev0",
42
+ "optimum_hash": "65ad733f16d4e34fee21f98cb1b7edd33b8c1662"
43
+ },
44
+ "evaluation": {
45
+ "time": [
46
+ {
47
+ "batch_size": 1,
48
+ "input_length": 64,
49
+ "baseline": {
50
+ "nb_forwards": 50,
51
+ "throughput": 16.67,
52
+ "latency_mean": 60.11770944,
53
+ "latency_std": 1.5934818902628691,
54
+ "latency_50": 60.264686,
55
+ "latency_90": 61.2435885,
56
+ "latency_95": 61.33683375,
57
+ "latency_99": 65.10274001999998,
58
+ "latency_999": 68.287157802
59
+ },
60
+ "optimized": {
61
+ "nb_forwards": 166,
62
+ "throughput": 55.33,
63
+ "latency_mean": 18.127647289156627,
64
+ "latency_std": 0.41933456288271403,
65
+ "latency_50": 18.0557965,
66
+ "latency_90": 18.814332,
67
+ "latency_95": 18.98832,
68
+ "latency_99": 19.21624665,
69
+ "latency_999": 19.220398929999998
70
+ }
71
+ }
72
+ ],
73
+ "others": {
74
+ "baseline": {
75
+ "precision": 0.9696969696969697,
76
+ "recall": 0.9696969696969697,
77
+ "f1": 0.9696969696969697,
78
+ "accuracy": 0.993127147766323
79
+ },
80
+ "optimized": {
81
+ "precision": 0.96875,
82
+ "recall": 0.9393939393939394,
83
+ "f1": 0.9538461538461539,
84
+ "accuracy": 0.9896907216494846
85
+ }
86
+ }
87
+ },
88
+ "max_eval_samples": 10,
89
+ "time_benchmark_args": {
90
+ "duration": 3,
91
+ "warmup_runs": 1
92
+ },
93
+ "model_type": "distilbert"
94
+ },
95
+ {
96
+ "model_name_or_path": "elastic/distilbert-base-uncased-finetuned-conll03-english",
97
+ "task": "token-classification",
98
+ "dataset": {
99
+ "path": "conll2003",
100
+ "eval_split": "validation",
101
+ "data_keys": {
102
+ "primary": "tokens",
103
+ "secondary": null
104
+ },
105
+ "ref_keys": [
106
+ "ner_tags"
107
+ ],
108
+ "name": null,
109
+ "calibration_split": "train"
110
+ },
111
+ "quantization_approach": "static",
112
+ "operators_to_quantize": [
113
+ "Add"
114
+ ],
115
+ "node_exclusion": [],
116
+ "aware_training": false,
117
+ "per_channel": false,
118
+ "calibration": {
119
+ "method": "minmax",
120
+ "num_calibration_samples": 100,
121
+ "calibration_histogram_percentile": null,
122
+ "calibration_moving_average": null,
123
+ "calibration_moving_average_constant": null
124
+ },
125
+ "framework": "onnxruntime",
126
+ "framework_args": {
127
+ "opset": 11,
128
+ "optimization_level": 1
129
+ },
130
+ "hardware": "Architecture: x86_64\nCPU op-mode(s): 32-bit, 64-bit\nByte Order: Little Endian\nAddress sizes: 46 bits physical, 48 bits virtual\nCPU(s): 8\nOn-line CPU(s) list: 0-7\nThread(s) per core: 2\nCore(s) per socket: 4\nSocket(s): 1\nNUMA node(s): 1\nVendor ID: GenuineIntel\nCPU family: 6\nModel: 85\nModel name: Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz\nStepping: 7\nCPU MHz: 2794.474\nBogoMIPS: 4999.99\nHypervisor vendor: KVM\nVirtualization type: full\nL1d cache: 128 KiB\nL1i cache: 128 KiB\nL2 cache: 4 MiB\nL3 cache: 35.8 MiB\nNUMA node0 CPU(s): 0-7\nVulnerability Itlb multihit: KVM: Vulnerable\nVulnerability L1tf: Mitigation; PTE Inversion\nVulnerability Mds: Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown\nVulnerability Meltdown: Mitigation; PTI\nVulnerability Spec store bypass: Vulnerable\nVulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization\nVulnerability Spectre v2: Mitigation; Retpolines, STIBP disabled, RSB filling\nVulnerability Srbds: Not affected\nVulnerability Tsx async abort: Not affected\nFlags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single pti fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid mpx avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves ida arat pku ospke\n",
131
+ "versions": {
132
+ "transformers": "4.20.1",
133
+ "optimum": "1.2.3.dev0",
134
+ "optimum_hash": "65ad733f16d4e34fee21f98cb1b7edd33b8c1662"
135
+ },
136
+ "evaluation": {
137
+ "time": [
138
+ {
139
+ "batch_size": 1,
140
+ "input_length": 64,
141
+ "baseline": {
142
+ "nb_forwards": 70,
143
+ "throughput": 23.33,
144
+ "latency_mean": 43.18653295714286,
145
+ "latency_std": 1.3760290540598898,
146
+ "latency_50": 42.9665905,
147
+ "latency_90": 45.0960815,
148
+ "latency_95": 45.782922799999994,
149
+ "latency_99": 46.19406632,
150
+ "latency_999": 46.419223532
151
+ },
152
+ "optimized": {
153
+ "nb_forwards": 79,
154
+ "throughput": 26.33,
155
+ "latency_mean": 38.12439892405064,
156
+ "latency_std": 0.07978961670190722,
157
+ "latency_50": 38.117262,
158
+ "latency_90": 38.2253002,
159
+ "latency_95": 38.2424445,
160
+ "latency_99": 38.3146249,
161
+ "latency_999": 38.35825069
162
+ }
163
+ }
164
+ ],
165
+ "others": {
166
+ "baseline": {
167
+ "precision": 0.9696969696969697,
168
+ "recall": 0.9696969696969697,
169
+ "f1": 0.9696969696969697,
170
+ "accuracy": 0.993127147766323
171
+ },
172
+ "optimized": {
173
+ "precision": 0.037037037037037035,
174
+ "recall": 0.12121212121212122,
175
+ "f1": 0.05673758865248226,
176
+ "accuracy": 0.10996563573883161
177
+ }
178
+ }
179
+ },
180
+ "max_eval_samples": 10,
181
+ "time_benchmark_args": {
182
+ "duration": 3,
183
+ "warmup_runs": 1
184
+ },
185
+ "model_type": "distilbert"
186
+ },
187
+ {
188
+ "model_name_or_path": "elastic/distilbert-base-uncased-finetuned-conll03-english",
189
+ "task": "token-classification",
190
+ "dataset": {
191
+ "path": "conll2003",
192
+ "eval_split": "validation",
193
+ "data_keys": {
194
+ "primary": "tokens",
195
+ "secondary": null
196
+ },
197
+ "ref_keys": [
198
+ "ner_tags"
199
+ ],
200
+ "name": null,
201
+ "calibration_split": "train"
202
+ },
203
+ "quantization_approach": "static",
204
+ "operators_to_quantize": [
205
+ "Add",
206
+ "MatMul"
207
+ ],
208
+ "node_exclusion": [],
209
+ "aware_training": false,
210
+ "per_channel": false,
211
+ "calibration": {
212
+ "method": "minmax",
213
+ "num_calibration_samples": 100,
214
+ "calibration_histogram_percentile": null,
215
+ "calibration_moving_average": null,
216
+ "calibration_moving_average_constant": null
217
+ },
218
+ "framework": "onnxruntime",
219
+ "framework_args": {
220
+ "opset": 11,
221
+ "optimization_level": 1
222
+ },
223
+ "hardware": "Architecture: x86_64\nCPU op-mode(s): 32-bit, 64-bit\nByte Order: Little Endian\nAddress sizes: 46 bits physical, 48 bits virtual\nCPU(s): 8\nOn-line CPU(s) list: 0-7\nThread(s) per core: 2\nCore(s) per socket: 4\nSocket(s): 1\nNUMA node(s): 1\nVendor ID: GenuineIntel\nCPU family: 6\nModel: 85\nModel name: Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz\nStepping: 7\nCPU MHz: 3100.839\nBogoMIPS: 4999.99\nHypervisor vendor: KVM\nVirtualization type: full\nL1d cache: 128 KiB\nL1i cache: 128 KiB\nL2 cache: 4 MiB\nL3 cache: 35.8 MiB\nNUMA node0 CPU(s): 0-7\nVulnerability Itlb multihit: KVM: Vulnerable\nVulnerability L1tf: Mitigation; PTE Inversion\nVulnerability Mds: Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown\nVulnerability Meltdown: Mitigation; PTI\nVulnerability Spec store bypass: Vulnerable\nVulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization\nVulnerability Spectre v2: Mitigation; Retpolines, STIBP disabled, RSB filling\nVulnerability Srbds: Not affected\nVulnerability Tsx async abort: Not affected\nFlags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single pti fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid mpx avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves ida arat pku ospke\n",
224
+ "versions": {
225
+ "transformers": "4.20.1",
226
+ "optimum": "1.2.3.dev0",
227
+ "optimum_hash": "65ad733f16d4e34fee21f98cb1b7edd33b8c1662"
228
+ },
229
+ "evaluation": {
230
+ "time": [
231
+ {
232
+ "batch_size": 1,
233
+ "input_length": 64,
234
+ "baseline": {
235
+ "nb_forwards": 51,
236
+ "throughput": 17.0,
237
+ "latency_mean": 58.88688531372549,
238
+ "latency_std": 0.7101158934812859,
239
+ "latency_50": 59.013485,
240
+ "latency_90": 59.654999,
241
+ "latency_95": 59.9591045,
242
+ "latency_99": 60.169608,
243
+ "latency_999": 60.262131600000004
244
+ },
245
+ "optimized": {
246
+ "nb_forwards": 124,
247
+ "throughput": 41.33,
248
+ "latency_mean": 24.301893822580645,
249
+ "latency_std": 1.3073645132666654,
250
+ "latency_50": 23.906192,
251
+ "latency_90": 26.371101600000003,
252
+ "latency_95": 26.70486145,
253
+ "latency_99": 26.872438550000002,
254
+ "latency_999": 26.962627384
255
+ }
256
+ }
257
+ ],
258
+ "others": {
259
+ "baseline": {
260
+ "precision": 0.9696969696969697,
261
+ "recall": 0.9696969696969697,
262
+ "f1": 0.9696969696969697,
263
+ "accuracy": 0.993127147766323
264
+ },
265
+ "optimized": {
266
+ "precision": 0.1044776119402985,
267
+ "recall": 0.21212121212121213,
268
+ "f1": 0.13999999999999999,
269
+ "accuracy": 0.6907216494845361
270
+ }
271
+ }
272
+ },
273
+ "max_eval_samples": 10,
274
+ "time_benchmark_args": {
275
+ "duration": 3,
276
+ "warmup_runs": 1
277
+ },
278
+ "model_type": "distilbert"
279
+ },
280
+ {
281
+ "model_name_or_path": "elastic/distilbert-base-uncased-finetuned-conll03-english",
282
+ "task": "token-classification",
283
+ "dataset": {
284
+ "path": "conll2003",
285
+ "eval_split": "validation",
286
+ "data_keys": {
287
+ "primary": "tokens",
288
+ "secondary": null
289
+ },
290
+ "ref_keys": [
291
+ "ner_tags"
292
+ ],
293
+ "name": null,
294
+ "calibration_split": "train"
295
+ },
296
+ "quantization_approach": "dynamic",
297
+ "operators_to_quantize": [
298
+ "Add"
299
+ ],
300
+ "node_exclusion": [],
301
+ "aware_training": false,
302
+ "per_channel": false,
303
+ "calibration": {
304
+ "method": "minmax",
305
+ "num_calibration_samples": 100,
306
+ "calibration_histogram_percentile": null,
307
+ "calibration_moving_average": null,
308
+ "calibration_moving_average_constant": null
309
+ },
310
+ "framework": "onnxruntime",
311
+ "framework_args": {
312
+ "opset": 11,
313
+ "optimization_level": 1
314
+ },
315
+ "hardware": "Architecture: x86_64\nCPU op-mode(s): 32-bit, 64-bit\nByte Order: Little Endian\nAddress sizes: 46 bits physical, 48 bits virtual\nCPU(s): 8\nOn-line CPU(s) list: 0-7\nThread(s) per core: 2\nCore(s) per socket: 4\nSocket(s): 1\nNUMA node(s): 1\nVendor ID: GenuineIntel\nCPU family: 6\nModel: 85\nModel name: Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz\nStepping: 7\nCPU MHz: 3099.924\nBogoMIPS: 4999.99\nHypervisor vendor: KVM\nVirtualization type: full\nL1d cache: 128 KiB\nL1i cache: 128 KiB\nL2 cache: 4 MiB\nL3 cache: 35.8 MiB\nNUMA node0 CPU(s): 0-7\nVulnerability Itlb multihit: KVM: Vulnerable\nVulnerability L1tf: Mitigation; PTE Inversion\nVulnerability Mds: Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown\nVulnerability Meltdown: Mitigation; PTI\nVulnerability Spec store bypass: Vulnerable\nVulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization\nVulnerability Spectre v2: Mitigation; Retpolines, STIBP disabled, RSB filling\nVulnerability Srbds: Not affected\nVulnerability Tsx async abort: Not affected\nFlags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single pti fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid mpx avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves ida arat pku ospke\n",
316
+ "versions": {
317
+ "transformers": "4.20.1",
318
+ "optimum": "1.2.3.dev0",
319
+ "optimum_hash": "65ad733f16d4e34fee21f98cb1b7edd33b8c1662"
320
+ },
321
+ "evaluation": {
322
+ "time": [
323
+ {
324
+ "batch_size": 1,
325
+ "input_length": 64,
326
+ "baseline": {
327
+ "nb_forwards": 51,
328
+ "throughput": 17.0,
329
+ "latency_mean": 59.487474725490195,
330
+ "latency_std": 0.7968899973526024,
331
+ "latency_50": 59.491817,
332
+ "latency_90": 60.506018,
333
+ "latency_95": 60.637663,
334
+ "latency_99": 61.1182795,
335
+ "latency_999": 61.46885605000001
336
+ },
337
+ "optimized": {
338
+ "nb_forwards": 104,
339
+ "throughput": 34.67,
340
+ "latency_mean": 29.123214990384614,
341
+ "latency_std": 0.08746558610670996,
342
+ "latency_50": 29.1146975,
343
+ "latency_90": 29.2330098,
344
+ "latency_95": 29.28693925,
345
+ "latency_99": 29.36537121,
346
+ "latency_999": 29.454484605
347
+ }
348
+ }
349
+ ],
350
+ "others": {
351
+ "baseline": {
352
+ "precision": 0.9696969696969697,
353
+ "recall": 0.9696969696969697,
354
+ "f1": 0.9696969696969697,
355
+ "accuracy": 0.993127147766323
356
+ },
357
+ "optimized": {
358
+ "precision": 0.9696969696969697,
359
+ "recall": 0.9696969696969697,
360
+ "f1": 0.9696969696969697,
361
+ "accuracy": 0.993127147766323
362
+ }
363
+ }
364
+ },
365
+ "max_eval_samples": 10,
366
+ "time_benchmark_args": {
367
+ "duration": 3,
368
+ "warmup_runs": 1
369
+ },
370
+ "model_type": "distilbert"
371
+ }
372
+ ]
tensorboard/1657532983.959344/events.out.tfevents.1657532983.ip-10-2-87-18.ec2.internal.1.1 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:cf6f7d2d351f8695caa7679f5b8dd1769981e0bf6d4dd6bf584971728832c4eb
3
+ size 738
tensorboard/1657532983.9607587/events.out.tfevents.1657532983.ip-10-2-87-18.ec2.internal.1.2 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:ecba1818395a4cb6a8232707c909dcef8390264c4bf8b172e4e07f34044f7bca
3
+ size 728
tensorboard/1657532983.9619267/events.out.tfevents.1657532983.ip-10-2-87-18.ec2.internal.1.3 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:fff97e82c2feff20cfc3ee3e98a56bc0a7aa9b963e1c064320c18155503310cb
3
+ size 737
tensorboard/1657532983.9629915/events.out.tfevents.1657532983.ip-10-2-87-18.ec2.internal.1.4 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:64352a6a984e9524f1c800931b7b7cb984b5dc0c1507c45283f69b6643459957
3
+ size 727
tensorboard/events.out.tfevents.1657532983.ip-10-2-87-18.ec2.internal.1.0 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:bc7c67092ca2575952c4f66309f0feff8493f9def91f2b9044a8619c84a6acaa
3
+ size 40