Hugging Face's logo

fxmarty
/
20220711-h10m00s56_example_conll2003

Token Classification
TensorBoard
distilbert
Model card Files Metrics Community
fxmarty commited on
Commit
7e0720e
•
1 Parent(s): d115d02

add experience

Browse files
Files changed (7) hide show
  1. README.md +63 -0
  2. runs.json +372 -0
  3. tensorboard/1657533660.912158/events.out.tfevents.1657533660.ip-10-0-146-241.ec2.internal.1.1 +3 -0
  4. tensorboard/1657533660.9135222/events.out.tfevents.1657533660.ip-10-0-146-241.ec2.internal.1.2 +3 -0
  5. tensorboard/1657533660.9148366/events.out.tfevents.1657533660.ip-10-0-146-241.ec2.internal.1.3 +3 -0
  6. tensorboard/1657533660.915938/events.out.tfevents.1657533660.ip-10-0-146-241.ec2.internal.1.4 +3 -0
  7. tensorboard/events.out.tfevents.1657533660.ip-10-0-146-241.ec2.internal.1.0 +3 -0
README.md ADDED
@@ -0,0 +1,63 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ pipeline_tag: token-classification
3
+ datasets:
4
+ - conll2003
5
+ metrics:
6
+ - precision
7
+ - recall
8
+ - f1
9
+ - accuracy
10
+ tags:
11
+ - distilbert
12
+ ---
13
+
14
+ **task**: `token-classification`
15
+ **Backend:** `sagemaker-training`
16
+ **Backend args:** `{'instance_type': 'ml.m5.2xlarge', 'supported_instructions': 'avx512'}`
17
+ **Number of evaluation samples:** `100`
18
+
19
+ Fixed parameters:
20
+ * **model_name_or_path**: `elastic/distilbert-base-uncased-finetuned-conll03-english`
21
+ * **dataset**:
22
+ * **path**: `conll2003`
23
+ * **eval_split**: `validation`
24
+ * **data_keys**: `{'primary': 'tokens'}`
25
+ * **ref_keys**: `['ner_tags']`
26
+ * **calibration_split**: `train`
27
+ * **node_exclusion**: `[]`
28
+ * **per_channel**: `False`
29
+ * **calibration**:
30
+ * **method**: `minmax`
31
+ * **num_calibration_samples**: `100`
32
+ * **framework**: `onnxruntime`
33
+ * **framework_args**:
34
+ * **opset**: `11`
35
+ * **optimization_level**: `1`
36
+ * **aware_training**: `False`
37
+
38
+ Benchmarked parameters:
39
+ * **quantization_approach**: `dynamic`, `static`
40
+ * **operators_to_quantize**: `['Add', 'MatMul']`, `['Add']`
41
+
42
+ # Evaluation
43
+ ## Non-time metrics
44
+ | quantization_approach | operators_to_quantize | | precision (original) | precision (optimized) | | recall (original) | recall (optimized) | | f1 (original) | f1 (optimized) | | accuracy (original) | accuracy (optimized) |
45
+ | :-------------------: | :-------------------: | :-: | :------------------: | :-------------------: | :-: | :---------------: | :----------------: | :-: | :-----------: | :------------: | :-: | :-----------------: | :------------------: |
46
+ | `dynamic` | `['Add', 'MatMul']` | \| | 0.974 | 0.974 | \| | 0.955 | 0.949 | \| | 0.964 | 0.962 | \| | 0.990 | 0.989 |
47
+ | `dynamic` | `['Add']` | \| | 0.974 | 0.974 | \| | 0.955 | 0.955 | \| | 0.964 | 0.964 | \| | 0.990 | 0.990 |
48
+ | `static` | `['Add', 'MatMul']` | \| | 0.974 | 0.081 | \| | 0.955 | 0.222 | \| | 0.964 | 0.118 | \| | 0.990 | 0.467 |
49
+ | `static` | `['Add']` | \| | 0.974 | 0.073 | \| | 0.955 | 0.182 | \| | 0.964 | 0.105 | \| | 0.990 | 0.290 |
50
+
51
+ ## Time metrics
52
+ Time benchmarks were run for 3 seconds per config.
53
+
54
+
55
+ Below, time metrics for batch size = 1, input length = 64.
56
+
57
+ | quantization_approach | operators_to_quantize | | latency_mean (original, ms) | latency_mean (optimized, ms) | | throughput (original, /s) | throughput (optimized, /s) |
58
+ | :-------------------: | :-------------------: | :-: | :-------------------------: | :--------------------------: | :-: | :-----------------------: | :------------------------: |
59
+ | `dynamic` | `['Add', 'MatMul']` | \| | 59.35 | 21.91 | \| | 17.00 | 45.67 |
60
+ | `dynamic` | `['Add']` | \| | 59.18 | 29.24 | \| | 17.00 | 34.33 |
61
+ | `static` | `['Add', 'MatMul']` | \| | 59.25 | 28.31 | \| | 17.00 | 35.33 |
62
+ | `static` | `['Add']` | \| | 58.77 | 31.80 | \| | 17.33 | 31.67 |
63
+
runs.json ADDED
@@ -0,0 +1,372 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ [
2
+ {
3
+ "model_name_or_path": "elastic/distilbert-base-uncased-finetuned-conll03-english",
4
+ "task": "token-classification",
5
+ "dataset": {
6
+ "path": "conll2003",
7
+ "eval_split": "validation",
8
+ "data_keys": {
9
+ "primary": "tokens",
10
+ "secondary": null
11
+ },
12
+ "ref_keys": [
13
+ "ner_tags"
14
+ ],
15
+ "name": null,
16
+ "calibration_split": "train"
17
+ },
18
+ "quantization_approach": "dynamic",
19
+ "operators_to_quantize": [
20
+ "Add",
21
+ "MatMul"
22
+ ],
23
+ "node_exclusion": [],
24
+ "aware_training": false,
25
+ "per_channel": false,
26
+ "calibration": {
27
+ "method": "minmax",
28
+ "num_calibration_samples": 100,
29
+ "calibration_histogram_percentile": null,
30
+ "calibration_moving_average": null,
31
+ "calibration_moving_average_constant": null
32
+ },
33
+ "framework": "onnxruntime",
34
+ "framework_args": {
35
+ "opset": 11,
36
+ "optimization_level": 1
37
+ },
38
+ "hardware": "Architecture: x86_64\nCPU op-mode(s): 32-bit, 64-bit\nByte Order: Little Endian\nAddress sizes: 46 bits physical, 48 bits virtual\nCPU(s): 8\nOn-line CPU(s) list: 0-7\nThread(s) per core: 2\nCore(s) per socket: 4\nSocket(s): 1\nNUMA node(s): 1\nVendor ID: GenuineIntel\nCPU family: 6\nModel: 85\nModel name: Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz\nStepping: 4\nCPU MHz: 3108.647\nBogoMIPS: 5000.01\nHypervisor vendor: KVM\nVirtualization type: full\nL1d cache: 128 KiB\nL1i cache: 128 KiB\nL2 cache: 4 MiB\nL3 cache: 33 MiB\nNUMA node0 CPU(s): 0-7\nVulnerability Itlb multihit: KVM: Vulnerable\nVulnerability L1tf: Mitigation; PTE Inversion\nVulnerability Mds: Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown\nVulnerability Meltdown: Mitigation; PTI\nVulnerability Spec store bypass: Vulnerable\nVulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization\nVulnerability Spectre v2: Mitigation; Retpolines, STIBP disabled, RSB filling\nVulnerability Srbds: Not affected\nVulnerability Tsx async abort: Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown\nFlags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single pti fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves ida arat pku ospke\n",
39
+ "versions": {
40
+ "transformers": "4.20.1",
41
+ "optimum": "1.2.3.dev0",
42
+ "optimum_hash": "65ad733f16d4e34fee21f98cb1b7edd33b8c1662"
43
+ },
44
+ "evaluation": {
45
+ "time": [
46
+ {
47
+ "batch_size": 1,
48
+ "input_length": 64,
49
+ "baseline": {
50
+ "nb_forwards": 51,
51
+ "throughput": 17.0,
52
+ "latency_mean": 59.34628984313726,
53
+ "latency_std": 0.778172196284656,
54
+ "latency_50": 59.237776,
55
+ "latency_90": 60.363116,
56
+ "latency_95": 60.5019055,
57
+ "latency_99": 61.45229,
58
+ "latency_999": 61.5998972
59
+ },
60
+ "optimized": {
61
+ "nb_forwards": 137,
62
+ "throughput": 45.67,
63
+ "latency_mean": 21.914979291970806,
64
+ "latency_std": 0.10463730867869543,
65
+ "latency_50": 21.891895,
66
+ "latency_90": 22.0202298,
67
+ "latency_95": 22.075110600000002,
68
+ "latency_99": 22.195937479999998,
69
+ "latency_999": 22.661796416
70
+ }
71
+ }
72
+ ],
73
+ "others": {
74
+ "baseline": {
75
+ "precision": 0.9742268041237113,
76
+ "recall": 0.9545454545454546,
77
+ "f1": 0.9642857142857142,
78
+ "accuracy": 0.9896296296296296
79
+ },
80
+ "optimized": {
81
+ "precision": 0.9740932642487047,
82
+ "recall": 0.9494949494949495,
83
+ "f1": 0.9616368286445013,
84
+ "accuracy": 0.9888888888888889
85
+ }
86
+ }
87
+ },
88
+ "max_eval_samples": 100,
89
+ "time_benchmark_args": {
90
+ "duration": 3,
91
+ "warmup_runs": 1
92
+ },
93
+ "model_type": "distilbert"
94
+ },
95
+ {
96
+ "model_name_or_path": "elastic/distilbert-base-uncased-finetuned-conll03-english",
97
+ "task": "token-classification",
98
+ "dataset": {
99
+ "path": "conll2003",
100
+ "eval_split": "validation",
101
+ "data_keys": {
102
+ "primary": "tokens",
103
+ "secondary": null
104
+ },
105
+ "ref_keys": [
106
+ "ner_tags"
107
+ ],
108
+ "name": null,
109
+ "calibration_split": "train"
110
+ },
111
+ "quantization_approach": "dynamic",
112
+ "operators_to_quantize": [
113
+ "Add"
114
+ ],
115
+ "node_exclusion": [],
116
+ "aware_training": false,
117
+ "per_channel": false,
118
+ "calibration": {
119
+ "method": "minmax",
120
+ "num_calibration_samples": 100,
121
+ "calibration_histogram_percentile": null,
122
+ "calibration_moving_average": null,
123
+ "calibration_moving_average_constant": null
124
+ },
125
+ "framework": "onnxruntime",
126
+ "framework_args": {
127
+ "opset": 11,
128
+ "optimization_level": 1
129
+ },
130
+ "hardware": "Architecture: x86_64\nCPU op-mode(s): 32-bit, 64-bit\nByte Order: Little Endian\nAddress sizes: 46 bits physical, 48 bits virtual\nCPU(s): 8\nOn-line CPU(s) list: 0-7\nThread(s) per core: 2\nCore(s) per socket: 4\nSocket(s): 1\nNUMA node(s): 1\nVendor ID: GenuineIntel\nCPU family: 6\nModel: 85\nModel name: Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz\nStepping: 4\nCPU MHz: 2898.018\nBogoMIPS: 5000.01\nHypervisor vendor: KVM\nVirtualization type: full\nL1d cache: 128 KiB\nL1i cache: 128 KiB\nL2 cache: 4 MiB\nL3 cache: 33 MiB\nNUMA node0 CPU(s): 0-7\nVulnerability Itlb multihit: KVM: Vulnerable\nVulnerability L1tf: Mitigation; PTE Inversion\nVulnerability Mds: Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown\nVulnerability Meltdown: Mitigation; PTI\nVulnerability Spec store bypass: Vulnerable\nVulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization\nVulnerability Spectre v2: Mitigation; Retpolines, STIBP disabled, RSB filling\nVulnerability Srbds: Not affected\nVulnerability Tsx async abort: Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown\nFlags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single pti fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves ida arat pku ospke\n",
131
+ "versions": {
132
+ "transformers": "4.20.1",
133
+ "optimum": "1.2.3.dev0",
134
+ "optimum_hash": "65ad733f16d4e34fee21f98cb1b7edd33b8c1662"
135
+ },
136
+ "evaluation": {
137
+ "time": [
138
+ {
139
+ "batch_size": 1,
140
+ "input_length": 64,
141
+ "baseline": {
142
+ "nb_forwards": 51,
143
+ "throughput": 17.0,
144
+ "latency_mean": 59.18282125490196,
145
+ "latency_std": 0.6426837410053724,
146
+ "latency_50": 59.074956,
147
+ "latency_90": 60.217791,
148
+ "latency_95": 60.3066755,
149
+ "latency_99": 60.4462285,
150
+ "latency_999": 60.50652175
151
+ },
152
+ "optimized": {
153
+ "nb_forwards": 103,
154
+ "throughput": 34.33,
155
+ "latency_mean": 29.235091689320388,
156
+ "latency_std": 0.0994740876199717,
157
+ "latency_50": 29.216962,
158
+ "latency_90": 29.3426696,
159
+ "latency_95": 29.4535227,
160
+ "latency_99": 29.55608076,
161
+ "latency_999": 29.56924647
162
+ }
163
+ }
164
+ ],
165
+ "others": {
166
+ "baseline": {
167
+ "precision": 0.9742268041237113,
168
+ "recall": 0.9545454545454546,
169
+ "f1": 0.9642857142857142,
170
+ "accuracy": 0.9896296296296296
171
+ },
172
+ "optimized": {
173
+ "precision": 0.9742268041237113,
174
+ "recall": 0.9545454545454546,
175
+ "f1": 0.9642857142857142,
176
+ "accuracy": 0.9896296296296296
177
+ }
178
+ }
179
+ },
180
+ "max_eval_samples": 100,
181
+ "time_benchmark_args": {
182
+ "duration": 3,
183
+ "warmup_runs": 1
184
+ },
185
+ "model_type": "distilbert"
186
+ },
187
+ {
188
+ "model_name_or_path": "elastic/distilbert-base-uncased-finetuned-conll03-english",
189
+ "task": "token-classification",
190
+ "dataset": {
191
+ "path": "conll2003",
192
+ "eval_split": "validation",
193
+ "data_keys": {
194
+ "primary": "tokens",
195
+ "secondary": null
196
+ },
197
+ "ref_keys": [
198
+ "ner_tags"
199
+ ],
200
+ "name": null,
201
+ "calibration_split": "train"
202
+ },
203
+ "quantization_approach": "static",
204
+ "operators_to_quantize": [
205
+ "Add",
206
+ "MatMul"
207
+ ],
208
+ "node_exclusion": [],
209
+ "aware_training": false,
210
+ "per_channel": false,
211
+ "calibration": {
212
+ "method": "minmax",
213
+ "num_calibration_samples": 100,
214
+ "calibration_histogram_percentile": null,
215
+ "calibration_moving_average": null,
216
+ "calibration_moving_average_constant": null
217
+ },
218
+ "framework": "onnxruntime",
219
+ "framework_args": {
220
+ "opset": 11,
221
+ "optimization_level": 1
222
+ },
223
+ "hardware": "Architecture: x86_64\nCPU op-mode(s): 32-bit, 64-bit\nByte Order: Little Endian\nAddress sizes: 46 bits physical, 48 bits virtual\nCPU(s): 8\nOn-line CPU(s) list: 0-7\nThread(s) per core: 2\nCore(s) per socket: 4\nSocket(s): 1\nNUMA node(s): 1\nVendor ID: GenuineIntel\nCPU family: 6\nModel: 85\nModel name: Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz\nStepping: 4\nCPU MHz: 3100.046\nBogoMIPS: 5000.01\nHypervisor vendor: KVM\nVirtualization type: full\nL1d cache: 128 KiB\nL1i cache: 128 KiB\nL2 cache: 4 MiB\nL3 cache: 33 MiB\nNUMA node0 CPU(s): 0-7\nVulnerability Itlb multihit: KVM: Vulnerable\nVulnerability L1tf: Mitigation; PTE Inversion\nVulnerability Mds: Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown\nVulnerability Meltdown: Mitigation; PTI\nVulnerability Spec store bypass: Vulnerable\nVulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization\nVulnerability Spectre v2: Mitigation; Retpolines, STIBP disabled, RSB filling\nVulnerability Srbds: Not affected\nVulnerability Tsx async abort: Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown\nFlags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single pti fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves ida arat pku ospke\n",
224
+ "versions": {
225
+ "transformers": "4.20.1",
226
+ "optimum": "1.2.3.dev0",
227
+ "optimum_hash": "65ad733f16d4e34fee21f98cb1b7edd33b8c1662"
228
+ },
229
+ "evaluation": {
230
+ "time": [
231
+ {
232
+ "batch_size": 1,
233
+ "input_length": 64,
234
+ "baseline": {
235
+ "nb_forwards": 51,
236
+ "throughput": 17.0,
237
+ "latency_mean": 59.248113901960785,
238
+ "latency_std": 0.614252094367223,
239
+ "latency_50": 59.360647,
240
+ "latency_90": 59.876494,
241
+ "latency_95": 60.110628,
242
+ "latency_99": 60.3347135,
243
+ "latency_999": 60.33478775
244
+ },
245
+ "optimized": {
246
+ "nb_forwards": 106,
247
+ "throughput": 35.33,
248
+ "latency_mean": 28.305577603773585,
249
+ "latency_std": 0.1333294316560668,
250
+ "latency_50": 28.283224,
251
+ "latency_90": 28.4987405,
252
+ "latency_95": 28.56634,
253
+ "latency_99": 28.63837805,
254
+ "latency_999": 28.83994
255
+ }
256
+ }
257
+ ],
258
+ "others": {
259
+ "baseline": {
260
+ "precision": 0.9742268041237113,
261
+ "recall": 0.9545454545454546,
262
+ "f1": 0.9642857142857142,
263
+ "accuracy": 0.9896296296296296
264
+ },
265
+ "optimized": {
266
+ "precision": 0.08073394495412844,
267
+ "recall": 0.2222222222222222,
268
+ "f1": 0.11843876177658141,
269
+ "accuracy": 0.4666666666666667
270
+ }
271
+ }
272
+ },
273
+ "max_eval_samples": 100,
274
+ "time_benchmark_args": {
275
+ "duration": 3,
276
+ "warmup_runs": 1
277
+ },
278
+ "model_type": "distilbert"
279
+ },
280
+ {
281
+ "model_name_or_path": "elastic/distilbert-base-uncased-finetuned-conll03-english",
282
+ "task": "token-classification",
283
+ "dataset": {
284
+ "path": "conll2003",
285
+ "eval_split": "validation",
286
+ "data_keys": {
287
+ "primary": "tokens",
288
+ "secondary": null
289
+ },
290
+ "ref_keys": [
291
+ "ner_tags"
292
+ ],
293
+ "name": null,
294
+ "calibration_split": "train"
295
+ },
296
+ "quantization_approach": "static",
297
+ "operators_to_quantize": [
298
+ "Add"
299
+ ],
300
+ "node_exclusion": [],
301
+ "aware_training": false,
302
+ "per_channel": false,
303
+ "calibration": {
304
+ "method": "minmax",
305
+ "num_calibration_samples": 100,
306
+ "calibration_histogram_percentile": null,
307
+ "calibration_moving_average": null,
308
+ "calibration_moving_average_constant": null
309
+ },
310
+ "framework": "onnxruntime",
311
+ "framework_args": {
312
+ "opset": 11,
313
+ "optimization_level": 1
314
+ },
315
+ "hardware": "Architecture: x86_64\nCPU op-mode(s): 32-bit, 64-bit\nByte Order: Little Endian\nAddress sizes: 46 bits physical, 48 bits virtual\nCPU(s): 8\nOn-line CPU(s) list: 0-7\nThread(s) per core: 2\nCore(s) per socket: 4\nSocket(s): 1\nNUMA node(s): 1\nVendor ID: GenuineIntel\nCPU family: 6\nModel: 85\nModel name: Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz\nStepping: 4\nCPU MHz: 3100.104\nBogoMIPS: 5000.01\nHypervisor vendor: KVM\nVirtualization type: full\nL1d cache: 128 KiB\nL1i cache: 128 KiB\nL2 cache: 4 MiB\nL3 cache: 33 MiB\nNUMA node0 CPU(s): 0-7\nVulnerability Itlb multihit: KVM: Vulnerable\nVulnerability L1tf: Mitigation; PTE Inversion\nVulnerability Mds: Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown\nVulnerability Meltdown: Mitigation; PTI\nVulnerability Spec store bypass: Vulnerable\nVulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization\nVulnerability Spectre v2: Mitigation; Retpolines, STIBP disabled, RSB filling\nVulnerability Srbds: Not affected\nVulnerability Tsx async abort: Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown\nFlags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single pti fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves ida arat pku ospke\n",
316
+ "versions": {
317
+ "transformers": "4.20.1",
318
+ "optimum": "1.2.3.dev0",
319
+ "optimum_hash": "65ad733f16d4e34fee21f98cb1b7edd33b8c1662"
320
+ },
321
+ "evaluation": {
322
+ "time": [
323
+ {
324
+ "batch_size": 1,
325
+ "input_length": 64,
326
+ "baseline": {
327
+ "nb_forwards": 52,
328
+ "throughput": 17.33,
329
+ "latency_mean": 58.76881473076923,
330
+ "latency_std": 0.964794406899851,
331
+ "latency_50": 58.816893,
332
+ "latency_90": 60.1946519,
333
+ "latency_95": 60.455782799999994,
334
+ "latency_99": 60.82715915,
335
+ "latency_999": 60.950738015
336
+ },
337
+ "optimized": {
338
+ "nb_forwards": 95,
339
+ "throughput": 31.67,
340
+ "latency_mean": 31.801677631578947,
341
+ "latency_std": 1.0425164191094953,
342
+ "latency_50": 31.513748,
343
+ "latency_90": 33.2824478,
344
+ "latency_95": 34.0926499,
345
+ "latency_99": 34.60494526,
346
+ "latency_999": 34.640664226000005
347
+ }
348
+ }
349
+ ],
350
+ "others": {
351
+ "baseline": {
352
+ "precision": 0.9742268041237113,
353
+ "recall": 0.9545454545454546,
354
+ "f1": 0.9642857142857142,
355
+ "accuracy": 0.9896296296296296
356
+ },
357
+ "optimized": {
358
+ "precision": 0.07346938775510205,
359
+ "recall": 0.18181818181818182,
360
+ "f1": 0.10465116279069768,
361
+ "accuracy": 0.2903703703703704
362
+ }
363
+ }
364
+ },
365
+ "max_eval_samples": 100,
366
+ "time_benchmark_args": {
367
+ "duration": 3,
368
+ "warmup_runs": 1
369
+ },
370
+ "model_type": "distilbert"
371
+ }
372
+ ]
tensorboard/1657533660.912158/events.out.tfevents.1657533660.ip-10-0-146-241.ec2.internal.1.1 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:6068f15fdc3083e05ffe7cec570d5a20ede97853e3e484c422ed7e9a133d2ad7
3
+ size 738
tensorboard/1657533660.9135222/events.out.tfevents.1657533660.ip-10-0-146-241.ec2.internal.1.2 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:ef13b3e6dadba242af7aa6b139c08b711cde7fb5162401d8f404f469464808fb
3
+ size 728
tensorboard/1657533660.9148366/events.out.tfevents.1657533660.ip-10-0-146-241.ec2.internal.1.3 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:12feeeb5f34f6c45feda82614ee7e80edd73c2a5f068f9585a7032aa2e1462c8
3
+ size 737
tensorboard/1657533660.915938/events.out.tfevents.1657533660.ip-10-0-146-241.ec2.internal.1.4 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:536dfe6af2eba8eceb81500c9310fb8602d48e950efbdfc400d52df8967f2a84
3
+ size 727
tensorboard/events.out.tfevents.1657533660.ip-10-0-146-241.ec2.internal.1.0 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:0a09facf2168400b4687190daf8d63bbf0ae98d99acbc8cf8e1a10140f111e7b
3
+ size 40