donotdelete2 / README.md
fxmarty's picture
add experience
927a97d
|
raw
history blame
2.81 kB
metadata
pipeline_tag: question-answering
datasets:
  - squad
metrics:
  - exact_match
  - f1
tags:
  - distilbert

task: question-answering

Fixed parameters:

  • model_name_or_path: distilbert-base-uncased-distilled-squad
  • dataset:
    • path: squad
    • name: None
    • calibration_split: train
    • eval_split: validation
    • data_keys: {'question': 'question', 'context': 'context'}
    • ref_keys: ['answers']
    • max_seq_length: 128
  • node_exclusion: []
  • per_channel: False
  • calibration:
    • method: minmax
    • num_calibration_samples: 20
    • calibration_histogram_percentile: None
    • calibration_moving_average: None
    • calibration_moving_average_constant: None
  • framework: onnxruntime
  • framework_args:
    • opset: 15
    • optimization_level: 1
  • aware_training: False

Benchmarked parameters:

  • quantization_approach: dynamic, static
  • operators_to_quantize: ['Add', 'MatMul'], ['Add']

Evaluation

Below, time metrics for

  • Batch size: 8
  • Input length: 128
    quantization_approach operators_to_quantize latency_mean (original, ms) latency_mean (optimized, ms) throughput (original, /s) throughput (optimized, /s) exact_match (original) exact_match (optimized) f1 (original) f1 (optimized)
    dynamic ['Add', 'MatMul'] | 247.65 75.80 | 4.50 13.50 | 90.000 85.000 | 90.000 86.667

| dynamic | ['Add'] | | | 234.61 | 191.78 | | | 4.50 | 5.50 | | | 90.000 | 90.000 | | | 90.000 | 90.000 | | static | ['Add', 'MatMul'] | | | 238.04 | 131.81 | | | 4.50 | 8.00 | | | 90.000 | 50.000 | | | 90.000 | 59.362 | | static | ['Add'] | | | 241.84 | 257.85 | | | 4.50 | 4.00 | | | 90.000 | 70.000 | | | 90.000 | 76.333 |