| language: en | |
| datasets: | |
| - squad | |
| metrics: | |
| - squad | |
| license: apache-2.0 | |
| model-index: | |
| - name: autoevaluate/distilbert-base-cased-distilled-squad | |
| results: | |
| - task: | |
| type: question-answering | |
| name: Question Answering | |
| dataset: | |
| name: squad_v2 | |
| type: squad_v2 | |
| config: squad_v2 | |
| split: validation | |
| metrics: | |
| - name: Exact Match | |
| type: exact_match | |
| value: 39.7675 | |
| verified: true | |
| - name: F1 | |
| type: f1 | |
| value: 43.6039 | |
| verified: true | |
| - name: total | |
| type: total | |
| value: 11869 | |
| verified: true | |
| # DistilBERT base cased distilled SQuAD | |
| > Note: This model is a clone of [`distilbert-base-cased-distilled-squad`](https://huggingface.co/distilbert-base-cased-distilled-squad) for internal testing. | |
| This model is a fine-tune checkpoint of [DistilBERT-base-cased](https://huggingface.co/distilbert-base-cased), fine-tuned using (a second step of) knowledge distillation on SQuAD v1.1. | |
| This model reaches a F1 score of 87.1 on the dev set (for comparison, BERT bert-base-cased version reaches a F1 score of 88.7). | |
| Using the question answering `Evaluator` from evaluate gives: | |
| ``` | |
| {'exact_match': 79.54588457899716, | |
| 'f1': 86.81181300991533, | |
| 'latency_in_seconds': 0.008683730778997168, | |
| 'samples_per_second': 115.15787689073015, | |
| 'total_time_in_seconds': 91.78703433400005} | |
| ``` | |
| which is roughly consistent with the official score. |