File size: 4,995 Bytes
cb2b286 f451ebc b315c71 330f287 cb2b286 f451ebc f3eda55 5dfabeb f3eda55 f451ebc |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 |
---
license: mit
datasets:
- squad_v2
- squad
language:
- en
library_name: transformers
tags:
- question-answering
- squad
- squad_v2
- t5
- lora
- peft
model-index:
- name: sjrhuschlee/flan-t5-large-squad2
results:
- task:
type: question-answering
name: Question Answering
dataset:
name: squad_v2
type: squad_v2
config: squad_v2
split: validation
metrics:
- type: exact_match
value: 86.785
name: Exact Match
- type: f1
value: 89.537
name: F1
- task:
type: question-answering
name: Question Answering
dataset:
name: squad
type: squad
config: plain_text
split: validation
metrics:
- type: exact_match
value: 85.998
name: Exact Match
- type: f1
value: 91.296
name: F1
- task:
type: question-answering
name: Question Answering
dataset:
name: squad_adversarial
type: squad_adversarial
config: AddOneSent
split: validation
metrics:
- type: exact_match
value: 75.322
name: Exact Match
- type: f1
value: 79.327
name: F1
---
# flan-t5-large for Extractive QA
This is the [flan-t5-large](https://huggingface.co/google/flan-t5-large) model, fine-tuned using the [SQuAD2.0](https://huggingface.co/datasets/squad_v2) dataset. It's been trained on question-answer pairs, including unanswerable questions, for the task of Extractive Question Answering.
This model was trained using LoRA available through the [PEFT library](https://github.com/huggingface/peft).
NOTE: The <cls> token must be manually added to the beginning of the question for this model to work properly. It uses the <cls> token to be able to make "no answer" predictions. The t5 tokenizer does not automatically add this special token which is why it is added manually.
## Overview
**Language model:** flan-t5-large
**Language:** English
**Downstream-task:** Extractive QA
**Training data:** SQuAD 2.0
**Eval data:** SQuAD 2.0
**Infrastructure**: 1x NVIDIA 3070
## Model Usage
### Using Transformers
This uses the merged weights (base model weights + LoRA weights) to allow for simple use in Transformers pipelines. It has the same performance as using the weights separately when using the PEFT library.
```python
import torch
from transformers import(
AutoModelForQuestionAnswering,
AutoTokenizer,
pipeline
)
model_name = "sjrhuschlee/flan-t5-large-squad2"
# a) Using pipelines
nlp = pipeline('question-answering', model=model_name, tokenizer=model_name)
qa_input = {
'question': f'{nlp.tokenizer.cls_token}Where do I live?', # '<cls>Where do I live?'
'context': 'My name is Sarah and I live in London'
}
res = nlp(qa_input)
# {'score': 0.984, 'start': 30, 'end': 37, 'answer': ' London'}
# b) Load model & tokenizer
model = AutoModelForQuestionAnswering.from_pretrained(model_name)
tokenizer = AutoTokenizer.from_pretrained(model_name)
question = f'{tokenizer.cls_token}Where do I live?' # '<cls>Where do I live?'
context = 'My name is Sarah and I live in London'
encoding = tokenizer(question, context, return_tensors="pt")
start_scores, end_scores = model(
encoding["input_ids"],
attention_mask=encoding["attention_mask"],
return_dict=False
)
all_tokens = tokenizer.convert_ids_to_tokens(input_ids[0].tolist())
answer_tokens = all_tokens[torch.argmax(start_scores):torch.argmax(end_scores) + 1]
answer = tokenizer.decode(tokenizer.convert_tokens_to_ids(answer_tokens))
# 'London'
```
## Metrics
```bash
# Squad v2
{
"eval_HasAns_exact": 85.08771929824562,
"eval_HasAns_f1": 90.598422845031,
"eval_HasAns_total": 5928,
"eval_NoAns_exact": 88.47771236333053,
"eval_NoAns_f1": 88.47771236333053,
"eval_NoAns_total": 5945,
"eval_best_exact": 86.78514276088605,
"eval_best_exact_thresh": 0.0,
"eval_best_f1": 89.53654936623764,
"eval_best_f1_thresh": 0.0,
"eval_exact": 86.78514276088605,
"eval_f1": 89.53654936623776,
"eval_runtime": 1908.3189,
"eval_samples": 12001,
"eval_samples_per_second": 6.289,
"eval_steps_per_second": 0.787,
"eval_total": 11873
}
# Squad
{
"eval_HasAns_exact": 85.99810785241249,
"eval_HasAns_f1": 91.296119057944,
"eval_HasAns_total": 10570,
"eval_best_exact": 85.99810785241249,
"eval_best_exact_thresh": 0.0,
"eval_best_f1": 91.296119057944,
"eval_best_f1_thresh": 0.0,
"eval_exact": 85.99810785241249,
"eval_f1": 91.296119057944,
"eval_runtime": 1508.9596,
"eval_samples": 10657,
"eval_samples_per_second": 7.062,
"eval_steps_per_second": 0.883,
"eval_total": 10570
}
```
### Using with Peft
**NOTE**: This requires code in the PR https://github.com/huggingface/peft/pull/473 for the PEFT library.
```python
#!pip install peft
from peft import LoraConfig, PeftModelForQuestionAnswering
from transformers import AutoModelForQuestionAnswering, AutoTokenizer
model_name = "sjrhuschlee/flan-t5-large-squad2"
``` |