|
--- |
|
library_name: peft |
|
license: mit |
|
datasets: |
|
- vishnun/NLP-KnowledgeGraph |
|
language: |
|
- en |
|
metrics: |
|
- accuracy |
|
pipeline_tag: token-classification |
|
tags: |
|
- nlp |
|
- deep learning |
|
- code |
|
--- |
|
## Training procedure |
|
|
|
- Used PEFT library from huggingface and leveraged LoRA procedure to tune the model. Below are the training metrics. |
|
|
|
| Epoch | Training Loss | Validation Loss | Precision | Recall | F1 | Accuracy | |
|
|------:|--------------:|----------------:|----------:|---------:|---------:|---------:| |
|
| 1 | 0.392600 | 0.347941 | 0.762406 | 0.631506 | 0.690810 | 0.882263 | |
|
| 2 | 0.336300 | 0.302746 | 0.775583 | 0.702650 | 0.737317 | 0.897062 | |
|
| 3 | 0.309500 | 0.294454 | 0.817472 | 0.701828 | 0.755249 | 0.905303 | |
|
| 4 | 0.296700 | 0.281895 | 0.839335 | 0.695757 | 0.760831 | 0.905240 | |
|
| 5 | 0.281700 | 0.273324 | 0.816995 | 0.752103 | 0.783207 | 0.914322 | |
|
| 6 | 0.257300 | 0.262116 | 0.813662 | 0.758553 | 0.785142 | 0.915958 | |
|
| 7 | 0.241200 | 0.255580 | 0.819946 | 0.764308 | 0.791150 | 0.918980 | |
|
| 8 | 0.229900 | 0.255078 | 0.819697 | 0.771074 | 0.794643 | 0.919821 | |
|
| 9 | 0.212800 | 0.248312 | 0.830942 | 0.776450 | 0.802772 | 0.922594 | |
|
| 10 | 0.200900 | 0.245995 | 0.831402 | 0.780244 | 0.805011 | 0.923544 | |
|
|
|
- Model got shrunk by nearly 60 times and with the same efficiency as distilbert-base-uncased |
|
|
|
## Inference |
|
|
|
```python |
|
|
|
from transformers import AutoTokenizer, AutoModel |
|
from peft import get_peft_config, PeftModel, PeftConfig, get_peft_model, LoraConfig, TaskType |
|
|
|
peft_model_id = "vishnun/lora-NLIGraph" |
|
config = PeftConfig.from_pretrained(peft_model_id) |
|
inference_model = AutoModelForTokenClassification.from_pretrained( |
|
config.base_model_name_or_path, num_labels=4, id2label=id2lab, label2id=lab2id |
|
) |
|
tokenizer = AutoTokenizer.from_pretrained(config.base_model_name_or_path) |
|
model = PeftModel.from_pretrained(inference_model, peft_model_id) |
|
|
|
text = "Arsenal will win the Premier League" |
|
inputs = tokenizer(text, return_tensors="pt") |
|
|
|
with torch.no_grad(): |
|
logits = model(**inputs).logits |
|
|
|
tokens = inputs.tokens() |
|
predictions = torch.argmax(logits, dim=2) |
|
|
|
for token, prediction in zip(tokens, predictions[0].numpy()): |
|
print((token, model.config.id2label[prediction])) |
|
|
|
## results : ('<s>', 'O') |
|
('Arsenal', 'SRC') |
|
('Ġwill', 'O') |
|
('Ġwin', 'REL') |
|
('Ġthe', 'O') |
|
('ĠPremier', 'TGT') |
|
('ĠLeague', 'O') |
|
('</s>', 'O') |
|
|
|
``` |
|
### Framework versions |
|
|
|
|
|
- PEFT 0.4.0 |
