File size: 4,602 Bytes
cf30567 cc4e08c cf30567 cc4e08c cf30567 cc4e08c cf30567 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 |
---
library_name: transformers
language:
- en
license: mit
base_model: FacebookAI/roberta-base
tags:
- generated_from_trainer
datasets:
- swag
metrics:
- accuracy
model-index:
- name: swag_base
results:
- task:
name: Multiple Choice
type: multiple-choice
dataset:
name: SWAG
type: swag
args: regular
metrics:
- name: Accuracy
type: accuracy
value: 0.7521243691444397
---
# swag_base
This model is a fine-tuned version of [FacebookAI/roberta-base](https://huggingface.co/FacebookAI/roberta-base) on the SWAG (Situations With Adversarial Generations) dataset.
## Model description
The model is designed to perform multiple-choice reasoning about real-world situations. Given a context and four possible continuations, it predicts the most plausible ending based on common sense understanding.
Key Features:
- Base model: RoBERTa-base
- Task: Multiple Choice Prediction
- Training dataset: SWAG
- Performance: 75.21% accuracy on evaluation set
## Training Procedure
### Training hyperparameters
- Learning rate: 5e-05
- Batch size: 16
- Number of epochs: 3
- Optimizer: AdamW
- Learning rate scheduler: Linear
- Training samples: 73,546
- Training time: 17m 53s
### Training Results
- Training loss: 0.73
- Evaluation loss: 0.7362
- Evaluation accuracy: 0.7521
- Training samples/second: 205.623
- Training steps/second: 12.852
## Usage Example
Here's how to use the model:
```python
from transformers import AutoTokenizer, AutoModelForMultipleChoice
import torch
# Load model and tokenizer
model_path = "real-jiakai/roberta-base-uncased-finetuned-swag"
tokenizer = AutoTokenizer.from_pretrained(model_path)
model = AutoModelForMultipleChoice.from_pretrained(model_path)
def predict_swag(context, endings, model, tokenizer):
encoding = tokenizer(
[context] * 4,
endings,
truncation=True,
max_length=128,
padding="max_length",
return_tensors="pt"
)
input_ids = encoding['input_ids'].unsqueeze(0)
attention_mask = encoding['attention_mask'].unsqueeze(0)
outputs = model(input_ids=input_ids, attention_mask=attention_mask)
logits = outputs.logits
predicted_idx = torch.argmax(logits).item()
return {
'context': context,
'predicted_ending': endings[predicted_idx],
'probabilities': torch.softmax(logits, dim=1)[0].tolist()
}
# Example scenarios
test_examples = [
{
'context': "Stephen Curry dribbles the ball at the three-point line",
'endings': [
"He quickly releases a perfect shot that swishes through the net", # Most plausible
"He suddenly starts dancing ballet on the court",
"He transforms the basketball into a pizza",
"He flies to the moon with the basketball"
]
},
{
'context': "Elon Musk walks into a SpaceX facility and looks at a rocket",
'endings': [
"He discusses technical details with the engineering team", # Most plausible
"He turns the rocket into a giant chocolate bar",
"He starts playing basketball with the rocket",
"He teaches the rocket to speak French"
]
}
]
for i, example in enumerate(test_examples, 1):
result = predict_swag(
example['context'],
example['endings'],
model,
tokenizer
)
print(f"\n=== Test Scenario {i} ===")
print(f"Initial Context: {result['context']}")
print(f"\nPredicted Most Likely Ending: {result['predicted_ending']}")
print("\nProbabilities for All Options:")
for idx, (ending, prob) in enumerate(zip(result['all_endings'], result['probabilities'])):
print(f"Option {idx}: {ending}")
print(f"Probability: {prob:.3f}")
print("\n" + "="*50)
```
## Limitations and Biases
The model's performance is limited by its training data and may not generalize well to all domains
Performance might vary depending on the complexity and domain of the input scenarios
The model may exhibit biases present in the training data
## Framework versions
Transformers 4.47.0.dev0
PyTorch 2.5.1+cu124
Datasets 3.1.0
Tokenizers 0.20.3
## Citation
If you use this model, please cite:
```
@inproceedings{zellers2018swagaf,
title={SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference},
author={Zellers, Rowan and Bisk, Yonatan and Schwartz, Roy and Choi, Yejin},
booktitle = "Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP)",
year={2018}
}
``` |