File size: 2,282 Bytes
914035f f10216c 32fc86b 914035f f10216c 914035f f10216c 914035f f10216c 914035f f10216c 914035f f10216c 914035f f10216c 914035f f10216c 914035f f10216c 914035f f10216c 914035f f10216c 914035f f10216c 914035f f10216c 32fc86b |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 |
---
tags:
- autotrain
- text-classification
base_model: FacebookAI/roberta-base
widget:
- text: I love AutoTrain
license: apache-2.0
datasets:
- AdamLucek/twittersentiment-llama-3.1-405B-labels
language:
- en
pipeline_tag: text-classification
library_name: transformers
---
# Roberta-Base Trained on Llama 3.1 405B Twitter Sentiment Classification
The [FacebookAI/roberta-base](https://huggingface.co/FacebookAI/roberta-base) 125M Parameter language model trained on annotated twitter sentiment data from [meta-llama/Meta-Llama-3.1-405B](https://huggingface.co/meta-llama/Meta-Llama-3.1-405B) for text classification.
**Evaluation**
Llama 3.1 405B Accuracy: 65.49%
Fine Tuned Roberta Accuracy: 63.38%
Essentially the same performance at **0.03%** of the parameters.
### Fine-tuning Data Description
Data and expected label used in accuracy calculation comes from [mteb/tweet_sentiment_extraction](https://huggingface.co/datasets/mteb/tweet_sentiment_extraction) dataset. Annotations made using a subset of the tweet sentiment extraction dataset, removing blank texts and removing entries deemed innapropriate by Llama 3.1 405B's filter. Generated annotations using Fireworks API, which is expected to be hosting Llama 3.1 405B in FP8 accuracy.
Final **train/test** split count ends at **4992/998**, available at [AdamLucek/twittersentiment-llama-3.1-405B-labels](https://huggingface.co/datasets/AdamLucek/twittersentiment-llama-3.1-405B-labels).
# Using the Model
```python
from transformers import pipeline
# Create sentiment Analysis pipeline
classifier = pipeline("sentiment-analysis", model="AdamLucek/roberta-llama3.1405B-twitter-sentiment")
classifier("Want to get a Blackberry but can`t afford it . Just watching the telly and relaxing. Hard sesion tomorrow.")
# Output: [{'label': 'neutral', 'score': 0.3881794810295105}]
```
## Model Trained Using AutoTrain - Validation Metrics
loss: 0.6081525683403015
f1_macro: 0.7293016589919367
f1_micro: 0.7567567567567568
f1_weighted: 0.7525753769969824
precision_macro: 0.7459781321674904
precision_micro: 0.7567567567567568
precision_weighted: 0.7607241180619724
recall_macro: 0.727181992488115
recall_micro: 0.7567567567567568
recall_weighted: 0.7567567567567568
accuracy: 0.7567567567567568 |