Spaces:
Running
Running
Report for cardiffnlp/twitter-roberta-base-offensive
#11
by
giskard-bot
- opened
Hey Team!🤗✨
We’re thrilled to share some amazing evaluation results that’ll make your day!🎉📊
We have identified 3 potential vulnerabilities in your model based on an automated scan.
This automated analysis evaluated the model on the dataset tweet_eval (subset offensive, split validation).
👉Overconfidence issues (2)
| Vulnerability | Level | Data slice | Metric | Transformation | Deviation |
|---|---|---|---|---|---|
| Overconfidence | major 🔴 | avg_word_length(text) < 4.191 |
Overconfidence rate = 0.697 | — | +30.00% than global |
🔍✨Examples
For records in the dataset where `avg_word_length(text)` < 4.191, we found a significantly higher number of overconfident wrong predictions (85 samples, corresponding to 69.67213114754098% of the wrong predictions in the data slice).| text | avg_word_length(text) | label | Predicted label |
|
|---|---|---|---|---|
| 1138 | @user An idiot. Where the fuck do they get these people? | 4.18182 | offensive | non-offensive (p = 0.95) |
| offensive (p = 0.05) | ||||
| 131 | @user @user Be coo you got people thinking I really eat ass bitch 😂 | 3.85714 | offensive | non-offensive (p = 0.94) |
| offensive (p = 0.06) | ||||
| 803 | @user She is a complete idiot | 4 | offensive | non-offensive (p = 0.94) |
| offensive (p = 0.06) |
| Vulnerability | Level | Data slice | Metric | Transformation | Deviation |
|---|---|---|---|---|---|
| Overconfidence | medium 🟡 | text_length(text) < 172.500 |
Overconfidence rate = 0.605 | — | +12.97% than global |
🔍✨Examples
For records in the dataset where `text_length(text)` < 172.500, we found a significantly higher number of overconfident wrong predictions (178 samples, corresponding to 60.544217687074834% of the wrong predictions in the data slice).| text | text_length(text) | label | Predicted label |
|
|---|---|---|---|---|
| 432 | @user @user @user Antifa JV squad? | 34 | offensive | non-offensive (p = 0.95) |
| offensive (p = 0.05) | ||||
| 1138 | @user An idiot. Where the fuck do they get these people? | 57 | offensive | non-offensive (p = 0.95) |
| offensive (p = 0.05) | ||||
| 890 | @user You are an asshole! | 25 | offensive | non-offensive (p = 0.94) |
| offensive (p = 0.06) |
👉Underconfidence issues (1)
| Vulnerability | Level | Data slice | Metric | Transformation | Deviation |
|---|---|---|---|---|---|
| Underconfidence | medium 🟡 | avg_word_length(text) >= 4.156 |
Overconfidence rate = 0.024 | — | +17.22% than global |
🔍✨Examples
For records in your dataset where `avg_word_length(text)` >= 4.156, we found a significantly higher number of underconfident predictions (24 samples, corresponding to 2.4% of the predictions in the data slice).| text | avg_word_length(text) | label | Predicted label |
|
|---|---|---|---|---|
| 850 | @user @user @user . #Hypocrisy to see so called conservatives call out supposed sexual deviancy when just about every sexual political scandal in recent memory involves Republicans and it's really #homophobia #RoyMoore #Kavanaugh #JimJordan #MarkFoley #BobPackwood #ClarenceThomas #DonaldTrump | 6.73684 | offensive | non-offensive (p = 0.50) |
| offensive (p = 0.50) | ||||
| 622 | @user @user @user @user @user @user That’s right...he lies all day long and he is still terrible at it...anyone else would have mastered it by now...he’s definitely got 10000hr | 5.10345 | non-offensive | non-offensive (p = 0.50) |
| offensive (p = 0.50) | ||||
| 262 | @user you never were a slave. Spartacus was a slave and a heroic figure. You are neither. | 4.29412 | offensive | non-offensive (p = 0.50) |
| offensive (p = 0.50) |
Disclaimer: it's important to note that automated scans may produce false positives or miss certain vulnerabilities. We encourage you to review the findings and assess the impact accordingly.
💡 What's Next?
- Checkout the Giskard Space and improve your model.
- The Giskard community is always buzzing with ideas. 🐢🤔 What do you want to see next? Your feedback is our favorite fuel, so drop your thoughts in the community forum! 🗣️💬 Together, we're building something extraordinary.
🙌 Big Thanks!
We're grateful to have you on this adventure with us. 🚀🌟 Here's to more breakthroughs, laughter, and code magic! 🥂✨ Keep hugging that code and spreading the love! 💻 #Giskard #Huggingface #AISafety 🌈👏 Your enthusiasm, feedback, and contributions are what seek. 🌟 Keep being awesome!