🏆 GermEval 2025: Call to Action Detection (Class-Weighted)

abullardUR@GermEval Shared Task 2025 Submission

🎯 Model Summary

This model is a fine-tuned version of LSX-UniWue/ModernGBERT_134M, specifically designed for Call to Action (C2A) Detection in German social media content. It was developed as part of the GermEval 2025 Shared Task on Harmful Content Detection.

🏅 Competition Performance

Final Ranking: 4th out of 9 teams
Primary Metric (Macro-F1): 0.82 (+39% over official baseline (0.59 → 0.82))
Approach: Class-weighted cross-entropy loss to handle severe class imbalance (9.3:1 ratio)

📊 Task Details

Task Type: Binary classification
Classes: False (no call to action), True (call to action detected)
Domain: German social media (Twitter, 2014-2016)
Data Source: Right-wing extremist network posts

⚠️ Limitations and Bias

Known Limitations

Domain Specificity: Trained on 2014-2016 German Twitter data from right-wing extremist networks
Temporal Bias: Language patterns may not reflect contemporary usage
Class Imbalance: 90.3% negative vs 9.7% positive examples (9.3:1 ratio)
Cultural Context: May not generalize to other German-speaking regions or contexts
Implicit Context: May struggle with coded language and contextual references

Ethical Considerations

Model trained on potentially harmful content for research purposes only
Should not be used to amplify or generate harmful content
Requires careful handling due to sensitive training data

🚀 How to Use

Quick Start

from transformers import AutoProcessor, AutoModelForSequenceClassification

# Load model and processor
model_id = "abullard1/germeval2025-c2a-moderngbert-cw"
processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForSequenceClassification.from_pretrained(model_id, trust_remote_code=True).eval()

# Run inference
text = "Kommt alle zur Demo am Samstag!"
inputs = processor(text, return_tensors="pt", truncation=True)
probs = model(**inputs).logits.softmax(-1).detach().cpu().numpy()
print(f"Predictions: {probs}")

Class Labels

Label 0: No call to action detected
Label 1: Call to action detected

📈 Training Details

Training Data

Source: GermEval 2025 Shared Task C2A dataset
Size: 6,840 samples
Split: 80% training (5,472), 20% validation (1,368)
Class Distribution: 90.3% negative, 9.7% positive

Training Procedure

Base Model: ModernGBERT-134M (8192 token context)
Architecture: Mean-pooling classification head
Loss Function: Class-weighted cross-entropy (inverse frequency weighting)
Optimizer: AdamW with linear scheduling
Early Stopping: Patience of 5 epochs on validation Macro-F1

Hyperparameters

Learning Rate: 3e-5
Weight Decay: 0.0973
Batch Size: 8/32 (train/eval)
Epochs: 8
Warmup Steps: 500

📚 Citation

@inproceedings{bullard2025germeval,
  title   = {abullardUR@GermEval Shared Task 2025: Fine-tuning ModernGBERT on Highly Imbalanced German Social Media for Harmful Content Detection},
  author  = {Bullard, Samuel},
  year    = {2025},
  booktitle = {Proceedings of KONVENS 2025 Workshops}
}

🙏 Acknowledgments

GermEval 2025 Organizers: University of Stuttgart and University of Mannheim
Prof. Dr. Udo Kruschwitz (University of Regensburg) for supervision
ModernGBERT Team: LSX-UniWue for the ModernGBERT-134M German language base-model