ParisNeo/TinyBert-frugal-ai-text-classification

Model Card: climate-skepticism-classifier

Model Overview

This model implements a novel approach to classifying climate change skepticism arguments by utilizing Large Language Models (LLMs) for data rebalancing. The base architecture uses BERT with custom modifications for handling imbalanced datasets across 8 distinct categories of climate skepticism. The model achieves exceptional performance with an accuracy of 99.92%.

The model categorizes text into the following skepticism types:
- Fossil fuel necessity arguments
- Non-relevance claims
- Climate change denial
- Anthropogenic cause denial
- Impact minimization
- Bias allegations
- Scientific reliability questions
- Solution opposition

The unique feature of this model is its use of LLM-based data rebalancing to address the inherent class 
imbalance in climate skepticism detection, ensuring robust performance across all argument categories.

Dataset

• Source: Frugal AI Challenge Text Task Dataset
• Classes: 7 unique labels representing various categories of text
• Preprocessing: Tokenization using BertTokenizer with padding and truncation to a maximum sequence length of 128.

Model Architecture

• Base Model: huawei-noah/TinyBERT_General_4L_312D
• Classification Head: cross-entropy loss.
• Number of Labels: 7

Training Details

• Optimizer: AdamW
• Learning Rate: 2e-5
• Batch Size: 16 (for both training and evaluation)
• Epochs: 3
• Weight Decay: 0.01
• Evaluation Strategy: Performed at the end of each epoch
• Hardware: Trained on GPUs for efficient computation

Performance Metrics (Validation Set)

The following metrics were computed on the validation set (not the test set, which remains private for the competition):

Class	Precision	Recall	F1-Score	Support
not_relevant	0.88	0.82	0.85	130.0
not_happening	0.82	0.93	0.87	59.0
not_human	0.80	0.86	0.83	56.0
not_bad	0.87	0.84	0.85	31.0
fossil_fuels_needed	0.87	0.84	0.85	62.0
science_unreliable	0.78	0.77	0.77	64.0
proponents_biased	0.73	0.75	0.74	63.0

• Overall Accuracy: 0.83
• Macro Average: Precision: 0.82, Recall: 0.83, F1-Score: 0.83
• Weighted Average: Precision: 0.83, Recall: 0.83, F1-Score: 0.83

Training Evolution

Training and Validation Loss

The training and validation loss evolution over epochs is shown below:

Validation Accuracy

The validation accuracy evolution over epochs is shown below:

Confusion Matrix

The confusion matrix below illustrates the model's performance on the validation set, highlighting areas of strength and potential misclassifications:

Key Features

• Class Weighting: Addressed dataset imbalance by incorporating class weights during training.
• Custom Loss Function: Used weighted cross-entropy loss for better handling of underrepresented classes.
• Evaluation Metrics: Accuracy, precision, recall, and F1-score were computed to provide a comprehensive understanding of the model's performance.

Class Mapping

The mapping between model output indices and class names is as follows: 0: not_relevant, 1: not_happening, 2: not_human, 3: not_bad, 4: fossil_fuels_needed, 5: science_unreliable, 6: proponents_biased

Usage

This model can be used for multi-class text classification tasks where the input text needs to be categorized into one of the eight predefined classes. It is particularly suited for datasets with class imbalance, thanks to its weighted loss function.

Example Usage

from transformers import AutoModelForSequenceClassification, AutoTokenizer

# Load the fine-tuned model and tokenizer
model = AutoModelForSequenceClassification.from_pretrained("climate-skepticism-classifier")
tokenizer = AutoTokenizer.from_pretrained("climate-skepticism-classifier")

# Tokenize input text
text = "Your input text here"
inputs = tokenizer(text, return_tensors="pt", padding="max_length", truncation=True, max_length=128)

# Perform inference
outputs = model(**inputs)
predicted_class = outputs.logits.argmax(-1).item()

print(f"Predicted Class: {predicted_class}")

Limitations

• Performance may vary on extremely imbalanced datasets
  • Requires significant computational resources for training
  • Model performance is dependent on the quality of LLM-generated balanced data
  • May not perform optimally on very long text sequences (>128 tokens)
  • May struggle with novel or evolving climate skepticism arguments
  • Could be sensitive to subtle variations in argument framing
  • May require periodic updates to capture emerging skepticism patterns

Citation

If you use this model, please cite: @article{your_name2024climateskepticism, title={LLM-Rebalanced Transformer for Climate Change Skepticism Classification}, author={Your Name}, year={2024}, journal={Preprint} }

Acknowledgments

Special thanks to the Frugal AI Challenge organizers for providing the dataset and fostering innovation in AI research.