Model Checkpoints for Multilingual Machine-Generated Text Portion Detection

Model Details

Model Description

Developed by: 1-800-SHARED-TASKS
Funded by: Cohere's Research Compute Grant (July 2024)
Model type: Transformer-based for multilingual text portion detection
Languages (NLP): 23 languages (expanding to 102)
License: Non-commercial; derivatives must remain non-commercial with proper attribution

Model Sources

Code Repository: [Github Placeholder]
Paper: [ACL Anthology Placeholder]
Presentation: Multi-lingual Machine-Generated Text Portion(s) Detection

Uses

The dataset is suitable for machine-generated text portion detection, token classification tasks, and other linguistic tasks. The methods applied here aim to improve the accuracy of detecting which portions of text are machine-generated, particularly in multilingual contexts. The dataset could be beneficial for research and development in areas like AI-generated text moderation, natural language processing, and understanding the integration of AI in content generation.

Training Details

The model was trained on a dataset consisting of approximately 330k text samples from LLMs Command-R-Plus (100k) and Aya-23-35B (230k). The dataset includes 10k samples per language for each LLM, with a distribution of 10% fully human-written texts, 10% entirely machine-generated texts, and 80% mixed cases.

Evaluation

Testing Data, Factors & Metrics

The model was evaluated on a multilingual dataset covering 23 languages. Metrics include Accuracy, Precision, Recall, and F1 Score at the word level (character level for Japanese and Chinese).

Results

Here are the word-level metrics for each language and ** character-level metrics for Japanese (JPN) and Chinese (ZHO):

Language	Accuracy	Precision	Recall	F1 Score
ARA	0.923	0.832	0.992	0.905
CES	0.884	0.869	0.975	0.919
DEU	0.917	0.895	0.983	0.937
ELL	0.929	0.905	0.984	0.943
ENG	0.917	0.818	0.986	0.894
FRA	0.927	0.929	0.966	0.947
HEB	0.963	0.961	0.988	0.974
HIN	0.890	0.736	0.975	0.839
IND	0.861	0.794	0.988	0.881
ITA	0.941	0.906	0.989	0.946
JPN**	0.832	0.747	0.965	0.842
KOR	0.937	0.918	0.992	0.954
NLD	0.916	0.872	0.985	0.925
PES	0.822	0.668	0.972	0.792
POL	0.903	0.884	0.986	0.932
POR	0.805	0.679	0.987	0.804
RON	0.931	0.924	0.985	0.953
RUS	0.885	0.818	0.971	0.888
SPA	0.888	0.809	0.990	0.890
TUR	0.849	0.735	0.981	0.840
UKR	0.768	0.637	0.987	0.774
VIE	0.866	0.757	0.975	0.853
ZHO**	0.803	0.698	0.970	0.814

Results on unseen generators and domains

M4GT-Bench (includes partial cases) - 89.38% word level accuracy [ unseen generators, seen domains ]
ETS Essays (only binary cases) - 99.21% overall accuracy [ unseen generators, unseen domains]
RAID-Bench (binary cases with adversarial inputs) - TBA overall accuracy [ unseen generators, unseen domains ]

Citation

To Be Replaced to arxi preprint
@misc {1-800-shared-tasks_2024,
    authors       = { {Ram Mohan Rao Kadiyala, Siddartha Pullakhandam, Kanwal Mehreen, Ashay Srivastava, Subhasya TippaReddy, Arvind Reddy Bobbili, Drishti Sharma, Suraj Chandrashekhar, Modabbir Adeeb, Srinadh Vura } },
    title        = { MGTD-Checkpoints (v1) },
    year         = 2024,
    url          = { https://huggingface.co/1-800-SHARED-TASKS/MGTD-Checkpoints },
    doi          = { 10.57967/hf/3193 },
    publisher    = { Hugging Face }
}

Authors

Core Contributors

Ram Kadiyala [contact@rkadiyala.com]
Siddartha Pullakhandam [pullakh2@uwm.edu]
Kanwal Mehreen [kanwal@traversaal.ai]
Ashay Srivastava [ashays06@umd.edu]
Subhasya TippaReddy [subhasyat@usf.edu]

Extended Crew

Arvind Reddy Bobbili [abobbili@cougarnet.uh.edu]
Drishti Sharma [drishtisharma96505@gmail.com]
Suraj Chandrashekhar [stelugar@umd.edu]
Modabbir Adeeb [madeeb@umd.edu]
Srinadh Vura [320106410055@andhrauniversity.edu.in]

1-800-SHARED-TASKS
/

MGTD-Checkpoints