--- license: cc-by-nc-4.0 language: - ar - cs - de - el - en - fr - hi - he - it - id - ja - ko - nl - fa - pl - pt - ro - ru - es - tr - uk - vi - zh --- # Model Checkpoints for Multilingual Machine-Generated Text Portion Detection ## Model Details ### Model Description - Developed by: 1-800-SHARED-TASKS - Funded by: Cohere's Research Compute Grant (July 2024) - Model type: Transformer-based for multilingual text portion detection - Languages (NLP): 23 languages (expanding to 102) - License: Non-commercial; derivatives must remain non-commercial with proper attribution ### Model Sources - **Code Repository:** [Github Placeholder] - **Paper:** [ACL Anthology Placeholder] - **Presentation:** [Multi-lingual Machine-Generated Text Portion(s) Detection](https://static1.squarespace.com/static/659ac5de66fdf20e1d607f2e/t/66d977a49597da76b6c260a1/1725527974250/MMGTD-Cohere.pdf) ## Uses The dataset is suitable for machine-generated text portion detection, token classification tasks, and other linguistic tasks. The methods applied here aim to improve the accuracy of detecting which portions of text are machine-generated, particularly in multilingual contexts. The dataset could be beneficial for research and development in areas like AI-generated text moderation, natural language processing, and understanding the integration of AI in content generation. ## Training Details The model was trained on a dataset consisting of approximately 330k text samples from LLMs Command-R-Plus (100k) and Aya-23-35B (230k). The dataset includes 10k samples per language for each LLM, with a distribution of 10% fully human-written texts, 10% entirely machine-generated texts, and 80% mixed cases. ## Evaluation ### Testing Data, Factors & Metrics The model was evaluated on a multilingual dataset covering 23 languages. Metrics include Accuracy, Precision, Recall, and F1 Score at the word level (character level for Japanese and Chinese). ### Results Here are the word-level metrics for each language and ** character-level metrics for Japanese (JPN) and Chinese (ZHO): <table> <tr> <th>Language</th> <th>Accuracy</th> <th>Precision</th> <th>Recall</th> <th>F1 Score</th> </tr> <tr> <td>ARA</td> <td>0.923</td> <td>0.832</td> <td>0.992</td> <td>0.905</td> </tr> <tr> <td>CES</td> <td>0.884</td> <td>0.869</td> <td>0.975</td> <td>0.919</td> </tr> <tr> <td>DEU</td> <td>0.917</td> <td>0.895</td> <td>0.983</td> <td>0.937</td> </tr> <tr> <td>ELL</td> <td>0.929</td> <td>0.905</td> <td>0.984</td> <td>0.943</td> </tr> <tr> <td>ENG</td> <td>0.917</td> <td>0.818</td> <td>0.986</td> <td>0.894</td> </tr> <tr> <td>FRA</td> <td>0.927</td> <td>0.929</td> <td>0.966</td> <td>0.947</td> </tr> <tr> <td>HEB</td> <td>0.963</td> <td>0.961</td> <td>0.988</td> <td>0.974</td> </tr> <tr> <td>HIN</td> <td>0.890</td> <td>0.736</td> <td>0.975</td> <td>0.839</td> </tr> <tr> <td>IND</td> <td>0.861</td> <td>0.794</td> <td>0.988</td> <td>0.881</td> </tr> <tr> <td>ITA</td> <td>0.941</td> <td>0.906</td> <td>0.989</td> <td>0.946</td> </tr> <tr> <td>JPN**</td> <td>0.832</td> <td>0.747</td> <td>0.965</td> <td>0.842</td> </tr> <tr> <td>KOR</td> <td>0.937</td> <td>0.918</td> <td>0.992</td> <td>0.954</td> </tr> <tr> <td>NLD</td> <td>0.916</td> <td>0.872</td> <td>0.985</td> <td>0.925</td> </tr> <tr> <td>PES</td> <td>0.822</td> <td>0.668</td> <td>0.972</td> <td>0.792</td> </tr> <tr> <td>POL</td> <td>0.903</td> <td>0.884</td> <td>0.986</td> <td>0.932</td> </tr> <tr> <td>POR</td> <td>0.805</td> <td>0.679</td> <td>0.987</td> <td>0.804</td> </tr> <tr> <td>RON</td> <td>0.931</td> <td>0.924</td> <td>0.985</td> <td>0.953</td> </tr> <tr> <td>RUS</td> <td>0.885</td> <td>0.818</td> <td>0.971</td> <td>0.888</td> </tr> <tr> <td>SPA</td> <td>0.888</td> <td>0.809</td> <td>0.990</td> <td>0.890</td> </tr> <tr> <td>TUR</td> <td>0.849</td> <td>0.735</td> <td>0.981</td> <td>0.840</td> </tr> <tr> <td>UKR</td> <td>0.768</td> <td>0.637</td> <td">0.987</td> <td>0.774</td> </tr> <tr> <td>VIE</td> <td>0.866</td> <td>0.757</td> <td>0.975</td> <td>0.853</td> </tr> <tr> <td>ZHO**</td> <td>0.803</td> <td>0.698</td> <td>0.970</td> <td>0.814</td> </tr> </table> ## **Authors** **Core Contributors** - Ram Kadiyala [[contact@rkadiyala.com](mailto:contact@rkadiyala.com)] - Siddartha Pullakhandam [[pullakh2@uwm.edu](mailto:pullakh2@uwm.edu)] - Kanwal Mehreen [[kanwal@traversaal.ai](mailto:kanwal@traversaal.ai)] - Ashay Srivastava [[ashays06@umd.edu](mailto:ashays06@umd.edu)] - Subhasya TippaReddy [[subhasyat@usf.edu](mailto:subhasyat@usf.edu)] **Extended Crew** - Arvind Reddy Bobbili [[abobbili@cougarnet.uh.edu](mailto:abobbili@cougarnet.uh.edu)] - Suraj Chandrashekhar [[stelugar@umd.edu](mailto:stelugar@umd.edu)] - Modabbir Adeeb [[madeeb@umd.edu](mailto:madeeb@umd.edu)] - Drishti Sharma [[drishtisharma96505@gmail.com](mailto:drishtisharma96505@gmail.com)] - Srinadh Vura [[320106410055@andhrauniversity.edu.in](mailto:320106410055@andhrauniversity.edu.in)] ## **Contact** [![Gmail](https://img.shields.io/badge/Gmail-D14836?style=for-the-badge&logo=gmail&logoColor=white)](mailto:contact@rkadiyala.com)