|
--- |
|
license: cc-by-nc-4.0 |
|
language: |
|
- ar |
|
- cs |
|
- de |
|
- el |
|
- en |
|
- fr |
|
- hi |
|
- he |
|
- it |
|
- id |
|
- ja |
|
- ko |
|
- nl |
|
- fa |
|
- pl |
|
- pt |
|
- ro |
|
- ru |
|
- es |
|
- tr |
|
- uk |
|
- vi |
|
- zh |
|
--- |
|
# Model Checkpoints for Multilingual Machine-Generated Text Portion Detection |
|
|
|
## Model Details |
|
|
|
### Model Description |
|
- Developed by: 1-800-SHARED-TASKS |
|
- Funded by: Cohere's Research Compute Grant (July 2024) |
|
- Model type: Transformer-based for multilingual text portion detection |
|
- Languages (NLP): 23 languages (expanding to 102) |
|
- License: Non-commercial; derivatives must remain non-commercial with proper attribution |
|
|
|
### Model Sources |
|
- **Code Repository:** [Github Placeholder] |
|
- **Paper:** [ACL Anthology Placeholder] |
|
- **Presentation:** [Multi-lingual Machine-Generated Text Portion(s) Detection](https://static1.squarespace.com/static/659ac5de66fdf20e1d607f2e/t/66d977a49597da76b6c260a1/1725527974250/MMGTD-Cohere.pdf) |
|
|
|
## Uses |
|
The dataset is suitable for machine-generated text portion detection, token classification tasks, and other linguistic tasks. The methods applied here aim to improve the accuracy of detecting which portions of text are machine-generated, particularly in multilingual contexts. The dataset could be beneficial for research and development in areas like AI-generated text moderation, natural language processing, and understanding the integration of AI in content generation. |
|
|
|
## Training Details |
|
The model was trained on a dataset consisting of approximately 330k text samples from LLMs Command-R-Plus (100k) and Aya-23-35B (230k). The dataset includes 10k samples per language for each LLM, with a distribution of 10% fully human-written texts, 10% entirely machine-generated texts, and 80% mixed cases. |
|
|
|
## Evaluation |
|
|
|
### Testing Data, Factors & Metrics |
|
The model was evaluated on a multilingual dataset covering 23 languages. Metrics include Accuracy, Precision, Recall, and F1 Score at the word level (character level for Japanese and Chinese). |
|
|
|
### Results |
|
Here are the word-level metrics for each language and ** character-level metrics for Japanese (JPN) and Chinese (ZHO): |
|
|
|
<table> |
|
<tr> |
|
<th>Language</th> |
|
<th>Accuracy</th> |
|
<th>Precision</th> |
|
<th>Recall</th> |
|
<th>F1 Score</th> |
|
</tr> |
|
<tr> |
|
<td>ARA</td> |
|
<td>0.923</td> |
|
<td>0.832</td> |
|
<td style="background-color: #e0e0e0;">0.992</td> |
|
<td>0.905</td> |
|
</tr> |
|
<tr> |
|
<td>CES</td> |
|
<td>0.884</td> |
|
<td>0.869</td> |
|
<td style="background-color: #e0e0e0;">0.975</td> |
|
<td>0.919</td> |
|
</tr> |
|
<tr> |
|
<td>DEU</td> |
|
<td>0.917</td> |
|
<td>0.895</td> |
|
<td style="background-color: #e0e0e0;">0.983</td> |
|
<td>0.937</td> |
|
</tr> |
|
<tr> |
|
<td>ELL</td> |
|
<td>0.929</td> |
|
<td>0.905</td> |
|
<td style="background-color: #e0e0e0;">0.984</td> |
|
<td>0.943</td> |
|
</tr> |
|
<tr> |
|
<td>ENG</td> |
|
<td>0.917</td> |
|
<td>0.818</td> |
|
<td style="background-color: #e0e0e0;">0.986</td> |
|
<td>0.894</td> |
|
</tr> |
|
<tr> |
|
<td>FRA</td> |
|
<td>0.927</td> |
|
<td>0.929</td> |
|
<td style="background-color: #e0e0e0;">0.966</td> |
|
<td>0.947</td> |
|
</tr> |
|
<tr> |
|
<td>HEB</td> |
|
<td>0.963</td> |
|
<td>0.961</td> |
|
<td style="background-color: #e0e0e0;">0.988</td> |
|
<td>0.974</td> |
|
</tr> |
|
<tr> |
|
<td>HIN</td> |
|
<td>0.890</td> |
|
<td>0.736</td> |
|
<td style="background-color: #e0e0e0;">0.975</td> |
|
<td>0.839</td> |
|
</tr> |
|
<tr> |
|
<td>IND</td> |
|
<td>0.861</td> |
|
<td>0.794</td> |
|
<td style="background-color: #e0e0e0;">0.988</td> |
|
<td>0.881</td> |
|
</tr> |
|
<tr> |
|
<td>ITA</td> |
|
<td>0.941</td> |
|
<td>0.906</td> |
|
<td style="background-color: #e0e0e0;">0.989</td> |
|
<td>0.946</td> |
|
</tr> |
|
<tr> |
|
<td>JPN**</td> |
|
<td>0.832</td> |
|
<td>0.747</td> |
|
<td style="background-color: #e0e0e0;">0.965</td> |
|
<td>0.842</td> |
|
</tr> |
|
<tr> |
|
<td>KOR</td> |
|
<td>0.937</td> |
|
<td>0.918</td> |
|
<td style="background-color: #e0e0e0;">0.992</td> |
|
<td>0.954</td> |
|
</tr> |
|
<tr> |
|
<td>NLD</td> |
|
<td>0.916</td> |
|
<td>0.872</td> |
|
<td style="background-color: #e0e0e0;">0.985</td> |
|
<td>0.925</td> |
|
</tr> |
|
<tr> |
|
<td>PES</td> |
|
<td>0.822</td> |
|
<td>0.668</td> |
|
<td style="background-color: #e0e0e0;">0.972</td> |
|
<td>0.792</td> |
|
</tr> |
|
<tr> |
|
<td>POL</td> |
|
<td>0.903</td> |
|
<td>0.884</td> |
|
<td style="background-color: #e0e0e0;">0.986</td> |
|
<td>0.932</td> |
|
</tr> |
|
<tr> |
|
<td>POR</td> |
|
<td>0.805</td> |
|
<td>0.679</td> |
|
<td style="background-color: #e0e0e0;">0.987</td> |
|
<td>0.804</td> |
|
</tr> |
|
<tr> |
|
<td>RON</td> |
|
<td>0.931</td> |
|
<td>0.924</td> |
|
<td style="background-color: #e0e0e0;">0.985</td> |
|
<td>0.953</td> |
|
</tr> |
|
<tr> |
|
<td>RUS</td> |
|
<td>0.885</td> |
|
<td>0.818</td> |
|
<td style="background-color: #e0e0e0;">0.971</td> |
|
<td>0.888</td> |
|
</tr> |
|
<tr> |
|
<td>SPA</td> |
|
<td>0.888</td> |
|
<td>0.809</td> |
|
<td style="background-color: #e0e0e0;">0.990</td> |
|
<td>0.890</td> |
|
</tr> |
|
<tr> |
|
<td>TUR</td> |
|
<td>0.849</td> |
|
<td>0.735</td> |
|
<td style="background-color: #e0e0e0;">0.981</td> |
|
<td>0.840</td> |
|
</tr> |
|
<tr> |
|
<td>UKR</td> |
|
<td>0.768</td> |
|
<td>0.637</td> |
|
<td style="background-color: #e0e0e0;">0.987</td> |
|
<td>0.774</td> |
|
</tr> |
|
<tr> |
|
<td>VIE</td> |
|
<td>0.866</td> |
|
<td>0.757</td> |
|
<td style="background-color: #e0e0e0;">0.975</td> |
|
<td>0.853</td> |
|
</tr> |
|
<tr> |
|
<td>ZHO**</td> |
|
<td>0.803</td> |
|
<td>0.698</td> |
|
<td style="background-color: #e0e0e0;">0.976</td> |
|
<td>0.814</td> |
|
</tr> |
|
</table> |
|
|
|
## **Authors** |
|
|
|
**Core Contributors** |
|
|
|
- Ram Kadiyala [[contact@rkadiyala.com](mailto:contact@rkadiyala.com)] |
|
- Siddartha Pullakhandam [[pullakh2@uwm.edu](mailto:pullakh2@uwm.edu)] |
|
- Kanwal Mehreen [[kanwal@traversaal.ai](mailto:kanwal@traversaal.ai)] |
|
- Ashay Srivastava [[ashays06@umd.edu](mailto:ashays06@umd.edu)] |
|
- Subhasya TippaReddy [[subhasyat@usf.edu](mailto:subhasyat@usf.edu)] |
|
|
|
|
|
**Extended Crew** |
|
- Arvind Reddy Bobbili [[abobbili@cougarnet.uh.edu](mailto:abobbili@cougarnet.uh.edu)] |
|
- Suraj Chandrashekhar [[stelugar@umd.edu](mailto:stelugar@umd.edu)] |
|
- Modabbir Adeeb [[madeeb@umd.edu](mailto:madeeb@umd.edu)] |
|
- Drishti Sharma [ ] |
|
- Srinadh Vura [ ] |
|
|
|
|
|
## **Contact** |
|
|
|
[](mailto:contact@rkadiyala.com) |
