README.md

---
license: cc-by-nc-4.0
language:
- ar
- cs
- de
- el
- en
- fr
- hi
- he
- it
- id
- ja
- ko
- nl
- fa
- pl
- pt
- ro
- ru
- es
- tr
- uk
- vi
- zh
---
# Model Checkpoints for Multilingual Machine-Generated Text Portion Detection

## Model Details

### Model Description
- Developed by: 1-800-SHARED-TASKS
- Funded by: Cohere's Research Compute Grant (July 2024)
- Model type: Transformer-based for multilingual text portion detection
- Languages (NLP): 23 languages (expanding to 102)
- License: Non-commercial; derivatives must remain non-commercial with proper attribution

### Model Sources
- **Code Repository:** [Github Placeholder]
- **Paper:** [ACL Anthology Placeholder]
- **Presentation:** [Multi-lingual Machine-Generated Text Portion(s) Detection](https://static1.squarespace.com/static/659ac5de66fdf20e1d607f2e/t/66d977a49597da76b6c260a1/1725527974250/MMGTD-Cohere.pdf)

## Uses
The dataset is suitable for machine-generated text portion detection, token classification tasks, and other linguistic tasks. The methods applied here aim to improve the accuracy of detecting which portions of text are machine-generated, particularly in multilingual contexts. The dataset could be beneficial for research and development in areas like AI-generated text moderation, natural language processing, and understanding the integration of AI in content generation.

## Training Details
The model was trained on a dataset consisting of approximately 330k text samples from LLMs Command-R-Plus (100k) and Aya-23-35B (230k). The dataset includes 10k samples per language for each LLM, with a distribution of 10% fully human-written texts, 10% entirely machine-generated texts, and 80% mixed cases.

## Evaluation

### Testing Data, Factors & Metrics
The model was evaluated on a multilingual dataset covering 23 languages. Metrics include Accuracy, Precision, Recall, and F1 Score at the word level (character level for Japanese and Chinese).

### Results
Here are the word-level metrics for each language and ** character-level metrics for Japanese (JPN) and Chinese (ZHO):

<table>
  <tr>
    <th>Language</th>
    <th>Accuracy</th>
    <th>Precision</th>
    <th>Recall</th>
    <th>F1 Score</th>
  </tr>
  <tr>
    <td>ARA</td>
    <td>0.923</td>
    <td>0.832</td>
    <td>0.992</td>
    <td>0.905</td>
  </tr>
  <tr>
    <td>CES</td>
    <td>0.884</td>
    <td>0.869</td>
    <td>0.975</td>
    <td>0.919</td>
  </tr>
  <tr>
    <td>DEU</td>
    <td>0.917</td>
    <td>0.895</td>
    <td>0.983</td>
    <td>0.937</td>
  </tr>
  <tr>
    <td>ELL</td>
    <td>0.929</td>
    <td>0.905</td>
    <td>0.984</td>
    <td>0.943</td>
  </tr>
  <tr>
    <td>ENG</td>
    <td>0.917</td>
    <td>0.818</td>
    <td>0.986</td>
    <td>0.894</td>
  </tr>
  <tr>
    <td>FRA</td>
    <td>0.927</td>
    <td>0.929</td>
    <td>0.966</td>
    <td>0.947</td>
  </tr>
  <tr>
    <td>HEB</td>
    <td>0.963</td>
    <td>0.961</td>
    <td>0.988</td>
    <td>0.974</td>
  </tr>
  <tr>
    <td>HIN</td>
    <td>0.890</td>
    <td>0.736</td>
    <td>0.975</td>
    <td>0.839</td>
  </tr>
  <tr>
    <td>IND</td>
    <td>0.861</td>
    <td>0.794</td>
    <td>0.988</td>
    <td>0.881</td>
  </tr>
  <tr>
    <td>ITA</td>
    <td>0.941</td>
    <td>0.906</td>
    <td>0.989</td>
    <td>0.946</td>
  </tr>
  <tr>
    <td>JPN**</td>
    <td>0.832</td>
    <td>0.747</td>
    <td>0.965</td>
    <td>0.842</td>
  </tr>
  <tr>
    <td>KOR</td>
    <td>0.937</td>
    <td>0.918</td>
    <td>0.992</td>
    <td>0.954</td>
  </tr>
  <tr>
    <td>NLD</td>
    <td>0.916</td>
    <td>0.872</td>
    <td>0.985</td>
    <td>0.925</td>
  </tr>
  <tr>
    <td>PES</td>
    <td>0.822</td>
    <td>0.668</td>
    <td>0.972</td>
    <td>0.792</td>
  </tr>
  <tr>
    <td>POL</td>
    <td>0.903</td>
    <td>0.884</td>
    <td>0.986</td>
    <td>0.932</td>
  </tr>
  <tr>
    <td>POR</td>
    <td>0.805</td>
    <td>0.679</td>
    <td>0.987</td>
    <td>0.804</td>
  </tr>
  <tr>
    <td>RON</td>
    <td>0.931</td>
    <td>0.924</td>
    <td>0.985</td>
    <td>0.953</td>
  </tr>
  <tr>
    <td>RUS</td>
    <td>0.885</td>
    <td>0.818</td>
    <td>0.971</td>
    <td>0.888</td>
  </tr>
  <tr>
    <td>SPA</td>
    <td>0.888</td>
    <td>0.809</td>
    <td>0.990</td>
    <td>0.890</td>
  </tr>
  <tr>
    <td>TUR</td>
    <td>0.849</td>
    <td>0.735</td>
    <td>0.981</td>
    <td>0.840</td>
  </tr>
  <tr>
    <td>UKR</td>
    <td>0.768</td>
    <td>0.637</td>
    <td>0.987</td>
    <td>0.774</td>
  </tr>
  <tr>
    <td>VIE</td>
    <td>0.866</td>
    <td>0.757</td>
    <td>0.975</td>
    <td>0.853</td>
  </tr>
  <tr>
    <td>ZHO**</td>
    <td>0.803</td>
    <td>0.698</td>
    <td>0.970</td>
    <td>0.814</td>
  </tr>
</table>

### **Results on unseen generators and domains**
- M4GT-Bench (includes partial cases) - 89.38% word level accuracy [ unseen generators, seen domains ]
- ETS Essays (only binary cases) - 99.21% overall accuracy [ unseen generators, unseen domains]
- RAID-Bench (binary cases with adversarial inputs) - TBA overall accuracy [ unseen generators, unseen domains ] 

## **Citation**
```
To Be Replaced to arxi preprint
@misc {1-800-shared-tasks_2024,
    authors       = { {Ram Mohan Rao Kadiyala, Siddartha Pullakhandam, Kanwal Mehreen, Ashay Srivastava, Subhasya TippaReddy, Arvind Reddy Bobbili, Drishti Sharma, Suraj Chandrashekhar, Modabbir Adeeb, Srinadh Vura } },
	title        = { MGTD-Checkpoints (v1) },
	year         = 2024,
	url          = { https://huggingface.co/1-800-SHARED-TASKS/MGTD-Checkpoints },
	doi          = { 10.57967/hf/3193 },
	publisher    = { Hugging Face }
}
```





## **Authors**

**Core Contributors**

- Ram Kadiyala [[contact@rkadiyala.com](mailto:contact@rkadiyala.com)]
- Siddartha Pullakhandam [[pullakh2@uwm.edu](mailto:pullakh2@uwm.edu)]
- Kanwal Mehreen [[kanwal@traversaal.ai](mailto:kanwal@traversaal.ai)]
- Ashay Srivastava [[ashays06@umd.edu](mailto:ashays06@umd.edu)]
- Subhasya TippaReddy [[subhasyat@usf.edu](mailto:subhasyat@usf.edu)]


**Extended Crew**
- Arvind Reddy Bobbili [[abobbili@cougarnet.uh.edu](mailto:abobbili@cougarnet.uh.edu)]
- Drishti Sharma [[drishtisharma96505@gmail.com](mailto:drishtisharma96505@gmail.com)]
- Suraj Chandrashekhar [[stelugar@umd.edu](mailto:stelugar@umd.edu)]
- Modabbir Adeeb [[madeeb@umd.edu](mailto:madeeb@umd.edu)]
- Srinadh Vura [[320106410055@andhrauniversity.edu.in](mailto:320106410055@andhrauniversity.edu.in)]


## **Contact**

 [![Gmail](https://img.shields.io/badge/Gmail-D14836?style=for-the-badge&logo=gmail&logoColor=white)](mailto:contact@rkadiyala.com)