Update README.md

README.md

@@ -1,13 +1,144 @@
----
-title: DeepfakeTextDetect
-emoji: 💻
-colorFrom: yellow
-colorTo: blue
-sdk: gradio
-sdk_version: 3.35.2
-app_file: app.py
-pinned: false
-license: mit
----
-
-Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
+<div align="center">
+<h1>Deepfake Text Detection in the Wild</h1>
+</div>
+
+<div align="center">
+<img src="https://img.shields.io/badge/Version-1.0.0-blue.svg" alt="Version"> 
+<img src="https://img.shields.io/badge/License-CC%20BY%204.0-green.svg" alt="License">
+<img src="https://img.shields.io/github/stars/yafuly/DeepfakeTextDetect?color=yellow" alt="Stars">
+<img src="https://img.shields.io/github/issues/yafuly/DeepfakeTextDetect?color=red" alt="Issues">
+
+
+
+
+<!-- **Authors:** -->
+<br>
+
+_**Yafu Li<sup>†</sup><sup>‡</sup>, Qintong Li<sup>§</sup>, Leyang Cui<sup>¶</sup>, Wei Bi<sup>¶</sup>,<br>**_
+
+_**Longyue Wang<sup>¶</sup>, Linyi Yang<sup>‡</sup>, Shuming Shi<sup>¶</sup>, Yue Zhang<sup>‡</sup><br>**_
+
+
+<!-- **Affiliations:** -->
+
+
+_<sup>†</sup> Zhejiang University,
+<sup>‡</sup> Westlake University,
+<sup>§</sup> The University of Hong Kong,
+<sup>¶</sup> Tencent AI Lab_
+
+
+Presenting a comprehensive benchmark dataset designed to assess the proficiency of deepfake detectors amidst real-world scenarios.
+</div>
+
+## 📌 Table of Contents
+- [Introduction](#-introduction)
+- [Activities](#-activities)
+- [Dataset Description](#-dataset-description)
+- [Try Detection](#computer--try-detection)
+- [How to Get the Data](#-how-to-get-the-data)
+- [Citation](#-citation)
+<!-- - [Contributing](#-contributing) -->
+
+## 🚀 Introduction
+Recent advances in large language models have enabled them to reach a level of text generation comparable to that of humans. 
+These models show powerful capabilities across a wide range of content, including news article writing, story generation, and scientific writing.
+Such capability further narrows the gap between human-authored and machine-generated texts, highlighting the importance of deepfake text detection to avoid potential risks such as fake news propagation and plagiarism.
+In practical scenarios, the detector faces texts from various domains or LLMs without knowing their sources.
+
+To this end, we build **a wild testbed for deepfake text detection**, by gathering texts from various human writings and deepfake texts generated by different LLMs. 
+This repository contains the data to testify deepfake detection methods described in our paper, [Deepfake Text Detection in the Wild](https://arxiv.org/abs/2305.13242). 
+Welcome to test your detection methods on our testbed!
+
+## 📅 Activities
+
+- **May 25, 2023**: Initial dataset release including texts from 10 domains and 27 LLMs, contributing to 6 testbeds with increasing detection difficulty.
+- 🎉 **June 19, 2023**: Update two 'wilder' testbeds! We go one step wilder by constructing an additional testset with texts from unseen domains generated by an unseen model, to testify the detection ability in more practical scenarios. 
+We consider four new datasets: CNN/DailyMail, DialogSum, PubMedQA and IMDb to test the detection of deepfake news, deepfake dialogues, deepfake scientific answers and deepfake movie reviews.
+We sample 200 instances from each dataset and use a newly developed LLM, i.e., GPT-4, with specially designed prompts to create deepfake texts, establishing an "Unseen Domains & Unseen Model" scenario.
+Previous work demonstrates that detection methods are vulnerable to being deceived by target texts.
+Therefore, we also paraphrase each sentence individually for both human-written and machine-generated texts, forming an even more challenging testbed. 
+We adopt gpt-3.5-trubo as the zero-shot paraphraser and consider all paraphrased texts as machine-generated.
+
+## 📝 Dataset Description
+
+The dataset consists of **447,674** human-written and machine-generated texts from a wide range of sources in the wild:
+- Human-written texts from **10 datasets** covering a wide range of writing tasks, e.g., news article writing, story generation, scientific writing, etc.
+- Machine-generated texts generated by **27 mainstream LLMs** from 7 sources, e.g., OpenAI, LLaMA, and EleutherAI, etc.
+- **6 systematic testbed**s with increasing wildness and detection difficulty.
+- **2 wilder test sets**: (1) texts collected from new datasets and generated by GPT-4; (2) paraphrased texts.
+
+
+<!-- - **Size**: X GB
+- **Number of Records**: X
+- **Time Span**: YYYY-MM-DD to YYYY-MM-DD
+
+Here's a brief overview of the types of data included:
+
+1. **Feature 1**: Description of feature 1.
+2. **Feature 2**: Description of feature 2.
+3. **Feature 3**: Description of feature 3.
+... -->
+
+
+
+
+
+
+## :computer:  Try Detection
+
+### Model Access
+Our Longformer detector, which has been trained on the entire dataset, is now accessible through [Huggingface](https://huggingface.co/nealcly/detection-longformer). Additionally, you can try detection directly using our [online demo](https://huggingface.co/spaces/yaful/DeepfakeTextDetect).
+
+### Deployment
+We have refined the threshold based on out-of-distribution settings. To ensure optimal performance, we recommend preprocessing texts before sending them to the detector. 
+```python
+import torch
+import os
+from transformers import AutoModelForSequenceClassification,AutoTokenizer
+from deployment import preprocess, detect
+
+# init
+device = 'cpu' # use 'cuda:0' if GPU is available
+model_dir = "nealcly/detection-longformer"
+tokenizer = AutoTokenizer.from_pretrained(model_dir)
+model = AutoModelForSequenceClassification.from_pretrained(model_dir).to(device)
+
+# preprocess
+text = preprocess(text)
+# detection
+result = detect(text,tokenizer,model,device)
+```
+
+### Detection Performance
+#### In-distribution Detection
+| Testbed | HumanRec | MachineRec | AvgRec | AUROC|
+|------------------------------|------------------------------|------------------------------|------------------------------|------------------------------|
+| Domain-specific & Model-specific | 97.30%  | 95.91% | 96.60% | 0.99 | 
+| Cross-domains & Model–specific | 95.25% | 96.94% | 96.10% | 0.99 |
+| Domain-specific & Cross-models | 89.78% | 97.24% | 93.51% | 0.99 |
+| Cross-domains & Cross-models | 82.80%  | 98.27% | 90.53% | 0.99 |
+#### Out-of-distribution Detection 
+| Testbed | HumanRec | MachineRec | AvgRec | AUROC|
+|------------------------------|------------------------------|------------------------------|------------------------------|------------------------------|
+| Unseen Model Sets |  86.09% | 89.15% | 87.62% | 0.95 |
+| Unseen Domains |  82.88% | 80.50% | 81.78% | 0.93 |
+#### Wilder Testsets
+| Testbed | HumanRec | MachineRec | AvgRec | AUROC|
+|------------------------------|------------------------------|------------------------------|------------------------------|------------------------------|
+| Unseen Domains & Unseen Model|88.78% |84.12% |86.54%|0.94|
+| Paraphrase| 88.78% |37.05% |62.92% |0.75|
+
+
+## 📥 How to Get the Data
+### Download
+The dataset is available for download at：
+1. [Google Drive](https://drive.google.com/drive/folders/1p09vDiEvoA-ZPmpqkB2WApcwMQWiiMRl?usp=sharing)
+2. [Tencent Weiyun](https://share.weiyun.com/JUWQxF4H)
+
+The folder contains 4 packages:
+1. source.zip: Source texts of human-written texts and corresponding texts generated by LLMs.
+2. processed.zip: This is a refined version of the "source" that filters out low-quality texts and specifies sources as CSV file names. For example, the "cmv_machine_specified_gpt-3.5-trubo.csv" file contains texts from the CMV domain generated by the "gpt-3.5-trubo" model using specific prompts, while "cmv_human" includes human-written CMV texts. We suggest trying this version to test your detection methods.
+3. testbeds_processed.zip: 6 testbeds based on the ''processed'' version, which can be directly used for detecting in-distribution and out-of-distribution detection performance.
+4. wilder_testsets.zip: 2 wilder test sets with texts processed, aiming for (1) detecting deepfake text generated by GPT-4, and (2) detecting deepfake text in paraphrased versions.
+<!-- # 🤝 Contributing -->