File size: 3,568 Bytes
f7d01eb |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 |
---
language:
- en
tags:
- bpo
- llama
- thudm
inference: false
---
<h1>Black-Box Prompt Optimization: Aligning Large Language Models without Model Training</h1>
- **Repository:** https://github.com/thu-coai/BPO
- **Paper:** https://arxiv.org/abs/2311.04155
- **Data:** https://huggingface.co/datasets/THUDM/BPO
# Black-box Prompt Optimization (BPO)
BPO is a black-box alignment technique that differs from training-based methods (like PPO or DPO). BPO only requires training of a plug-and-play model and optimizes LLMs through optimizing user inputs. Therefore, it can be used on a variety of open-source or API-based LLMs.
## Model Details
### Data
Prompt优化模型由隐含人类偏好特征的prompt优化对训练得到，数据集的详细信息在这里。
The Prompt Optimization Model is trained on prompt optimization pairs which contain human preference features. Detailed information on the dataset can be found [here](https://huggingface.co/datasets/CCCCCC/BPO).
### Backbone Model
The prompt preference optimizer is built on `Llama-2-7b-chat-hf`.
### Language
English
### Performance
| Model A| Model B | A win | tie | B win |
|-------------|-------------|----|----|----|
| gpt-3.5-turbo + BPO | gpt-3.5-turbo | **60.0** | 8.7 | 31.3 |
| claude-2 + BPO | claude-2 | **57.5** | 5.0 | 37.5 |
| llama-2-13b-chat + BPO | llama-2-70b-chat | **61.3** | 0.0 | 38.7 |
| vicuna-13b + BPO | vicuna-13b + PPO | **52.5** | 3.7 | 43.7 |
| vicuna-13b + BPO | vicuna-13b + DPO | **53.8** | 2.5 | 43.7 |
| vicuna-13b + DPO + BPO | vicuna-13b + DPO | **60.0** | 2.5 | 37.5 |
## Intended Use
### Prompt Template
We adopt a prompt template as
```
[INST] You are an expert prompt engineer. Please help me improve this prompt to get a more helpful and harmless response:\n{user prompt} [/INST]
```
### Inference code
Here is an example code for inference:
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
model_path = 'Your-Model-Path'
prompt_template = "[INST] You are an expert prompt engineer. Please help me improve this prompt to get a more helpful and harmless response:\n{} [/INST]"
model = AutoModelForCausalLM.from_pretrained(model_path).cuda()
tokenizer = AutoTokenizer.from_pretrained(model_path)
text = 'Tell me about Harry Potter'
prompt = prompt_template.format(text)
model_inputs = tokenizer(prompt, return_tensors="pt").to("cuda:0")
output = model.generate(**model_inputs, max_new_tokens=1024, do_sample=True, top_p=0.9, temperature=0.6, num_beams=1)
resp = tokenizer.decode(output[0], skip_special_tokens=True).split('[/INST]')[1].strip()
print(resp)
```
See our [Github Repo](https://github.com/thu-coai/BPO/blob/main/src/infer_example.py) for more detailed usage (e.g. more aggressive optimization).
### Other Known Limitations
- Task coverage is not sufficient, as we only used open-source data to get about 14k optimized prompts. Clearly, it is impossible to cover a wide range of user queries, so the current model may not perform well on every prompt.
- Due to the small ratio of long-context-based tasks and mathematical problems, the prompt optimizer underperforms when dealing with these tasks.
## Citation
If you find our model is useful in your work, please cite it with:
```
@article{cheng2023black,
title={Black-Box Prompt Optimization: Aligning Large Language Models without Model Training},
author={Cheng, Jiale and Liu, Xiao and Zheng, Kehan and Ke, Pei and Wang, Hongning and Dong, Yuxiao and Tang, Jie and Huang, Minlie},
journal={arXiv preprint arXiv:2311.04155},
year={2023}
}
``` |