language:
- en
tags:
- bpo
- llama
- thudm
inference: false
Black-Box Prompt Optimization: Aligning Large Language Models without Model Training
- Repository: https://github.com/thu-coai/BPO
- Paper: https://arxiv.org/abs/2311.04155
- Data: https://huggingface.co/datasets/THUDM/BPO
Black-box Prompt Optimization (BPO)
BPO is a black-box alignment technique that differs from training-based methods (like PPO or DPO). BPO only requires training of a plug-and-play model and optimizes LLMs through optimizing user inputs. Therefore, it can be used on a variety of open-source or API-based LLMs.
Model Details
Data
Prompt优化模型由隐含人类偏好特征的prompt优化对训练得到，数据集的详细信息在这里。 The Prompt Optimization Model is trained on prompt optimization pairs which contain human preference features. Detailed information on the dataset can be found here.
Backbone Model
The prompt preference optimizer is built on Llama-2-7b-chat-hf.
Language
English
Performance
| Model A | Model B | A win | tie | B win |
|---|---|---|---|---|
| gpt-3.5-turbo + BPO | gpt-3.5-turbo | 60.0 | 8.7 | 31.3 |
| claude-2 + BPO | claude-2 | 57.5 | 5.0 | 37.5 |
| llama-2-13b-chat + BPO | llama-2-70b-chat | 61.3 | 0.0 | 38.7 |
| vicuna-13b + BPO | vicuna-13b + PPO | 52.5 | 3.7 | 43.7 |
| vicuna-13b + BPO | vicuna-13b + DPO | 53.8 | 2.5 | 43.7 |
| vicuna-13b + DPO + BPO | vicuna-13b + DPO | 60.0 | 2.5 | 37.5 |
Intended Use
Prompt Template
We adopt a prompt template as
[INST] You are an expert prompt engineer. Please help me improve this prompt to get a more helpful and harmless response:\n{user prompt} [/INST]
Inference code
Here is an example code for inference:
from transformers import AutoModelForCausalLM, AutoTokenizer
model_path = 'Your-Model-Path'
prompt_template = "[INST] You are an expert prompt engineer. Please help me improve this prompt to get a more helpful and harmless response:\n{} [/INST]"
model = AutoModelForCausalLM.from_pretrained(model_path).cuda()
tokenizer = AutoTokenizer.from_pretrained(model_path)
text = 'Tell me about Harry Potter'
prompt = prompt_template.format(text)
model_inputs = tokenizer(prompt, return_tensors="pt").to("cuda:0")
output = model.generate(**model_inputs, max_new_tokens=1024, do_sample=True, top_p=0.9, temperature=0.6, num_beams=1)
resp = tokenizer.decode(output[0], skip_special_tokens=True).split('[/INST]')[1].strip()
print(resp)
See our Github Repo for more detailed usage (e.g. more aggressive optimization).
Other Known Limitations
- Task coverage is not sufficient, as we only used open-source data to get about 14k optimized prompts. Clearly, it is impossible to cover a wide range of user queries, so the current model may not perform well on every prompt.
- Due to the small ratio of long-context-based tasks and mathematical problems, the prompt optimizer underperforms when dealing with these tasks.
Citation
If you find our model is useful in your work, please cite it with:
@article{cheng2023black,
title={Black-Box Prompt Optimization: Aligning Large Language Models without Model Training},
author={Cheng, Jiale and Liu, Xiao and Zheng, Kehan and Ke, Pei and Wang, Hongning and Dong, Yuxiao and Tang, Jie and Huang, Minlie},
journal={arXiv preprint arXiv:2311.04155},
year={2023}
}