|
--- |
|
license: gemma |
|
--- |
|
## Introduction |
|
This repo contains Gemma-2-9b-Medical, a medical language model with 9 billion parameters. This model builds upon the foundation of Gemma-2-9b-base and has been tuned with diverse medical and general instructions. We also use the three strategies in the paper 'Efficient Continual Pre-training by Mitigating the Stability Gap' to mitigate the stability gap during instruction tuning, which boosts the model's medical task performance and reduces the computation consumption. |
|
|
|
## 💻 Usage |
|
|
|
```python |
|
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig |
|
import torch |
|
model_name = "YiDuo1999/Gemma-2-9b-medical" |
|
device_map = 'auto' |
|
model = AutoModelForCausalLM.from_pretrained( model_name, trust_remote_code=True,use_cache=False,device_map=device_map) |
|
tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True) |
|
tokenizer.pad_token = tokenizer.eos_token |
|
def askme(question): |
|
sys_message = ''' |
|
You are an AI Medical Assistant trained on a vast dataset of health information. Please be thorough and |
|
provide an informative answer. If you don't know the answer to a specific medical inquiry, advise seeking professional help. |
|
''' |
|
# Create messages structured for the chat template |
|
messages = [{"role": "system", "content": sys_message}, {"role": "user", "content": question}] |
|
|
|
# Applying chat template |
|
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True) |
|
inputs = tokenizer(prompt, return_tensors="pt").to("cuda") |
|
outputs = model.generate(**inputs, max_new_tokens=100, use_cache=True) |
|
|
|
# Extract and return the generated text, removing the prompt |
|
response_text = tokenizer.batch_decode(outputs)[0].strip() |
|
answer = response_text.split('<|im_start|>assistant')[-1].strip() |
|
return answer |
|
``` |
|
## 🏆 Evaluation |
|
For question-answering tasks, we have |
|
|
|
| Model | MMLU-Medical | PubMedQA | MedMCQA | MedQA-4-Option | Avg | |
|
|:-------------------------------|:-------------|:---------|:--------|:---------------|:-----| |
|
| Mistral-7B-instruct | 55.8 | 17.8 | 40.2 | 41.1 | 37.5 | |
|
| Zephyr-7B-instruct-β | 63.3 | 46.0 | 43.0 | 48.5 | 48.7 | |
|
| PMC-Llama-7B | 59.7 | 59.2 | 57.6 | 49.2 | 53.6 | |
|
| Medalpaca-13B | 55.2 | 50.4 | 21.2 | 20.2 | 36.7 | |
|
| AlpaCare-13B | 60.2 | 53.8 | 38.5 | 30.4 | 45.7 | |
|
| BioMedGPT-LM 7B | 52.0 | 58.6 | 34.9 | 39.3 | 46.2 | |
|
| Me-Llama-13B | - | 70.0 | 44.9 | 42.7 | - | |
|
| Llama-3-8B instruct | 82.0 | 74.6 | 57.1 | 60.3 | 68.5 | |
|
| JSL-Med-Sft-Llama-3-8B | 83.0 | 75.4 | 57.5 | 74.8 | 72.7 | |
|
| GPT-3.5-turbo-1106 | 74.0 | 72.6 | 34.9 | 39.3 | 60.6 | |
|
| GPT-4 | 85.5 | 69.2 | 69.5 | 83.9 | 77.0 | |
|
| Gemma-2-9b-int | 75.0 | 76.0 | 40.3 | 48.9 | 60.0 | |
|
| Gemma-2-9b-Medical | 75.0 | 76.0 | 61.3 | 59.7 | 68.0 | |
|
| Llama-3-physician-8B instruct | 80.0 | 76.0 | 80.2 | 60.3 | 74.1 | |
|
|
|
## Citation |
|
``` |
|
@inproceedings{Guo2024EfficientCP, |
|
title={Efficient Continual Pre-training by Mitigating the Stability Gap}, |
|
author={Yiduo Guo and Jie Fu and Huishuai Zhang and Dongyan Zhao and Yikang Shen}, |
|
year={2024}, |
|
url={https://api.semanticscholar.org/CorpusID:270688100} |
|
} |
|
``` |
