dmis-lab/llama-3-meerkat-70b-v1.0

Meerkat-70B (Version 1.0)

🚀 Meerkat-70B is a new instruction-tuned medical AI system of the Meerkat model family. The model was based on the Meta's Llama-3-70B-Instruct model and fine-tuned using our new synthetic dataset consisting of high-quality chain-of-thought reasoning paths sourced from 18 medical textbooks, along with diverse instruction-following datasets. This equips the model with high-level medical reasoning capabilities required for solving complex medical problems. For further insights into our model, please refer to our paper!

📄 Paper: Small Language Models Learn Enhanced Reasoning Skills from Medical Textbooks

Quick Start

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_id = "dmis-lab/llama-3-meerkat-70b-v1.0"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,  # You can choose to use this when there's not enough GPU memory available.
    device_map="auto",
)

# Multi-turn dialogue example
messages =[
    {"role": "system", "content": "You are a helpful doctor or healthcare professional. Guide the conversation to provide useful, complete, and scientifically-grounded answers to user questions. You have the option to compose a concise, single-turn conversation if the user's input is comprehensive to provide accurate answers. However, if essential details are missing, you should engage in a multi-turn dialogue, asking follow-up questions to gather a thorough medical history and records.\n\n"},
    {"role": "user", "content": "Hello, doctor. I'm really concerned about my 10-year-old son. We recently discovered a painless mass in his left testicle, so we brought him to the pediatrician."},
    {"role": "assistant", "content": "I understand your concern. Let's gather some more information. Has your son experienced any other symptoms along with the mass?"},
    {"role": "user", "content": "Other than the mass, my son hasn't shown any symptoms. He's been his usual self, playing and eating normally."}
]

input_ids = tokenizer.apply_chat_template(
    messages,
    add_generation_prompt=True,
    return_tensors="pt"
).to(model.device)

terminators = [
    tokenizer.eos_token_id,
    tokenizer.convert_tokens_to_ids("<|eot_id|>")
]

outputs = model.generate(
    input_ids,
    max_new_tokens=1000,
    eos_token_id=terminators,
    do_sample=True,
    temperature=0.7,
)
response = outputs[0][input_ids.shape[-1]:]
print(tokenizer.decode(response, skip_special_tokens=True))

Prompt Details

To reproduce the results reported in our paper, it is advisable to utilize the identical system messages used during model training. Please refer to the guidelines detailed below.

USMLE

When solving USMLE-style questions such as MedQA and MedBullets, use the following system message:

messages = [
    {"role": "system", "content": "The following is a multiple-choice question about medical knowledge. Solve this in a step-by-step fashion, starting by summarizing the available information. Output a single option from the given options as the final answer. You are strongly required to follow the specified output format; conclude your response with the phrase \"the answer is ([option_id]) [answer_string]\".\n\n"},
    {"role": "user", "content": "Two weeks after undergoing an emergency cardiac catherization with stenting for unstable angina pectoris, a 61-year-old man has decreased urinary output and malaise. He has type 2 diabetes mellitus and osteoarthritis of the hips. Prior to admission, his medications were insulin and naproxen. He was also started on aspirin, clopidogrel, and metoprolol after the coronary intervention. His temperature is 38\u00b0C (100.4\u00b0F), pulse is 93/min, and blood pressure is 125/85 mm Hg. Examination shows mottled, reticulated purplish discoloration of the feet. Laboratory studies show:\nHemoglobin count 14 g/dL\nLeukocyte count 16,400/mm3\nSegmented neutrophils 56%\nEosinophils 11%\nLymphocytes 31%\nMonocytes 2%\nPlatelet count 260,000/mm3\nErythrocyte sedimentation rate 68 mm/h\nSerum\nUrea nitrogen 25 mg/dL\nCreatinine 4.2 mg/dL\nRenal biopsy shows intravascular spindle-shaped vacuoles. Which of the following is the most likely cause of this patient's symptoms?\" (A) Renal papillary necrosis (B) Cholesterol embolization (C) Eosinophilic granulomatosis with polyangiitis (D) Polyarteritis nodosa"},
]

The model generates reasoning paths to solve the problem and then sequentially provides the predicted answers. Since the model ends its response with "the answer is," it is straightforward to extract the predicted answer for comparison with the actual answer.

Multiple-choice Exams

For other types of multiple-choice exams such as MedMCQA or MMLU, use the following simple system message:

messages = [
    {"role": "system", "content": "Answer the multiple-choice question about medical knowledge.\n\n"},
    {"role": "user", "content": "In a Robertsonian translocation fusion occurs at the: (A) telomeres. (B) centromeres. (C) histones. (D) ends of the long arms."},
]

Other Use Cases

Our model was trained using the AlpaCare instruction dataset comprising 52K examples, to enhance its generalization capabilities across diverse user prompts. Feel free to design and test your prompts and to share your thoughts with us, whether the model exceeds expectations or falls short!

Reproducing MedQA Performance with vLLM

Here is an example code for fast model evaluation in MedQA using vLLM. To adapt this code for other datasets like MedMCQA or MMLU, simply modify the instructions and update the dataset paths as needed.

# export CUDA_VISIBLE_DEVICES=0,1

import re
from datasets import load_dataset
from vllm import LLM, SamplingParams
USMLE_INSTRUCTION = (
    "The following is a multiple-choice question about medical knowledge. Solve this in"
    " a step-by-step fashion, starting by summarizing the available information. Output"
    " a single option from the given options as the final answer. You are strongly"
    " required to follow the specified output format; conclude your response with the"
    ' phrase "the answer is ([option_id]) [answer_string]".\n\n'
)
llm = LLM(
    model="dmis-lab/llama-3-meerkat-70b-v1.0",
    dtype="bfloat16",
    gpu_memory_utilization=0.9,
    max_model_len=2048,
    trust_remote_code=True,
    tensor_parallel_size=2
)

tokenizer = llm.get_tokenizer()

inputs, labels = [], []
for sample in load_dataset(
    "GBaker/MedQA-USMLE-4-options", split="test", trust_remote_code=True
):
    options = sorted(sample["options"].items())
    options = " ".join(map(lambda x: f"({x[0]}) {x[1]}", options))
    content = tokenizer.apply_chat_template(
        [{"role": "system", "content": USMLE_INSTRUCTION}, {"role": "user", "content": sample["question"] + " " + options}],
        add_generation_prompt=True,
        tokenize=False,
    )
    inputs.append(content)
    labels.append(sample["answer_idx"])

generated = llm.generate(
    inputs,
    SamplingParams(
        temperature=0.0,
        stop_token_ids=[tokenizer.vocab["<|eot_id|>"]],
        max_tokens=1024,
    ),
)
def extract_answer(text: str, options: str = "ABCD") -> str:
    return (re.findall(rf"he answer is \(([{options}])\)", text) or [options[0]])[-1]

correctness = []

for g, l in zip(generated, labels):
    correctness.append(extract_answer(g.outputs[0].text) == l)

print(sum(correctness) / len(correctness))

Evaluation

We tested models on seven medical benchmarks: MedQA, USMLE sample test, Medbullets-4, Medbullets-5 , MedMCQA, MMLU-Medical, and JAMA Clinical Challenge.

Model	Average	MedQA	USMLE	Medbullets-4	Medbullets-5	MedMCQA	MMLU-Medical
GPT-4	76.6	81.4	86.6	68.8	63.3	72.4	87.1
GPT-3.5	54.8	53.6	58.5	51.0	47.4	51.0	67.3
MediTron-70B (Ensemble, 5 runs)	-	70.2	-	-	-	66.0	78.0
MediTron-7B	51.0	50.2	44.6	51.1	45.5	57.9	56.7
BioMistral-7B	55.4	54.3	51.4	52.3	48.7	61.1	64.6
Meerkat-7B	62.6	70.6	70.3	58.7	52.9	60.6	70.5
Meerkat-8B (New)	67.3	74.0	74.2	62.3	55.5	62.7	75.2
Meerkat-70B (New)	77.9	82.6	87.4	71.4	65.3	73.9	86.9

Please note that the scores in MMLU-Medical were calculated based on the average accuracies across six medical-related subjects in the original MMLU benchmark, and each result for a single subject is presented below.

Model	Average	Cliniq Knowledge	Medical Genetics	Anatomy	Professional Medicine	College Biology	College Medicine
GPT-4	87.1	86.4	92.0	80.0	93.8	93.8	76.3
GPT-3.5	67.3	68.7	68.0	60.7	69.9	72.9	63.6
MediTron-70B (Ensemble, 5 runs)	78.0	75.5	85.9	69.4	82.3	86.7	68.0
MediTron-7B	56.7	57.7	63.8	56.9	56.0	57.1	48.9
BioMistral-7B	64.6	59.9	64.0	56.5	60.4	59.0	54.7
Meerkat-7B	70.5	71.6	74.8	63.2	77.3	70.8	65.2
Meerkat-8B (New)	75.2	74.3	76.7	74.8	75.3	76.1	74.3
Meerkat-70B (New)	86.9	87.2	88.2	84.4	87.2	87.9	86.6

Reference

Please see the information below to cite our paper.

@article{kim2024small,
  title={Small language models learn enhanced reasoning skills from medical textbooks},
  author={Kim, Hyunjae and Hwang, Hyeon and Lee, Jiwoo and Park, Sihyeon and Kim, Dain and Lee, Taewhoo and Yoon, Chanwoong and Sohn, Jiwoong and Choi, Donghee and Kang, Jaewoo},
  journal={arXiv preprint arXiv:2404.00376},
  year={2024}
}

Acknowledgement

Research supported with Cloud TPUs from Google’s TPU Research Cloud (TRC).

Contact

Feel free to email hyunjae-kim@korea.ac.kr if you have any questions.