InMedData/InMD-X-RHE · Hugging Face

InMD-X: Large Language Models for Internal Medicine Doctors

We introduce InMD-X, a collection of multiple large language models specifically designed to cater to the unique characteristics and demands of Internal Medicine Doctors (IMD). InMD-X represents a groundbreaking development in natural language processing, offering a suite of language models fine-tuned for various aspects of the internal medicine field. These models encompass a wide range of medical sub-specialties, enabling IMDs to perform more efficient and accurate research, diagnosis, and documentation. InMD-X’s versatility and adaptability make it a valuable tool for improving the healthcare industry, enhancing communication between healthcare professionals, and advancing medical research. Each model within InMD-X is meticulously tailored to address specific challenges faced by IMDs, ensuring the highest level of precision and comprehensiveness in clinical text analysis and decision support.

(This model card is for the RHEUMATOLOGY subspecialty.)

Model Description

Model type: [CausalLM]
Language(s) (NLP): [English]
License: CC-BY-NC-SA
Finetuned from model [optional]: Intel/neural-chat-7b-v3-1

Model Sources [optional]

Paper [optional]: InMD-X

Uses

import torch
from peft import PeftModel, PeftConfig
from transformers import AutoModelForCausalLM, AutoTokenizer
peft_model_id = "InMedData/InMD-X-RHE"
config = PeftConfig.from_pretrained(peft_model_id)
model = AutoModelForCausalLM.from_pretrained(config.base_model_name_or_path, return_dict=True, load_in_8bit=True, device_map='auto')
tokenizer = AutoTokenizer.from_pretrained(config.base_model_name_or_path)
# Load the Lora model
model = PeftModel.from_pretrained(model, peft_model_id)
pipeline = transformers.pipeline(
    "text-generation",
    model=model,
    tokenizer = tokenizer,
    device_map="auto" # if you have GPU
)
def inference(pipeline, Qustion,answer_only = False):
    sequences = pipeline("Answer the next question in one sentence.\n"+
    Qustion,
    do_sample=True, 
    top_k=10,
    top_p = 0.9,
    temperature = 0.2,
    num_return_sequences=1,
    eos_token_id=tokenizer.eos_token_id,
    max_length=500, # can increase the length of sequence
    )
    
    Answers = []
    for seq in sequences:
        
        Answer = seq['generated_text'].split(Qustion)[-1].replace("\n","")
        Answers.append(Answer)
    return Answers
        
question = 'What is the association between long-term beta-blocker use after myocardial infarction (MI) and the risk of reinfarction and death?'
answers = inference(pipeline, question)
print(answers)

List of LoRA config

based on Parameter-Efficient Fine-Tuning (PEFT)

Parameter	PT	SFT
r	8	8
lora alpha	32	32
lora dropout	0.05	0.05
target	q, k, v, o,up, down, gate	q, k, v, o,up,down, gate

List of Training arguments

based on Transformer Reinforcement Learning (TRL)

Parameter	PT	SFT
train epochs	3	1
per device train batch size	1	1
optimizer	adamw_hf	adamw_hf
evaluation strategy	no	no
learning_rate	1e-4	1e-4

Experimental setup

Ubuntu 22.04.3 LTS
GPU - NVIDIA A100(40GB)
Python: 3.10.12
Pytorch:2.1.1+cu118
Transformer:4.37.0.dev0

Limitations

InMD-X consists of a collection of segmented models. The integration of the models has not yet been fully accomplished, resulting in each model being fragmented. Due to the absence of benchmarks, the segmented models have not been adequately evaluated. Future research will involve the development of new benchmarks and the integration of models to facilitate an objective evaluation.

Non-commercial use

These models are available exclusively for research purposes and are not intended for commercial use.

INMED DATA

INMED DATA is developing large language models (LLMs) specifically tailored for medical applications. For more information, please visit our website [TBD].

InMedData
/

InMD-X-RHE