README.md

metadata

license: agpl-3.0
language:
  - en
  - zh
tags:
  - AI4S
  - MoE

SciDFM: Dialogue Foundation Model for Science

SciDFM is the pioneering open-sourced dialogue foundation model tailored for science, which integrates a mixture-of-experts architecture into a transformer-based framework, aiming at enhancing its sophisticated scientific reasoning and understanding capabilities. SciDFM achieves strong performance on general scientific benchmarks such as SciEval and SciQ, and it reachs a SOTA performance on domain-specific benchmark among models of similar size.

News

• 2024-06-28 The parameter of SciDFM-MoE-A5.6B-v1.0 is open-soursed! Technical report is coming soon.

Model Details

SciDFM is based on a transformer architecture, and follows modifications of Llama, i.e. RMSNorm, RoPE and SwiGLU. SciDFM use the same hyper-parameters of OpenLLaMa-3B. And in order to better model knowledge of different disciplines, we replace the feed-forward block with Mixture-of-Expert (MoE) layers.

Training Details

SciDFM is pre-trained on a large corpus containing ~300B science tokens and ~270B general tokens for two epochs, resulting in about 1.1T tokens consuming. And we further fine-tune SciDFM using ~9.3M instruction-following samples for 5 epochs to improve the performances on the downstream benchmarks.

Usage Details

Local Inference

To load and run SciDFM locally, here is an example:

import torch
from transformers import LlamaTokenizer, AutoModelForCausalLM

model_name_or_id = "OpenDFM/SciDFM-MoE-A5.6B-v1.0"
tokenizer = LlamaTokenizer.from_pretrained(model_name_or_id, use_fast=False)
model = AutoModelForCausalLM.from_pretrained(model_name_or_id, torch_dtype=torch.bfloat16, device_map="auto", trust_remote_code=True)

chat_template = "<|user|>:{instruction}<|assistant|>:"
input_text = "What is Mixture-of-Experts (MoE) in computer science?"
input_text = chat_template.format(instruction=input_text)

inputs = tokenizer(input_text, return_tensors="pt").to("cuda")
generation_config = GenerationConfig(
    do_sample=True,
    top_k=20,
    top_p=0.9,
    temperature=0.9,
    max_new_tokens=1024,
    eos_token_id=tokenizer.eos_token_id
)

outputs = model.generate(**inputs, generation_config=generation_config)
generated_text = tokenizer.decode(outputs, skip_special_tokens=True)[0][len(input_text):]
print(generated_text.strip())

SMILES preprocess

When there involves SMILES notation in your input, we recommend to preprocess the SMILES with the rdkit package to canonicalize the SMILES. Here is an example:

from rdkit import Chem
def canonicalize_smiles(smiles):
    mol = Chem.MolFromSmiles(smiles)
    if mol is None:
        return None
    return Chem.MolToSmiles(mol, isomericSmiles=True, kekuleSmiles=False)

or directly:

from rdkit import Chem
def canonicalize_smiles(smiles):
    return Chem.CanonSmiles(smiles, useChiral=True)

Special Tokens preprocess

If there is SMILES expression in your input, please first process it with the following function:

import sentencepiece as spm

smiles_model = spm.SentencePieceProcessor(model_file="smiles.model")

def convert_smiles(smiles_str):
   pieces = smiles_model.encode_as_pieces(smiles_str)[1:]
   smiles = "".join([f"[ChemDFM_Start_SMILES_Unit]{piece}[ChemDFM_End_SMILES_Unit]" for piece in pieces])
   return smiles

convert_smiles("C(C(=O)O)N")

And if there is protein sequece in your input, please first process it with the following function:

def convert_protein(p_str):
   res = [f"<<protein>>{s}" for s in p_str]
   return "".join(res)

convert_protein("MIRLGAPQTL")

Evaluation

We briefly compare SciDFM-MoE-A5.6B-v1.0 with similar-sized instruction-tuned LLMs on scientific evaluation benchmarks. The results are shown below:

Model	SciEval	SciQ	ARC_c	ARC_e	GSM8K	MATH	MedQA	MMCQA	PMQA	Avg
LLaMa2-7B	27.06	57.00	36.43	46.59	3.94	3.96	26.32	29.84	66.80	32.95
Galactica-6.7B	46.28	74.20	44.28	61.83	2.80	6.32	30.48	36.46	48.80	38.91
LLaMa2-13B	33.88	78.10	56.66	72.35	22.82	3.90	32.68	34.28	77.80	45.45
ChatGLM2-6B	54.25	75.80	57.08	73.57	25.09	7.18	27.42	34.21	60.40	45.94
Galactica-30B	54.24	83.10	57.85	75.04	13.65	8.66	37.71	48.43	58.80	48.35
LLaMa3-8B	59.70	90.00	71.16	84.05	5.91	7.00	48.78	52.74	26.60	49.59
ChatGLM3-6B	51.13	77.60	60.84	75.97	60.27	23.52	24.59	31.39	51.80	50.53
SciGLM-6B	61.22	88.70	77.47	86.57	42.23	16.40	42.81	44.94	73.60	59.12
SciDFM	62.48	88.00	64.76	81.48	59.14	27.28	44.54	53.10	78.00	61.56
ChatGLM3-6B-base	60.34	89.00	78.58	87.37	59.82	22.64	42.73	45.14	74.40	61.96
Llama3-8B-Instruct	64.91	91.60	76.45	87.33	76.57	26.26	56.48	59.31	72.00	67.44

Citation

@article{sun2024scidfm,
  title={SciDFM: A Large Language Model with Mixture-of-Experts for Science},
  author={Sun, Liangtai and Luo, Danyu and Ma, Da and Zhao, Zihan and Chen, Baocai and Shen, Zhennan and Zhu, Su and Chen, Lu and Chen, Xin and Yu, Kai},
  journal={arXiv preprint arXiv:2409.18412},
  year={2024}
}