HachiML/SkillTree-Math-OpenMath-Mistral-7B-v0.1

SkillTree Model Collection

Applying a skill to your model with SkillTree is akin to unlocking a new ability in a video game's skill tree. Just as you would enhance your character's capabilities by selecting and activating specific skills, you can augment your model's abilities by integrating specialized skills. Follow these steps to imbue your model with new prowess, enhancing its performance and versatility in a straightforward and intuitive manner.
Please note that SkillTree abilities may not function in all cases. To determine whether a specific skill is operational, refer to the Functionality Status.

What is SkillTree?

SkillTree represents a set of model weights derived from further pre-training or fine-tuning Large Language Models (LLMs) to extract specific capabilities, such as code generation or chatting abilities. These extracted "skills" can be combined with a specific LLM base model to enhance its capabilities. The concept is inspired by ChatVector, aiming to modularize and transfer distinct skills across models.

SkillTree Details

• Functionality Status: Functional / Non-Functional / Not Verified
• Base Model: mistralai/Mistral-7B-v0.1
• Skill Model: nvidia/OpenMath-Mistral-7B-v0.1-hf
• Enhanced Model(optional): HachiML/Swallow-MS-7b-v0.1-MathSkill-OpenMath
• Skill type: Math Reasoning

Uses

Limitation

• Model Architecture: Mistral
• Model Size: 7.24B
• Compatible Models[optional]:

How to Apply Skill (Example)

# Import library
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

# Load the target model to be applied skill
base_model_name = "tokyotech-llm/Swallow-MS-7b-v0.1"
tokenizer = AutoTokenizer.from_pretrained(base_model_name)
model = AutoModelForCausalLM.from_pretrained(
    base_model_name,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

# Load SkillTree
skill_tree = AutoModelForCausalLM.from_pretrained(
    "HachiML/SkillTree-Math-OpenMath-Mistral-7B-v0.1",
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

# Apply the skill to the target model
def apply_skill(model, skill_tree):
    # excluded object
    skip_layers = ["model.embed_tokens.weight", "model.norm.weight", "lm_head.weight"]
    # apply skill
    for k, v in model.state_dict().items():
        # layernorm is also excluded
        if (k in skip_layers) or ("layernorm" in k):
            continue
        vector = skill_tree.state_dict()[k]
        new_v = v + vector.to(v.device)
        v.copy_(new_v)
    return model

model = apply_skill(model, skill_tree)

# Push to hub
model_name = "HachiML/Swallow-MS-7b-v0.1-MathSkill-OpenMath"
tokenizer.save_pretrained(f"./models/{model_name}", repo_id=model_name, push_to_hub=True)
model.save_pretrained(f"./models/{model_name}", repo_id=model_name, push_to_hub=True)