Swallow-70b-RP-EX

Important Notice:

For personal and academic use only.

This is a merge of vocabulary expansion models, so tokenizer.model is not included. Users who want to convert to GGUL/GGML should be careful.

Description

This model is suitable for role-playing and storytelling, and has great multi-turn chat capabilities. This is probably due to the extremely high multi-turn chat performance of karakuri-ai/karakuri-lm-70b-chat-v0.1. Thank you for providing such a wonderful model.

This was created for personal and academic use only. This merge model uses only fine-tune models of Llama2, but some of the models used include those whose licenses for commercial use are unclear.

If there is a license problem, the rights holder should contact me directly. No license changes will be made due to contact from others.

In particular, karakuri-ai/karakuri-lm-70b-chat-v0.1 is currently distributed under cc-by-sa-4.0, and licensing discussions are taking place here. Since I am not a legal expert, I decided this license according to theirs.

Test environment

This model was tested using text-generation-webui. I use preset simple-1 and Null preset for Generation.

Recommendation

Use Null preset and modified temperature settings:

• temperature: 0.3
• top_p: 1.0
• repetition_penalty: 1.0
• top_k: 0

As a result of testing, lower temperature and smaller top_k may give better outputs.

Tested temperature Range

• temperature: 0.3 - 1.0

Tested repetition_penalty Range

• repetition_penalty: 1.0 - 1.15

Prompt template

Swallow Style (Alpaca format)

以下に、あるタスクを説明する指示があり、それに付随する入力が更なる文脈を提供しています。リクエストを適切に完了するための回答を記述してください。

### 指示:
{instruction}

### 応答:

Although not fully tested, karakuri-ai/karakuri-lm-70b-chat-v0.1, Doctor-Shotgun/lzlv-limarpv3-l2-70b and alac/Waxwing-Storytelling-70B-LoRA prompt styles are also available.

Use the instruct model

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

model_name = "nitky/Swallow-70b-RP-EX"

tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype=torch.bfloat16, low_cpu_mem_usage=True, device_map="auto", load_in_4bit = True)


PROMPT_DICT = {
    "prompt_input": (
        "以下に、あるタスクを説明する指示があり、それに付随する入力が更なる文脈を提供しています。"
        "リクエストを適切に完了するための回答を記述してください。\n\n"
        "### 指示:\n{instruction}\n\n### 入力:\n{input}\n\n### 応答:"

    ),
    "prompt_no_input": (
        "以下に、あるタスクを説明する指示があります。"
        "リクエストを適切に完了するための回答を記述してください。\n\n"
        "### 指示:\n{instruction}\n\n### 応答:"
    ),
}

def create_prompt(instruction, input=None):
    """
    Generates a prompt based on the given instruction and an optional input.
    If input is provided, it uses the 'prompt_input' template from PROMPT_DICT.
    If no input is provided, it uses the 'prompt_no_input' template.

    Args:
        instruction (str): The instruction describing the task.
        input (str, optional): Additional input providing context for the task. Default is None.

    Returns:
        str: The generated prompt.
    """
    if input:
        # Use the 'prompt_input' template when additional input is provided
        return PROMPT_DICT["prompt_input"].format(instruction=instruction, input=input)
    else:
        # Use the 'prompt_no_input' template when no additional input is provided
        return PROMPT_DICT["prompt_no_input"].format(instruction=instruction)

# Example usage
instruction_example = "以下のトピックに関する詳細な情報を提供してください。"
input_example = "東京工業大学の主なキャンパスについて教えてください"
prompt = create_prompt(instruction_example, input_example)

input_ids = tokenizer.encode(
    prompt,
    add_special_tokens=False,
    return_tensors="pt"
)

tokens = model.generate(
    input_ids.to(device=model.device),
    max_new_tokens=200,
    temperature=0.3,
    do_sample=True,
)

out = tokenizer.decode(tokens[0], skip_special_tokens=True)
print(out)

Merge Details

Merge Method

This model was merged using the DARE TIES and the SLERP merge method using tokyotech-llm/Swallow-70b-instruct-hf as a base.

Models Merged

The following models were included in the merge:

• karakuri-ai/karakuri-lm-70b-chat-v0.1
• GOAT-AI/GOAT-70B-Storytelling
• dreamgen/opus-v0.5-70b
• Doctor-Shotgun/lzlv-limarpv3-l2-70b
• [LoRA] alac/Waxwing-Storytelling-70B-LoRA

Configuration

Command example:

# please change the path and options according to your environment
mergekit-mega --cuda Swallow-70b-RP-EX.yml ~/text-generation-webui/models

The following YAML configuration was used to produce this model:

models:
  - model: nitky/Swallow-70b-RP
    # no parameters necessary for base model
  - model: karakuri-ai/karakuri-lm-70b-chat-v0.1
    parameters:
      density: 1
      weight:
      - filter: mlp
        value: 0.1
      - filter: self_attn
        value: 0.4
      - value: 0 # fallback for rest of tensors.
merge_method: dare_ties
base_model: nitky/Swallow-70b-RP
dtype: bfloat16
tokenizer_source: union
name: Swallow-70b-RP-EX-base
---
models:
  - model: nitky/Swallow-70b-RP
    # no parameters necessary for base model
  - model: karakuri-ai/karakuri-lm-70b-chat-v0.1
    parameters:
      density: 1
      weight:
      - filter: mlp
        value: [0.4, 0.1, 0.4, 0.1, 0.4, 0.1, 0.4, 0.1, 0.1]
      - filter: self_attn
        value: [0.4, 0.4, 0.1, 0.4, 0.1, 0.4, 0.1, 0.4, 0.4]
      - value: 0 # fallback for rest of tensors.
merge_method: dare_ties
base_model: nitky/Swallow-70b-RP
dtype: bfloat16
tokenizer_source: union
name: Swallow-70b-RP-EX-flavor
---
slices:
  - sources:
      - model: Swallow-70b-RP-EX-base
        layer_range: [0, 80]
      - model: Swallow-70b-RP-EX-flavor
        layer_range: [0, 80]
merge_method: slerp
base_model: Swallow-70b-RP-EX-base
parameters:
  t: # model stabilization
    - filter: self_attn
      value: [0, 0.5, 0.3, 0.7, 1]
    - filter: mlp
      value: [1, 0.5, 0.7, 0.3, 0]
    - value: 0.5 # fallback for rest of tensors
dtype: bfloat16
name: Swallow-70b-RP-EX