metadata

library_name: transformers
tags: []

DiTy/gemma-2-9b-it-function-calling

This model is a fine-tuned version of google/gemma-2-9b-it for the Function Calling task on non-synthetic data, fully annotated by humans only, on the English version of the DiTy/function-calling dataset.

Model card tree

How prepare your functions (tools) for Function Calling
Just use chat template for generation
Prompt structure and expected content

Usage (HuggingFace Transformers)

Below we share some code snippets on how to get quickly started with running the model. First, install the Transformers library with:

pip install -U transformers

Prepare your functions for Function Calling

You should write the functions (tools) used by the model in Python code and make sure to add Python docstrings as in the example below:

def get_weather(city: str):
    """
    A function that returns the weather in a given city.
    
    Args:
        city: The city to get the weather for.
    """
    import random
    
    return "sunny" if random.random() > 0.5 else "rainy"


def get_sunrise_sunset_times(city: str):
    """
    A function that returns the time of sunrise and sunset at the present moment, for a given city, in the form of a list: [sunrise_time, sunset_time].
    
    Args:
        city: The city to get the sunrise and sunset times for.
    """
    
    return ["6:00 AM", "6:00 PM"]

Just use chat template

Next, you need to download the model and tokenizer:

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained(
    "DiTy/gemma-2-9b-it-function-calling",
    device_map="auto",
    torch_dtype=torch.bfloat16,  # use float16 or float32 if bfloat16 is not available to you.
    cache_dir=PATH_TO_MODEL_DIR,  # optional
)
tokenizer = AutoTokenizer.from_pretrained(
    "DiTy/gemma-2-9b-it-function-calling",
    cache_dir=PATH_TO_MODEL_DIR,  # optional
)

To get the result of generation, just use apply_chat_template. In order to take into account our written functions (tools), we need to pass them as a list through the tools attribute and also use add_prompt_generation=True.

history_messages = [
    {"role": "system", "content": "You are a helpful assistant with access to the following functions. Use them if required - "},
    {"role": "user", "content": "Hi, can you tell me the time of sunrise in Los Angeles?"},
]

inputs = tokenizer.apply_chat_template(
    history_messages,
    tokenize=False,
    add_generation_prompt=True,  # adding prompt for generation
    tools=[get_weather, get_sunrise_sunset_times],  # our functions (tools)
)

Then our inputs will look like this:

<bos><start_of_turn>user
You are a helpful assistant with access to the following functions. Use them if required - {
    "name": "get_weather",
    "description": "A function that returns the weather in a given city.",
    "parameters": {
        "type": "object",
        "properties": {
            "city": {
                "type": "string",
                "description": "The city to get the weather for."
            }
        },
        "required": [
            "city"
        ]
    }
},
{
    "name": "get_sunrise_sunset_times",
    "description": "A function that returns the time of sunrise and sunset at the present moment, for a given city, in the form of a list: [sunrise_time, sunset_time].",
    "parameters": {
        "type": "object",
        "properties": {
            "city": {
                "type": "string",
                "description": "The city to get the sunrise and sunset times for."
            }
        },
        "required": [
            "city"
        ]
    }
}

Hi, can you tell me the time of sunrise in Los Angeles?<end_of_turn>
<start_of_turn>model

Now we can generate a model's response. Be careful because, after apply_chat_template, there is no need to add special tokens during tokenization. So, use add_special_tokens=False:

terminator_ids = [
    tokenizer.eos_token_id,
    tokenizer.convert_tokens_to_ids("<end_of_turn>"),
]

prompt_ids =  tokenizer.encode(inputs, add_special_tokens=False, return_tensors='pt').to(model.device)
generated_ids = model.generate(
    prompt_ids,
    max_new_tokens=512,
    eos_token_id=terminator_ids,
    bos_token_id=tokenizer.bos_token_id,
)
generated_response = tokenizer.decode(generated_ids[0][prompt_ids.shape[-1]:], skip_special_tokens=False)  # `skip_special_tokens=False` for debug

We get the generation as a function call:

Function call: {"name": "get_sunrise_sunset_times", "arguments": {"city": "Los Angeles"}}<end_of_turn>

Great, now we can pick up and process the results with our called function, and then provide the model with the function's response:

history_messages = [
    {"role": "system", "content": "You are a helpful assistant with access to the following functions. Use them if required - "},
    {"role": "user", "content": "Hi, can you tell me the time of sunrise in Los Angeles?"},
    {"role": "function-call", "content": '{"name": "get_sunrise_sunset_times", "arguments": {"city": "Los Angeles"}}'},
    {"role": "function-response", "content": '{"times_list": ["6:00 AM", "6:00 PM"]}'},  # a hypothetical response from our function
]

inputs = tokenizer.apply_chat_template(
    history_messages,
    tokenize=False,
    add_generation_prompt=True,  # adding prompt for generation
    tools=[get_weather, get_sunrise_sunset_times],  # our functions (tools)
)

Let's make sure the inputs are correct:

<bos><start_of_turn>user
You are a helpful assistant with access to the following functions. Use them if required - {
    "name": "get_weather",
    "description": "A function that returns the weather in a given city.",
    "parameters": {
        "type": "object",
        "properties": {
            "city": {
                "type": "string",
                "description": "The city to get the weather for."
            }
        },
        "required": [
            "city"
        ]
    }
},
{
    "name": "get_sunrise_sunset_times",
    "description": "A function that returns the time of sunrise and sunset at the present moment, for a given city, in the form of a list: [sunrise_time, sunset_time].",
    "parameters": {
        "type": "object",
        "properties": {
            "city": {
                "type": "string",
                "description": "The city to get the sunrise and sunset times for."
            }
        },
        "required": [
            "city"
        ]
    }
}

Hi, can you tell me the time of sunrise in Los Angeles?<end_of_turn>
<start_of_turn>model
Function call: {"name": "get_sunrise_sunset_times", "arguments": {"city": "Los Angeles"}}<end_of_turn>
<start_of_turn>user
Function response: {"times_list": ["6:00 AM", "6:00 PM"]}<end_of_turn>
<start_of_turn>model

Similarly, we generate a response from the model:

prompt_ids =  tokenizer.encode(inputs, add_special_tokens=False, return_tensors='pt').to(model.device)
generated_ids = model.generate(
    prompt_ids,
    max_new_tokens=512,
    eos_token_id=terminator_ids,
    bos_token_id=tokenizer.bos_token_id,
)
generated_response = tokenizer.decode(generated_ids[0][prompt_ids.shape[-1]:], skip_special_tokens=False)  # `skip_special_tokens=False` for debug

As a result, we get the model's response:

The sunrise time in Los Angeles is 6:00 AM.<end_of_turn>

Usage via transformers `pipeline`

Generation via pipeline

from transformers import pipeline


generation_pipeline = pipeline(
    "text-generation",
    model="DiTy/gemma-2-9b-it-function-calling",
    model_kwargs={
        "torch_dtype": torch.bfloat16,  # use float16 or float32 if bfloat16 is not supported for you. 
        "cache_dir": PATH_TO_MODEL_DIR,  # OPTIONAL
    },
    device_map="auto",
)

history_messages = [
    {"role": "system", "content": "You are a helpful assistant with access to the following functions. Use them if required - "},
    {"role": "user", "content": "Hi, can you tell me the time of sunrise in Los Angeles?"},
    {"role": "function-call", "content": '{"name": "get_sunrise_sunset_times", "arguments": {"city": "Los Angeles"}}'},
    {"role": "function-response", "content": '{"times_list": ["6:00 AM", "6:00 PM"]}'},
]

inputs = generation_pipeline.tokenizer.apply_chat_template(
    history_messages,
    tokenize=False,
    add_generation_prompt=True,
    tools=[get_weather, get_sunrise_sunset_times],
)

terminator_ids = [
    generation_pipeline.tokenizer.eos_token_id,
    generation_pipeline.tokenizer.convert_tokens_to_ids("<end_of_turn>")
]

outputs = generation_pipeline(
    inputs,
    max_new_tokens=512,
    eos_token_id=terminator_ids,
)

print(outputs[0]["generated_text"][len(inputs):])

Prompt structure and expected content

For the most correct operation of the model, it is assumed that apply_chat_template will be used. It is necessary to transmit the message history in a certain format.

history_messages = [
    {"role": "...", "content": "..."},
    ...
]

The following roles are available for use:

system - an optional role, its content is always placed at the very beginning and before listing the functions available to the model (tools). You can always use the standard option that was used during the training: "You are a helpful assistant with access to the following functions. Use them if required - "
user - the user's request is transmitted through this role.
function-call - The body of the function call is passed through this role. Although the model is trained to generate a function call in the form of "Function call: {...}<end_of_turn>", you should still pass only the body "{...}" to the "content" field, since using apply_chat_template, the postscript in the instructions is added automatically.
function-response - in this role, we must pass the response of our function in the "content" field as a dictionary '{"name_returnable_value": value}'.
model - the content under this role is considered to be the generated text of the model.

Chat history with Function Calling

[
    {"role": "system", "content": "You are a helpful assistant with access to the following functions. Use them if required - "},
    {"role": "user", "content": "Hi, can you tell me the time of sunrise in Los Angeles?"},
    {"role": "function-call", "content": '{"name": "get_sunrise_sunset_times", "arguments": {"city": "Los Angeles"}}'},
    {"role": "function-response", "content": '{"times_list": ["6:00 AM", "6:00 PM"]}'},
]

It looks like:

<bos><start_of_turn>user
You are a helpful assistant with access to the following functions. Use them if required - {
    "name": "get_weather",
    "description": "A function that returns the weather in a given city.",
    "parameters": {
        "type": "object",
        "properties": {
            "city": {
                "type": "string",
                "description": "The city to get the weather for."
            }
        },
        "required": [
            "city"
        ]
    }
},
{
    "name": "get_sunrise_sunset_times",
    "description": "A function that returns the time of sunrise and sunset at the present moment, for a given city, in the form of a list: [sunrise_time, sunset_time].",
    "parameters": {
        "type": "object",
        "properties": {
            "city": {
                "type": "string",
                "description": "The city to get the sunrise and sunset times for."
            }
        },
        "required": [
            "city"
        ]
    }
}

Hi, can you tell me the time of sunrise in Los Angeles?<end_of_turn>
<start_of_turn>model
Function call: {"name": "get_sunrise_sunset_times", "arguments": {"city": "Los Angeles"}}<end_of_turn>
<start_of_turn>user
Function response: {"times_list": ["6:00 AM", "6:00 PM"]}<end_of_turn>

Chat history with a standard user-model template

[
    {"role": "system", "content": "You are a helpful assistant"},
    {"role": "user", "content": "Tell me about California"},
]

It looks like:

<bos><start_of_turn>user
You are a helpful assistant

Tell me about California<end_of_turn>

Model Description

This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.

Developed by: [More Information Needed]
Funded by [optional]: [More Information Needed]
Shared by [optional]: [More Information Needed]
Model type: [More Information Needed]
Language(s) (NLP): [More Information Needed]
License: [More Information Needed]
Finetuned from model [optional]: [More Information Needed]

Model Sources [optional]

Repository: [More Information Needed]
Paper [optional]: [More Information Needed]
Demo [optional]: [More Information Needed]

Uses

Direct Use

[More Information Needed]

Downstream Use [optional]

[More Information Needed]

Out-of-Scope Use

[More Information Needed]

Bias, Risks, and Limitations

[More Information Needed]

Recommendations

Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.

How to Get Started with the Model

Use the code below to get started with the model.

[More Information Needed]

Training Details

Training Data

[More Information Needed]

Training Procedure

Preprocessing [optional]

[More Information Needed]

Training Hyperparameters

Training regime: [More Information Needed]

Speeds, Sizes, Times [optional]

[More Information Needed]

Evaluation

Testing Data, Factors & Metrics

Testing Data

[More Information Needed]

Factors

[More Information Needed]

Metrics

[More Information Needed]

Results

[More Information Needed]

Summary

Model Examination [optional]

[More Information Needed]

Environmental Impact

Carbon emissions can be estimated using the Machine Learning Impact calculator presented in Lacoste et al. (2019).

Hardware Type: [More Information Needed]
Hours used: [More Information Needed]
Cloud Provider: [More Information Needed]
Compute Region: [More Information Needed]
Carbon Emitted: [More Information Needed]

Technical Specifications [optional]

Model Architecture and Objective

[More Information Needed]

Compute Infrastructure

[More Information Needed]

Hardware

[More Information Needed]

Software

[More Information Needed]

Citation [optional]

BibTeX:

[More Information Needed]

APA:

[More Information Needed]

Glossary [optional]

[More Information Needed]

More Information [optional]

[More Information Needed]

Model Card Authors [optional]

[More Information Needed]

Model Card Contact

[More Information Needed]

DiTy/gemma-2-9b-it-function-calling

Model card tree

Usage (HuggingFace Transformers)

Prepare your functions for Function Calling

Just use chat template

Usage via transformers pipeline

Prompt structure and expected content

Chat history with Function Calling

Chat history with a standard user-model template

Model Description

Model Sources [optional]

Uses

Direct Use

Downstream Use [optional]

Out-of-Scope Use

Bias, Risks, and Limitations

Recommendations

How to Get Started with the Model

Training Details

Training Data

Training Procedure

Preprocessing [optional]

Training Hyperparameters

Speeds, Sizes, Times [optional]

Evaluation

Testing Data, Factors & Metrics

Testing Data

Factors

Metrics

Results

Summary

Model Examination [optional]

Environmental Impact

Technical Specifications [optional]

Model Architecture and Objective

Compute Infrastructure

Hardware

Software

Citation [optional]

Glossary [optional]

More Information [optional]

Model Card Authors [optional]

Model Card Contact

Usage via transformers `pipeline`