Extra tokens while function calling
Hello! I've encountered some problems when using all Hammer-2.1 models via vLLM - when using vllm and passing tools argument, model no longer chat and just outputs \n[]\n with triple "`". When tools is not passed model behave just fine. Also, tools are called with same tokens, example of output:
ChatCompletionMessage(content="\n[{'name': 'notes', 'arguments': {'color': true, 'type': 'food', 'product': 'cat food'}}]\n", refusal=None, role='assistant', audio=None, function_call=None, tool_calls=[])
Using Hammer tools from GitHub doesn't solve the problem
Hi, when you pass tools to the Hammer-2.1 models using vLLM, the model switches to a tool invocation prompt mode. In this mode, the model is designed to either perform a tool action or verify if the input does not require a tool invocation. As a result, when it outputs \n[]\n, it indicates that the model has refused to perform a non-tool invocation action.
Currently, the parser in vLLM does not support the output format from Hammer models properly. We appreciate you bringing this issue, and we have made fixes to the Hammer client on GitHub. Please try it again and see if it resolves your problem.
Thanks a lot for your work! Updated Hammer client works perfectly now.