Question about chat template and fine-tuning
In the chat template, we have the following at the end:
{{ '<|start_header_id|>assistant<|end_header_id|>
' }}
This produces an extra assistant sequence, even if I specify add_generation_prompt=False when using tokenizer.apply_chat_template.
Here is an example of applying chat template:
tokenizer.apply_chat_template(conversation=[{'content': 'Example text user', 'role': 'user'}, {'content': 'Example text assistant', 'role': 'assistant'}], add_generation_prompt=False, tokenize=False)
Result:
<|begin_of_text|><|start_header_id|>user<|end_header_id|>
Example text user<|eot_id|><|start_header_id|>assistant<|end_header_id|>
Example text assistant<|eot_id|><|start_header_id|>assistant<|end_header_id|>
Is the extra assistant headers intended?
when i try to use the trl to fine-tuning(sft) the Llama3.1-8B-instruct, after finish the checkpoint loading there is issue occupy："TypeError: Object of type BitsAndBytesConfig is not JSON serializable". Does anyone fix it?
In the chat template, we have the following at the end:
{{ '<|start_header_id|>assistant<|end_header_id|>' }}
This produces an extra assistant sequence, even if I specify add_generation_prompt=False when using tokenizer.apply_chat_template.
Here is an example of applying chat template:
tokenizer.apply_chat_template(conversation=[{'content': 'Example text user', 'role': 'user'}, {'content': 'Example text assistant', 'role': 'assistant'}], add_generation_prompt=False, tokenize=False)Result:
<|begin_of_text|><|start_header_id|>user<|end_header_id|> Example text user<|eot_id|><|start_header_id|>assistant<|end_header_id|> Example text assistant<|eot_id|><|start_header_id|>assistant<|end_header_id|>Is the extra assistant headers intended?
If you look in other models' chat templates in their tokenizer_config.json, there is "{% if add_generation_prompt %}{{ '<|start_header_id|>assistant<|end_header_id|>\n\n' }}{% endif %}", but here with 3.1 we don't have "{% if add_generation_prompt %}" part, so it's the same whether "add_generation_prompt" is True or False.
