System prompt bleeds into the responses
I haven't done much tests on this model yet, but I am very grateful for another series of German language models. The ones I am currently aware of are the EM_German and Sauerkraut models. Since the immediate first response was in German, sounds to me like you are on a promising path :-)
The other day I took a closer look at the OpenHermes dataset (https://huggingface.co/datasets/teknium/OpenHermes-2.5), which has great performance for a tinetune, and 99.9% of its prompts are without a system prompt, apart from ~10 or so that contained textbook_generator. I don't know if you have done any experimentation in that regard, but if not, I hope it will be of help. Although, I also did some tests with another german model (https://huggingface.co/Weyaxi/SauerkrautLM-UNA-SOLAR-Instruct/discussions/6#65a0804b6e2b627610014119) and in my very rudimentary tests, having a german system prompt helped the model to respond in german in its first response, but I did not test enough. My findings are not enough to conclude anything apart from one thing: Either having a particular system prompt or not having a system prompt is important, so choose with care.