Bram Vanroy PRO

Nice! In my experience preference tuning with the ultra feedback datasets does not really change benchmark scores (and sometimes even makes them worse) but it does seem to improve the real-world user experience when chatting with the model.

I'm also not sure if orpo only on UF is better than sft on UC + DPO on UF, especially if you're also trying to do language adaptation. That, or first continue pretraining the model and then doing orpo.

While the "rules" of OpenAI do get frustrating from time to time, I do not blame others who do not follow the same path as I do. If I am asked why my licenses are different from someone else's I will answer according to what I've written in the post above (the rules suck and our vague, I understand why people do what they do and I do what I do because of other reasons). But I definitely do not want to go around and point fingers pre-emptively in hopes that people just use my models. Our community for Dutch is already quite small so I rather just lift each other up and build on each others work through friendly "competition" than to compete in bad faith.

So I think that for my future models, I'll just make use of ultrachat+ultrafeedback, which should be cleared for training apache 2.0 models because they were created with Azure. This may negatively impact the model's performance (especially for code because it does not include the Stack Overflow set) but I hope the impact is limited.

What do you mean with compliance in this context? I'm not sure how I can market being non-commercial as a good thing 😅

Cool! Looking forward to what you'll build with this!

I think the correct place for this is to make a new issue on their issue tracker: https://github.com/ScandEval/ScandEval/issues

Understandable. I'm especially attracted to the broad vocabulary, which can be of use for language adaptation.

What kind of weird results? In terms of loss, or really qualitative output?

Indeed, there is not a lot of metadata. There's also a discrepancy between the no. scores/languages and the no. paragraphs in the text. I've notified the authors about that. CulturaX is an attractive dataset, too!

Really cool, crowd-sourcing like this can be very powerful!

Just joined and I have one question/recommendation: the name "prompt" is ambiguous. I got the following conversation:

Marv is a chatbot that reluctantly answers questions with sarcastic responses:

You: How many pounds are in a kilogram?
Marv: This again? There are 2.2 pounds in a kilogram. Please make a note of this.
You: What does HTML stand for?
Marv: Was Google too busy? Hypertext Markup Language. The T is for try to ask better questions in the future.
You: When did the first airplane fly?
Marv: On December 17, 1903, Wilbur and Orville Wright made the first flights. I wish they’d come and take me away.
You: What is the meaning of life?
Marv: I’m not sure. I’ll ask my friend Google.
You: Why is the sky blue?

Intuitively, when I am asked to "rate the prompt" I would expect to have to rate the user prompt that is used to trigger a response, so does that mean I should rate "How many pounds are in a kilogram?" Or do I have to rate all the responses of "Marv"? Or do I have to rate the whole conversation? The guidelines are also not very strictly clear to me because the example that I get is a conversation, so there is a lot potential to be rated (one user prompt, all user prompts, whole conversation, etc.)

Hope you see my confusion. To make sure that everyone is rating the same aspects, this could be clarified!

From my limited experience, looking at potential unwanted text properties of chosen v. rejected can be another one to investigate. I think the model may learn, for instance, that longer sequences are better and will therefore learn to generate longer sequences regardless of the quality of the content. You can catch this in the metrics, I believe, but only in the log probs (which would then also be very low for the chosen text, perhaps even lower than the rejected text). You'll likely not notice this in the rewards metrics.