maxwell andrews

Hi Victor, the playground is working, but appears to be only for LLM models. I need to compare embedding models. What I want to be able to do is select a task, for example clustering, provide my own dataset, and then find & run a bunch of models against the task automatically so I can compare the results. Ideally I could find the top 50 models, see how they perform, and then narrow it down to a few to study in more detail.

Here's the problem. This is the current task list on HF:

And then within task classification, for example, there are 68,000+ models:

So finding the right set of these to start with is a huge barrier to actually utilizing these models practically. The best we have is the mteb leaderboard which while it can give some idea of relative performance, may not match at all the specific task or domain one is trying to accomplish. Right now, one must go to the leaderboard, pick one you think might work, try it, then pick another one and try it, etc. It's a very manual process and you don't know if there is yet another that might be better.

can you try https://hf.co/playground and tell me if this helps for your use case?

I get a 403 error saying I need to be a "hugging face member." However I'm logged in an have an active billing method, so not sure what the issue is.

I want to create a pipeline signature and provide a few examples, then have the hub go and test a bunch of models against that scenario and pick the best ones for me. For example, let's say I want to build a username matching function. I should be able to provide some example positive and negative matches, expand those examples into a small eval dataset using one-click synthetic data expansion, and then have the hub go and try hundreds of models against that eval dataset and find the top performers in several parameter size classes.