Expert Support case study: Bolstering a RAG app with LLM-as-a-Judge
โข
24
Aymeric Roucher
m-ric
AI & ML interests
MLE at Hugging Face ๐ค
LLMs, Agents, RAG, Multimodal.
Recent Activity
liked
a dataset
about 21 hours ago
mlabonne/orca-agentinstruct-1M-v1-cleaned
reacted to
cfahlgren1's
post
with โค๏ธ
about 21 hours ago
posted
an
update
about 23 hours ago
Articles
Organizations
m-ric's activity
reacted to
cfahlgren1's
post with โค๏ธ
about 21 hours ago
Post
2030
You can clean and format datasets entirely in the browser with a few lines of SQL.
In this post, I replicate the process @mlabonne used to clean the new microsoft/orca-agentinstruct-1M-v1 dataset.
The cleaning process consists of:
- Joining the separate splits together / add split column
- Converting string messages into list of structs
- Removing empty system prompts
https://huggingface.co/blog/cfahlgren1/the-beginners-guide-to-cleaning-a-dataset
Here's his new cleaned dataset: mlabonne/orca-agentinstruct-1M-v1-cleaned
In this post, I replicate the process @mlabonne used to clean the new microsoft/orca-agentinstruct-1M-v1 dataset.
The cleaning process consists of:
- Joining the separate splits together / add split column
- Converting string messages into list of structs
- Removing empty system prompts
https://huggingface.co/blog/cfahlgren1/the-beginners-guide-to-cleaning-a-dataset
Here's his new cleaned dataset: mlabonne/orca-agentinstruct-1M-v1-cleaned
- 1 reply
posted
an
update
about 23 hours ago
Post
363
๐ Meta teams use a fine-tuned Llama model to fix production issues in seconds
One of Meta's engineering teams shared how they use a fine-tuned small Llama (Llama-2-7B, so not even a very recent model) to identify the root cause of production issues with 42% accuracy.
๐ค 42%, is that not too low?
โก๏ธ Usually, whenever there's an issue in production, engineers dive into recent code changes to find the offending commit. At Meta's scale (thousands of daily changes), this is like finding a needle in a haystack.
๐ก So when the LLM-based suggestion is right, it cuts incident resolution time from hours to seconds!
How did they do it?
๐ Two-step approach:
โฃ Heuristics (code ownership, directory structure, runtime graphs) reduce thousands of potential changes to a manageable set
โฃ Fine-tuned Llama 2 7B ranks the most likely culprits
๐ Training pipeline:
โฃ Continued pre-training on Meta's internal docs and wikis
โฃ Supervised fine-tuning on past incident investigations
โฃ Training data mimicked real-world constraints (2-20 potential changes per incident)
๐ฎ Now future developments await:
โฃ Language models could handle more of the incident response workflow (runbooks, mitigation, post-mortems)
โฃ Improvements in model reasoning should boost accuracy further
Read it in full ๐ https://www.tryparity.com/blog/how-meta-uses-llms-to-improve-incident-response
One of Meta's engineering teams shared how they use a fine-tuned small Llama (Llama-2-7B, so not even a very recent model) to identify the root cause of production issues with 42% accuracy.
๐ค 42%, is that not too low?
โก๏ธ Usually, whenever there's an issue in production, engineers dive into recent code changes to find the offending commit. At Meta's scale (thousands of daily changes), this is like finding a needle in a haystack.
๐ก So when the LLM-based suggestion is right, it cuts incident resolution time from hours to seconds!
How did they do it?
๐ Two-step approach:
โฃ Heuristics (code ownership, directory structure, runtime graphs) reduce thousands of potential changes to a manageable set
โฃ Fine-tuned Llama 2 7B ranks the most likely culprits
๐ Training pipeline:
โฃ Continued pre-training on Meta's internal docs and wikis
โฃ Supervised fine-tuning on past incident investigations
โฃ Training data mimicked real-world constraints (2-20 potential changes per incident)
๐ฎ Now future developments await:
โฃ Language models could handle more of the incident response workflow (runbooks, mitigation, post-mortems)
โฃ Improvements in model reasoning should boost accuracy further
Read it in full ๐ https://www.tryparity.com/blog/how-meta-uses-llms-to-improve-incident-response
reacted to
reach-vb's
post with ๐ฅ
2 days ago
Post
3914
What a brilliant week for Open Source AI!
Qwen 2.5 Coder by Alibaba - 0.5B / 1.5B / 3B / 7B / 14B/ 32B (Base + Instruct) Code generation LLMs, with 32B tackling giants like Gemnini 1.5 Pro, Claude Sonnet
Qwen/qwen25-coder-66eaa22e6f99801bf65b0c2f
LLM2CLIP from Microsoft - Leverage LLMs to train ultra-powerful CLIP models! Boosts performance over the previous SOTA by ~17%
microsoft/llm2clip-672323a266173cfa40b32d4c
Athene v2 Chat & Agent by NexusFlow - SoTA general LLM fine-tuned from Qwen 2.5 72B excels at Chat + Function Calling/ JSON/ Agents
Nexusflow/athene-v2-6735b85e505981a794fb02cc
Orca Agent Instruct by Microsoft - 1 million instruct pairs covering text editing, creative writing, coding, reading comprehension, etc - permissively licensed
microsoft/orca-agentinstruct-1M-v1
Ultravox by FixieAI - 70B/ 8B model approaching GPT4o level, pick any LLM, train an adapter with Whisper as Audio Encoder
reach-vb/ultravox-audio-language-model-release-67373b602af0a52b2a88ae71
JanusFlow 1.3 by DeepSeek - Next iteration of their Unified MultiModal LLM Janus with RectifiedFlow
deepseek-ai/JanusFlow-1.3B
Common Corpus by Pleais - 2,003,039,184,047 multilingual, commercially permissive and high quality tokens!
PleIAs/common_corpus
I'm sure I missed a lot, can't wait for the next week!
Put down in comments what I missed! ๐ค
Qwen 2.5 Coder by Alibaba - 0.5B / 1.5B / 3B / 7B / 14B/ 32B (Base + Instruct) Code generation LLMs, with 32B tackling giants like Gemnini 1.5 Pro, Claude Sonnet
Qwen/qwen25-coder-66eaa22e6f99801bf65b0c2f
LLM2CLIP from Microsoft - Leverage LLMs to train ultra-powerful CLIP models! Boosts performance over the previous SOTA by ~17%
microsoft/llm2clip-672323a266173cfa40b32d4c
Athene v2 Chat & Agent by NexusFlow - SoTA general LLM fine-tuned from Qwen 2.5 72B excels at Chat + Function Calling/ JSON/ Agents
Nexusflow/athene-v2-6735b85e505981a794fb02cc
Orca Agent Instruct by Microsoft - 1 million instruct pairs covering text editing, creative writing, coding, reading comprehension, etc - permissively licensed
microsoft/orca-agentinstruct-1M-v1
Ultravox by FixieAI - 70B/ 8B model approaching GPT4o level, pick any LLM, train an adapter with Whisper as Audio Encoder
reach-vb/ultravox-audio-language-model-release-67373b602af0a52b2a88ae71
JanusFlow 1.3 by DeepSeek - Next iteration of their Unified MultiModal LLM Janus with RectifiedFlow
deepseek-ai/JanusFlow-1.3B
Common Corpus by Pleais - 2,003,039,184,047 multilingual, commercially permissive and high quality tokens!
PleIAs/common_corpus
I'm sure I missed a lot, can't wait for the next week!
Put down in comments what I missed! ๐ค
posted
an
update
2 days ago
Post
1272
Great feature alert: ๐ฌ๐ผ๐ ๐ฐ๐ฎ๐ป ๐ป๐ผ๐ ๐๐๐ฒ ๐ฎ๐ป๐ ๐ฆ๐ฝ๐ฎ๐ฐ๐ฒ ๐ฎ๐ ๐ฎ ๐๐ผ๐ผ๐น ๐ณ๐ผ๐ฟ ๐๐ผ๐๐ฟ ๐๐ฟ๐ฎ๐ป๐๐ณ๐ผ๐ฟ๐บ๐ฒ๐ฟ๐.๐ฎ๐ด๐ฒ๐ป๐! ๐ ๏ธ๐ฅ๐ฅ
This lets you take the coolest spaces, like FLUX.1-dev, and use them in agentic workflows with a few lines of code! ๐งโ๐ป
On the video below, I set up my fake vacation pictures where I'm awesome at surfing (I'm really not) ๐
Head to the doc to learn this magic ๐ https://huggingface.co/docs/transformers/main/en/agents_advanced#import-a-space-as-a-tool-
This lets you take the coolest spaces, like FLUX.1-dev, and use them in agentic workflows with a few lines of code! ๐งโ๐ป
On the video below, I set up my fake vacation pictures where I'm awesome at surfing (I'm really not) ๐
Head to the doc to learn this magic ๐ https://huggingface.co/docs/transformers/main/en/agents_advanced#import-a-space-as-a-tool-
posted
an
update
6 days ago
Post
360
๐ ๐ฒ๐๐ฎ ๐๐ฒ๐ฎ๐บ ๐ท๐๐๐ ๐ฑ๐ฟ๐ผ๐ฝ๐ฝ๐ฒ๐ฑ ๐๐ต๐ฒ ๐ณ๐ถ๐ฟ๐๐ ๐ช๐ฎ๐๐ฒ๐ฟ๐บ๐ฎ๐ฟ๐ธ๐ถ๐ป๐ด ๐บ๐ผ๐ฑ๐ฒ๐น ๐๐ต๐ฎ๐ ๐ป๐ผ๐ ๐ฒ๐ฑ๐ถ๐ ๐ฐ๐ฎ๐ป ๐ฏ๐ฟ๐ฒ๐ฎ๐ธ!๐ก๏ธ
๐ค Ever heard of watermarking? It's a technique that allows you to mark in an image its original source. It's our best shield against AI-generated deepfakes, or content stolen from artists! ๐จ
๐ญ Watermarking systems are actually a pair of models: a watermark embedder that applies the watermark on the image, and its corresponding decoder that should detect the original watermark.
โ But current methods were very limited: they can only apply and detect the watermark on your image as a whole. So, if you're an attacker it's easy to break: just crop it! add text on top! or whatever, really, anything would work to break the watermark.
A team of researchers at Meta was not happy with this. ๐ค
So to withstand real-world attacks, they decided to make a watermarking model that would also work on any sub-part of the image. It's a real paradigm shift: they consider watermarking not as an image classification task, but as an image segmentation task!
๐๏ธ ๐๐ฟ๐ฐ๐ต๐ถ๐๐ฒ๐ฐ๐๐๐ฟ๐ฒ
โธ The "Embedder" (a variational autoencoder + embedder, 1.1M parameters in total) encodes a n-bit message into a watermark signal that is added to the original image
โธ [Only during training] The "Augmenter" randomly distorts the image: masks parts, crops, resizes, compresses. It's basically torture at this point.
โธ The "Extractor" (a vision transformer, or ViT, with 96M parameters) then re-extracts the message from the distorted image, by predicting a (1+n) vector per pixel to predict the watermarked parts and decode corresponding messages.
The performance blows existing models out of the water, they even created new tasks (segmentation-related) just to grok them!
Gerat work @pierrefdz and @tomsander1998 !
Paper here ๐ Watermark Anything with Localized Messages (2411.07231)
๐ค Ever heard of watermarking? It's a technique that allows you to mark in an image its original source. It's our best shield against AI-generated deepfakes, or content stolen from artists! ๐จ
๐ญ Watermarking systems are actually a pair of models: a watermark embedder that applies the watermark on the image, and its corresponding decoder that should detect the original watermark.
โ But current methods were very limited: they can only apply and detect the watermark on your image as a whole. So, if you're an attacker it's easy to break: just crop it! add text on top! or whatever, really, anything would work to break the watermark.
A team of researchers at Meta was not happy with this. ๐ค
So to withstand real-world attacks, they decided to make a watermarking model that would also work on any sub-part of the image. It's a real paradigm shift: they consider watermarking not as an image classification task, but as an image segmentation task!
๐๏ธ ๐๐ฟ๐ฐ๐ต๐ถ๐๐ฒ๐ฐ๐๐๐ฟ๐ฒ
โธ The "Embedder" (a variational autoencoder + embedder, 1.1M parameters in total) encodes a n-bit message into a watermark signal that is added to the original image
โธ [Only during training] The "Augmenter" randomly distorts the image: masks parts, crops, resizes, compresses. It's basically torture at this point.
โธ The "Extractor" (a vision transformer, or ViT, with 96M parameters) then re-extracts the message from the distorted image, by predicting a (1+n) vector per pixel to predict the watermarked parts and decode corresponding messages.
The performance blows existing models out of the water, they even created new tasks (segmentation-related) just to grok them!
Gerat work @pierrefdz and @tomsander1998 !
Paper here ๐ Watermark Anything with Localized Messages (2411.07231)
reacted to
maxiw's
post with โค๏ธ๐๐ฅ
7 days ago
Post
4482
I was curious to see what people post here on HF so I created a dataset with all HF Posts:
maxiw/hf-posts
Some interesting stats:
Top 5 Authors by Total Impressions:
-----------------------------------
@merve : 171,783 impressions (68 posts)
@fdaudens : 135,253 impressions (81 posts)
@singhsidhukuldeep : 122,591 impressions (81 posts)
@akhaliq : 119,526 impressions (78 posts)
@MonsterMMORPG : 112,500 impressions (45 posts)
Top 5 Users by Number of Reactions Given:
----------------------------------------
@osanseviero : 1278 reactions
@clem : 910 reactions
@John6666 : 899 reactions
@victor : 674 reactions
@samusenps : 655 reactions
Top 5 Most Used Reactions:
-------------------------
โค๏ธ: 7048 times
๐ฅ: 5921 times
๐: 4856 times
๐: 2549 times
๐ค: 2065 times
Some interesting stats:
Top 5 Authors by Total Impressions:
-----------------------------------
@merve : 171,783 impressions (68 posts)
@fdaudens : 135,253 impressions (81 posts)
@singhsidhukuldeep : 122,591 impressions (81 posts)
@akhaliq : 119,526 impressions (78 posts)
@MonsterMMORPG : 112,500 impressions (45 posts)
Top 5 Users by Number of Reactions Given:
----------------------------------------
@osanseviero : 1278 reactions
@clem : 910 reactions
@John6666 : 899 reactions
@victor : 674 reactions
@samusenps : 655 reactions
Top 5 Most Used Reactions:
-------------------------
โค๏ธ: 7048 times
๐ฅ: 5921 times
๐: 4856 times
๐: 2549 times
๐ค: 2065 times
- 9 replies
posted
an
update
7 days ago
Post
3624
๐ง๐ต๐ฒ ๐ป๐ฒ๐
๐ ๐ฏ๐ถ๐ด ๐๐ผ๐ฐ๐ถ๐ฎ๐น ๐ป๐ฒ๐๐๐ผ๐ฟ๐ธ ๐ถ๐ ๐ป๐ผ๐ ๐ฆ, ๐ถ๐'๐ ๐๐๐ฏ ๐ฃ๐ผ๐๐๐! [INSERT STONKS MEME WITH LASER EYES]
See below: I got 105k impressions since regularly posting Hub Posts, coming close to my 275k on Twitter!
โ๏ธ Computed with the great dataset maxiw/hf-posts
โ๏ธ Thanks to Qwen2.5-Coder-32B for showing me how to access dict attributes in a SQL request!
cc @merve who's far in front of me
See below: I got 105k impressions since regularly posting Hub Posts, coming close to my 275k on Twitter!
โ๏ธ Computed with the great dataset maxiw/hf-posts
โ๏ธ Thanks to Qwen2.5-Coder-32B for showing me how to access dict attributes in a SQL request!
cc @merve who's far in front of me
-
- 9 replies
posted
an
update
9 days ago
Post
2336
A non-Instruct LLM assistant is mostly useless. ๐ง
Since it's mostly a model trained to complete text, when you ask it a question like "What to do during a stopover in Paris?", it can just go on and on adding more details to your question instead of answering, which would be valid to complete text from its training corpus, but not to answer questions.
โก๏ธ So the post-training stage includes an important Instruction tuning step where you teach your model how to be useful : answer questions, be concise, be polite... RLHF is a well known technique for this.
For people interested to understand how this step works, the folks at Adaptive ML have made a great guide!
Read it here ๐ https://www.adaptive-ml.com/post/from-zero-to-ppo
Since it's mostly a model trained to complete text, when you ask it a question like "What to do during a stopover in Paris?", it can just go on and on adding more details to your question instead of answering, which would be valid to complete text from its training corpus, but not to answer questions.
โก๏ธ So the post-training stage includes an important Instruction tuning step where you teach your model how to be useful : answer questions, be concise, be polite... RLHF is a well known technique for this.
For people interested to understand how this step works, the folks at Adaptive ML have made a great guide!
Read it here ๐ https://www.adaptive-ml.com/post/from-zero-to-ppo
posted
an
update
10 days ago
Post
3144
๐ค๐๐ฒ๐ป๐ฎ.๐ฑ-๐๐ผ๐ฑ๐ฒ๐ฟ-๐ฏ๐ฎ๐: ๐ป๐ฒ๐ ๐ฏ๐ฒ๐๐-๐ถ๐ป-๐ฐ๐น๐ฎ๐๐ ๐ผ๐ฝ๐ฒ๐ป ๐ฐ๐ผ๐ฑ๐ถ๐ป๐ด ๐บ๐ผ๐ฑ๐ฒ๐น, ๐ฏ๐ฒ๐ฎ๐๐ ๐๐ฃ๐ง-๐ฐ๐ผ ๐ผ๐ป ๐บ๐ผ๐๐ ๐ฐ๐ผ๐ฑ๐ถ๐ป๐ด ๐ฏ๐ฒ๐ป๐ฐ๐ต๐บ๐ฎ๐ฟ๐ธ๐!๐ฅ
๐ช It's the first time Open-Source coding model of this size class that clearly matches GPT-4o's coding capabilities!
โจ Completes the previous two Qwen 2.5 Coder release with 4 new size: 0.5B, 3B, 14B, 32B
๐ Support long context up to 128K (for the 14B and 32B models)
โ Drop-in replacement to GPT-4o as a coding assistant on Cursor or for Artifacts!
๐ค Models available right now on the Hub, under Apache 2.0 license!
They have setup a crazy Artifacts demo, you should go have a look!
๐ Qwen/Qwen2.5-Coder-Artifacts
๐ช It's the first time Open-Source coding model of this size class that clearly matches GPT-4o's coding capabilities!
โจ Completes the previous two Qwen 2.5 Coder release with 4 new size: 0.5B, 3B, 14B, 32B
๐ Support long context up to 128K (for the 14B and 32B models)
โ Drop-in replacement to GPT-4o as a coding assistant on Cursor or for Artifacts!
๐ค Models available right now on the Hub, under Apache 2.0 license!
They have setup a crazy Artifacts demo, you should go have a look!
๐ Qwen/Qwen2.5-Coder-Artifacts
posted
an
update
10 days ago
Post
772
๐๐ฟ๐ฒ ๐๐ฐ๐ฎ๐น๐ถ๐ป๐ด ๐น๐ฎ๐๐ ๐ผ๐๐ฒ๐ฟ? ๐ ๐ฟ๐ฒ๐ฝ๐ผ๐ฟ๐ ๐ณ๐ฟ๐ผ๐บ ๐๐ต๐ฒ ๐๐ป๐ณ๐ผ๐ฟ๐บ๐ฎ๐๐ถ๐ผ๐ป ๐ฎ๐ป๐ป๐ผ๐๐ป๐ฐ๐ฒ๐ฑ ๐๐ต๐ฎ๐ ๐ข๐ฝ๐ฒ๐ป๐๐ ๐ถ๐ ๐๐ฒ๐ฒ๐ถ๐ป๐ด ๐ฑ๐ถ๐บ๐ถ๐ป๐ถ๐๐ต๐ถ๐ป๐ด ๐ฟ๐ฒ๐๐๐ฟ๐ป๐ ๐ณ๐ฟ๐ผ๐บ ๐๐ฐ๐ฎ๐น๐ถ๐ป๐ด ๐๐ฝ ๐๐ต๐ฒ ๐ป๐ฒ๐
๐ ๐๐ฃ๐ง ๐บ๐ผ๐ฑ๐ฒ๐น๐.
๐ What are scaling laws? These are empiric laws that say "Every time you increase compute spent in training 10-fold, your LLM's performance will go up by a predictable tick". Of course, they apply only if you train your model with the right methods.
The image below illustrates it: they're from a paper by Google, "Scaling Autoregressive Models for Content-Rich Text-to-Image Generation", and they show how quality and instruction following of models improve when you scale the model up (which is equivalent to scaling up the compute spent in training).
โก๏ธ These scaling laws have immense impact: they triggered the largest gold rush ever, with companies pouring billions into scaling up theiur training. Microsoft and OpenAI spent 100B into their "Startgate" mega training cluster, due to start running in 2028.
๐ค So, what about these reports of scaling laws slowing down?
If they are true, they would mean a gigantic paradigm shift, as the hundreds of billions poured by AI companies into scaling could be a dead-end. โ๏ธ
But I doubt it: until the most recent publications, scaling laws showed no signs of weakness, and the researchers at the higher end of the scale-up seems to imply the scaling up continues.
Wait and see!
๐ What are scaling laws? These are empiric laws that say "Every time you increase compute spent in training 10-fold, your LLM's performance will go up by a predictable tick". Of course, they apply only if you train your model with the right methods.
The image below illustrates it: they're from a paper by Google, "Scaling Autoregressive Models for Content-Rich Text-to-Image Generation", and they show how quality and instruction following of models improve when you scale the model up (which is equivalent to scaling up the compute spent in training).
โก๏ธ These scaling laws have immense impact: they triggered the largest gold rush ever, with companies pouring billions into scaling up theiur training. Microsoft and OpenAI spent 100B into their "Startgate" mega training cluster, due to start running in 2028.
๐ค So, what about these reports of scaling laws slowing down?
If they are true, they would mean a gigantic paradigm shift, as the hundreds of billions poured by AI companies into scaling could be a dead-end. โ๏ธ
But I doubt it: until the most recent publications, scaling laws showed no signs of weakness, and the researchers at the higher end of the scale-up seems to imply the scaling up continues.
Wait and see!
- 1 reply
posted
an
update
13 days ago
Post
1611
๐๐ป๐ฑ๐ฟ๐ผ๐ถ๐ฑ๐๐ฎ๐ฏ: ๐๐ถ๐ฟ๐๐ ๐ฒ๐๐ฒ๐ฟ ๐๐๐๐๐ฒ๐บ๐ฎ๐๐ถ๐ฐ ๐ฏ๐ฒ๐ป๐ฐ๐ต๐บ๐ฎ๐ฟ๐ธ ๐ณ๐ผ๐ฟ ๐๐ป๐ฑ๐ฟ๐ผ๐ถ๐ฑ ๐บ๐ผ๐ฏ๐ถ๐น๐ฒ ๐ฎ๐ด๐ฒ๐ป๐๐ ๐๐ต๐ผ๐๐ ๐๐ต๐ฎ๐ ๐๐บ๐ฎ๐น๐น, ๐ณ๐ถ๐ป๐ฒ-๐๐๐ป๐ฒ๐ฑ ๐ผ๐ฝ๐ฒ๐ป ๐บ๐ผ๐ฑ๐ฒ๐น๐ ๐ฐ๐ฎ๐ป ๐ฝ๐ผ๐๐ฒ๐ฟ ๐ฎ ๐๐๐ฅ๐ฉ๐๐ฆ ๐๐๐๐๐ฒ๐บ ๐ผ๐ป ๐๐ผ๐๐ฟ ๐๐บ๐ฎ๐ฟ๐๐ฝ๐ต๐ผ๐ป๐ฒ ๐ฑ๐ฅ
A team from Tsinghua University just released AndroidLab, the first systematic framework to evaluate and train Android mobile agents that works with both text-only and multimodal models.
They show that fine-tuning small open-source models can significantly boost performance, matching that of much bigger closed models like GPT-4o.
The team built:
๐ย A reproducible benchmark with 138 tasks across 9 apps to evaluate mobile agents systematically
๐๐ฑย A framework supporting both text-only (via XML) and visual (via marked screenshots) interfaces
โ ย An instruction dataset of 10.5k operation traces for training mobile agents
Key insights:
- ๐ Fine-tuning improves performance BY A LOT: Open-source model Llama-3.1-8B improves from 2% to 24% success rate after training, nearly reaching GPT-4o performance although itโs much smaller
- โ๏ธ Text-only agents match multimodal ones: XML-based agents achieve similar performance to screenshot-based multimodal agents.
Read their paper here ๐ AndroidLab: Training and Systematic Benchmarking of Android Autonomous Agents (2410.24024)
A team from Tsinghua University just released AndroidLab, the first systematic framework to evaluate and train Android mobile agents that works with both text-only and multimodal models.
They show that fine-tuning small open-source models can significantly boost performance, matching that of much bigger closed models like GPT-4o.
The team built:
๐ย A reproducible benchmark with 138 tasks across 9 apps to evaluate mobile agents systematically
๐๐ฑย A framework supporting both text-only (via XML) and visual (via marked screenshots) interfaces
โ ย An instruction dataset of 10.5k operation traces for training mobile agents
Key insights:
- ๐ Fine-tuning improves performance BY A LOT: Open-source model Llama-3.1-8B improves from 2% to 24% success rate after training, nearly reaching GPT-4o performance although itโs much smaller
- โ๏ธ Text-only agents match multimodal ones: XML-based agents achieve similar performance to screenshot-based multimodal agents.
Read their paper here ๐ AndroidLab: Training and Systematic Benchmarking of Android Autonomous Agents (2410.24024)
posted
an
update
16 days ago
Post
2483
๐๐๐ป๐๐๐ฎ๐ป-๐๐ฎ๐ฟ๐ด๐ฒ ๐ท๐๐๐ ๐ฟ๐ฒ๐น๐ฒ๐ฎ๐๐ฒ๐ฑ ๐ฏ๐ ๐ง๐ฒ๐ป๐ฐ๐ฒ๐ป๐: ๐๐ฎ๐ฟ๐ด๐ฒ๐๐ ๐ฒ๐๐ฒ๐ฟ ๐ผ๐ฝ๐ฒ๐ป ๐ ๐ผ๐ ๐๐๐ , ๐ผ๐ป๐น๐ ๐ฑ๐ฎ๐ ๐ฎ๐ฐ๐๐ถ๐๐ฒ ๐ฝ๐ฎ๐ฟ๐ฎ๐บ๐ฒ๐๐ฒ๐ฟ๐ ๐ฏ๐๐ ๐ฏ๐ฒ๐ฎ๐๐ ๐๐๐ฎ๐ ๐ ๐ฏ.๐ญ-๐ฐ๐ฌ๐ฑ๐ ๐ผ๐ป ๐บ๐ผ๐๐ ๐ฎ๐ฐ๐ฎ๐ฑ๐ฒ๐บ๐ถ๐ฐ ๐ฏ๐ฒ๐ป๐ฐ๐ต๐บ๐ฎ๐ฟ๐ธ๐ ๐
โก Mixture of Experts (MoE) architecture: 389 B parameters in total, but only 52B are activated for any input
๐งช Trained on 7T tokens, including 1.5T tokens of synthetic data
๐๏ธ Architecture : Novel "recycle routing" prevents token dropping when experts are overrloaded
๐ Great benchmark results: Surpasses Llama-3-405B-Instruct in most benchmarks although it has 8x fewer active parameters
โฃ Impressive perf on MATH: 77.4
๐ย Large context length: up to 256K tokens
๐ License:
โฃ Commercial use allowed, except if your products have >100M monthly active users
โฃ No access in the EU
๐คย Model weights available on HF!
Read the full paper here ๐ย Hunyuan-Large: An Open-Source MoE Model with 52 Billion Activated Parameters by Tencent (2411.02265)
โก Mixture of Experts (MoE) architecture: 389 B parameters in total, but only 52B are activated for any input
๐งช Trained on 7T tokens, including 1.5T tokens of synthetic data
๐๏ธ Architecture : Novel "recycle routing" prevents token dropping when experts are overrloaded
๐ Great benchmark results: Surpasses Llama-3-405B-Instruct in most benchmarks although it has 8x fewer active parameters
โฃ Impressive perf on MATH: 77.4
๐ย Large context length: up to 256K tokens
๐ License:
โฃ Commercial use allowed, except if your products have >100M monthly active users
โฃ No access in the EU
๐คย Model weights available on HF!
Read the full paper here ๐ย Hunyuan-Large: An Open-Source MoE Model with 52 Billion Activated Parameters by Tencent (2411.02265)
posted
an
update
19 days ago
Post
1539
๐ง ย ๐๐๐๐๐ฅ: ๐ณ๐ถ๐ฟ๐๐ ๐บ๐๐น๐๐ถ๐บ๐ผ๐ฑ๐ฎ๐น ๐ฏ๐ฒ๐ป๐ฐ๐ต๐บ๐ฎ๐ฟ๐ธ ๐๐ผ ๐บ๐ฎ๐ธ๐ฒ ๐บ๐ผ๐ฑ๐ฒ๐น๐ ๐ณ๐ผ๐ฟ๐ด๐ฒ๐ ๐๐ต๐ฎ๐ ๐๐ฒ ๐๐ฎ๐ป๐ ๐๐ต๐ฒ๐บ ๐๐ผ ๐ณ๐ผ๐ฟ๐ด๐ฒ๐
With privacy concerns rising, we sometimes need our models to "forget" specific information - like a person's data - while keeping everything else intact. Researchers just released CLEAR, the first benchmark to test how well this works with both text and images.
โย Bad news: Current methods either fail to truly forget or end up forgetting way too much. It's like trying to remove a single ingredient from a baked cake!
โจย But there's hope: Adding simple mathematical constraints (L1 regularization) during the forgetting process significantly improves results.
๐ฏย Key insights:
โ ย The benchmark tests forgetting on 200 fictional personas
โฃ 3,770 visual Q&A pairs
โฃ 4,000 textual Q&A pairs
โฃ Additional real-world tests
๐ย Most current forgetting methods don't work well with both text and images
โฃ They either remember what they should forget
โฃ Or they forget too much unrelated information
โจย Simple mathematical constraints work surprisingly well
โฃ L1 regularization prevents excessive forgetting
โฃ Works especially well with the LLMU method
๐ย Read the full paper here: CLEAR: Character Unlearning in Textual and Visual Modalities (2410.18057)
With privacy concerns rising, we sometimes need our models to "forget" specific information - like a person's data - while keeping everything else intact. Researchers just released CLEAR, the first benchmark to test how well this works with both text and images.
โย Bad news: Current methods either fail to truly forget or end up forgetting way too much. It's like trying to remove a single ingredient from a baked cake!
โจย But there's hope: Adding simple mathematical constraints (L1 regularization) during the forgetting process significantly improves results.
๐ฏย Key insights:
โ ย The benchmark tests forgetting on 200 fictional personas
โฃ 3,770 visual Q&A pairs
โฃ 4,000 textual Q&A pairs
โฃ Additional real-world tests
๐ย Most current forgetting methods don't work well with both text and images
โฃ They either remember what they should forget
โฃ Or they forget too much unrelated information
โจย Simple mathematical constraints work surprisingly well
โฃ L1 regularization prevents excessive forgetting
โฃ Works especially well with the LLMU method
๐ย Read the full paper here: CLEAR: Character Unlearning in Textual and Visual Modalities (2410.18057)
posted
an
update
20 days ago
Post
2334
> Oasis: First Real-Time Video Game Without a Game Engine! ๐ฎ
DecartAI & Etched just released Oasis - a fully AI-generated video game running at 20 FPS (frames per second). The model takes keyboard inputs and generates everything - physics, rules, graphics - on the fly, without any game engine.
โก๏ธ What makes this special? Current text-to-video models (Mochi-1, Sora, Kling) generate about 1 frame every 10-20 seconds (that's the kind of device I had to play LoL back in the day, thus my low rankings). Oasis is 200 times faster, making it the first playable AI-generated game.
โ๏ธ Under the hood, it uses a vision transformer to encode space and a diffusion model to generate frames. The secret sauce is "dynamic noising" - a technique that keeps the video stable between frames.
Key insights:
โก๏ธ Generates 20 FPS, vs 0.2 FPS for other DIT-based video models
โฃ The specialized hardware Sohu developed by Etched allows to handle 10x more player than H100
๐ฎ Features real game mechanics
โฃ Movement, jumping, item management
โฃ Physics and lighting
โฃ Procedurally generated worlds
โ ๏ธ Current limitations
โฃ Blurry graphics at a distance
โฃ Objects sometimes change appearance
โฃ Memory issues in long sessions
Try it yourself, the playable demo is impressive! ๐ https://oasis.decart.ai/welcome
Code ๐ https://github.com/etched-ai/open-oasis
Read it in full ๐ https://oasis-model.github.io/
DecartAI & Etched just released Oasis - a fully AI-generated video game running at 20 FPS (frames per second). The model takes keyboard inputs and generates everything - physics, rules, graphics - on the fly, without any game engine.
โก๏ธ What makes this special? Current text-to-video models (Mochi-1, Sora, Kling) generate about 1 frame every 10-20 seconds (that's the kind of device I had to play LoL back in the day, thus my low rankings). Oasis is 200 times faster, making it the first playable AI-generated game.
โ๏ธ Under the hood, it uses a vision transformer to encode space and a diffusion model to generate frames. The secret sauce is "dynamic noising" - a technique that keeps the video stable between frames.
Key insights:
โก๏ธ Generates 20 FPS, vs 0.2 FPS for other DIT-based video models
โฃ The specialized hardware Sohu developed by Etched allows to handle 10x more player than H100
๐ฎ Features real game mechanics
โฃ Movement, jumping, item management
โฃ Physics and lighting
โฃ Procedurally generated worlds
โ ๏ธ Current limitations
โฃ Blurry graphics at a distance
โฃ Objects sometimes change appearance
โฃ Memory issues in long sessions
Try it yourself, the playable demo is impressive! ๐ https://oasis.decart.ai/welcome
Code ๐ https://github.com/etched-ai/open-oasis
Read it in full ๐ https://oasis-model.github.io/
posted
an
update
24 days ago
Post
1821
I'm very proud to have supported
@CGIAR
and
@Digigreen
in making http://Farmer.chat, an app that supports 20k smallholder farmers on a daily basis ๐พ
There are ~500 million smallholder farmers globally, playing a critical role in global food security. Having access to accurate information is essential for them.
๐ฌ An โagricultural extension serviceโ offers technical advice on agriculture, and also supplies farmers with the necessary inputs and services to support their agricultural production.
But agriculture extension agents are not in large enough numbers to cope with all the requests, especially in countries like Kenya, India, Ethiopia, and Nigeria.
๐ So the team set out to build an app called http://Farmer.Chat, to provide an agricultural extension service, by building on the immense knowledge accumulated by CGIAR.
โจ The app is technically impressive: behind the Whatsapp-type UX, an agent interprets the user's intent, and identifies which tool to call to best answer their request: weather API, RAG on a CGIAR-provided knowledge base, market data, etc. The RAG on the knowledge base is in itself a work of art.
๐ฏ A key part of building such a complex system is to be able to evaluate it properly. During our bi-weekly sessions with the team, I could support them in implementing the method called "LLM-as-a-judge" to tackle this problem.
It worked really well : thanks to the amazing work of the team, the app now successfully answered over 300 thousand requests, in 6 different languages, and it keeps growing!
โก๏ธ @Vinsingh , @rajgreen and I just wrote a blog post to describe how the app works, especially the LLM-as-a-judge system!
Read it here ๐ https://huggingface.co/blog/digital-green-llm-judge
There are ~500 million smallholder farmers globally, playing a critical role in global food security. Having access to accurate information is essential for them.
๐ฌ An โagricultural extension serviceโ offers technical advice on agriculture, and also supplies farmers with the necessary inputs and services to support their agricultural production.
But agriculture extension agents are not in large enough numbers to cope with all the requests, especially in countries like Kenya, India, Ethiopia, and Nigeria.
๐ So the team set out to build an app called http://Farmer.Chat, to provide an agricultural extension service, by building on the immense knowledge accumulated by CGIAR.
โจ The app is technically impressive: behind the Whatsapp-type UX, an agent interprets the user's intent, and identifies which tool to call to best answer their request: weather API, RAG on a CGIAR-provided knowledge base, market data, etc. The RAG on the knowledge base is in itself a work of art.
๐ฏ A key part of building such a complex system is to be able to evaluate it properly. During our bi-weekly sessions with the team, I could support them in implementing the method called "LLM-as-a-judge" to tackle this problem.
It worked really well : thanks to the amazing work of the team, the app now successfully answered over 300 thousand requests, in 6 different languages, and it keeps growing!
โก๏ธ @Vinsingh , @rajgreen and I just wrote a blog post to describe how the app works, especially the LLM-as-a-judge system!
Read it here ๐ https://huggingface.co/blog/digital-green-llm-judge
posted
an
update
28 days ago
Post
1936
๐๐ Cohere releases Aya 8B & 32B: SOTA multilingual models for 23 languages !
How did they manage to beat top contenders while also adding 23 languages?
๐ ๐ง๐ฟ๐ฎ๐ถ๐ป ๐ผ๐ป ๐๐๐ป๐๐ต๐ฒ๐๐ถ๐ฐ ๐ฑ๐ฎ๐๐ฎ:
โข Synthetic data has been said to cause model-collapse after too much training
โข Cohere has introduced "data arbitrage" to prevent this by strategically sampling from a pool of several teacher models instead of one single teacher
โข First train a model pool for each different groups of languages, and employ an internal Reward Model named "Arbiter" to evaluate and select the optimal generation. Then only the best generation is kept as the final completion for each prompt
โก๏ธ This process is particularly effective for multilingual setting, where no single teacher model performs in all languages : here "Multilingual Arbitrage" singlehandedly improves win rates of the 8B model vs Gemma-2-9B by 10 points!
๐งฉ ๐จ๐๐ฒ ๐บ๐ผ๐ฑ๐ฒ๐น ๐บ๐ฒ๐ฟ๐ด๐ถ๐ป๐ด: Rather than struggling to find the right mix of data in training a single model for multilingual use, just train language specific models then merge them!
โข Maximize diversity between merged checkpoints by training each on different language families.
โข Experimented fancy techniques (SLERP, TIES, DARE-TIES) but found out weighted averaging to be the most consistent!
โก๏ธ Merging had 3x more gains at high 35B scale vs the 8B scale - consistent with literature findings that merging is more effective at scale
โก๏ธ ๐๐ฟ๐ฒ๐ฎ๐ ๐ฝ๐ฒ๐ฟ๐ณ๐ผ๐ฟ๐บ๐ฎ๐ป๐ฐ๐ฒ: Automatic evaluations on Arena-Hard-Auto dataset:
โก๏ธ Aya Expanse 8B beats models from its weight class such as Gemma 2 9B, Llama 3.1 8B, and the recent Ministral 8B, with win rates ranging from 60.4% to 70.6%
โก๏ธ Aya Expanse 32B outperforms Gemma 2 27B, Mistral 8x22B, and Llama 3.1 70B (2x its size)
โข โ ๏ธ But this performance eval comes from only one benchmark! Let's wait for Open LLM leaderboard evals;
๐ CC by NC license
Blog post here: https://huggingface.co/blog/aya-expanse
How did they manage to beat top contenders while also adding 23 languages?
๐ ๐ง๐ฟ๐ฎ๐ถ๐ป ๐ผ๐ป ๐๐๐ป๐๐ต๐ฒ๐๐ถ๐ฐ ๐ฑ๐ฎ๐๐ฎ:
โข Synthetic data has been said to cause model-collapse after too much training
โข Cohere has introduced "data arbitrage" to prevent this by strategically sampling from a pool of several teacher models instead of one single teacher
โข First train a model pool for each different groups of languages, and employ an internal Reward Model named "Arbiter" to evaluate and select the optimal generation. Then only the best generation is kept as the final completion for each prompt
โก๏ธ This process is particularly effective for multilingual setting, where no single teacher model performs in all languages : here "Multilingual Arbitrage" singlehandedly improves win rates of the 8B model vs Gemma-2-9B by 10 points!
๐งฉ ๐จ๐๐ฒ ๐บ๐ผ๐ฑ๐ฒ๐น ๐บ๐ฒ๐ฟ๐ด๐ถ๐ป๐ด: Rather than struggling to find the right mix of data in training a single model for multilingual use, just train language specific models then merge them!
โข Maximize diversity between merged checkpoints by training each on different language families.
โข Experimented fancy techniques (SLERP, TIES, DARE-TIES) but found out weighted averaging to be the most consistent!
โก๏ธ Merging had 3x more gains at high 35B scale vs the 8B scale - consistent with literature findings that merging is more effective at scale
โก๏ธ ๐๐ฟ๐ฒ๐ฎ๐ ๐ฝ๐ฒ๐ฟ๐ณ๐ผ๐ฟ๐บ๐ฎ๐ป๐ฐ๐ฒ: Automatic evaluations on Arena-Hard-Auto dataset:
โก๏ธ Aya Expanse 8B beats models from its weight class such as Gemma 2 9B, Llama 3.1 8B, and the recent Ministral 8B, with win rates ranging from 60.4% to 70.6%
โก๏ธ Aya Expanse 32B outperforms Gemma 2 27B, Mistral 8x22B, and Llama 3.1 70B (2x its size)
โข โ ๏ธ But this performance eval comes from only one benchmark! Let's wait for Open LLM leaderboard evals;
๐ CC by NC license
Blog post here: https://huggingface.co/blog/aya-expanse
posted
an
update
about 1 month ago
Post
858
๐๐ผ๐ ๐๐ผ ๐ฟ๐ฒ-๐ฟ๐ฎ๐ป๐ธ ๐๐ผ๐๐ฟ ๐๐ป๐ถ๐ฝ๐ฝ๐ฒ๐๐ ๐ถ๐ป ๐ฅ๐๐ โ ColBERT, Rerankers, Cross-Encoders
Letโs say youโre doing RAG, and in an effort to improve performance, you try to rerank a few possible source snippets by their relevancy to a query.
How can you score similarity between your query and any source document? ๐ค ๐ โ๏ธ ๐
๐ญ. ๐๐๐๐ ๐๐๐ฒ ๐ฒ๐บ๐ฏ๐ฒ๐ฑ๐ฑ๐ถ๐ป๐ด๐ : ๐ก๐ผ-๐ถ๐ป๐๐ฒ๐ฟ๐ฎ๐ฐ๐๐ถ๐ผ๐ป ๐๏ธ
This means that you encode each token from both the query and the doc as separate vectors, then average the tokens of each separately to get in total 2 vectors, then you compute similarity via cosine or something.
โก๏ธ Notable examples: Check the top of the MTEB leaderboard!
๐ฎ. ๐๐ฎ๐๐ฒ-๐ถ๐ป๐๐ฒ๐ฟ๐ฎ๐ฐ๐๐ถ๐ผ๐ป: ๐๐ต๐ถ๐ ๐ถ๐ ๐๐ผ๐น๐๐๐ฅ๐ง ๐
These encode each token from both query and doc as separate vectors as before, but compare all together without previously averaging them and losing information.
This is more accurate than no-interaction but also slower because you have to compare n*m vectors instead of 2. At least you can store documents in memory. And ColBERT has some optimisations like pooling to be faster.
โก๏ธ Notable examples: ColBERTv2, mxbai-colbert-large-v1, jina-colbert-v2
๐ฏ. ๐๐ฎ๐ฟ๐น๐ ๐ถ๐ป๐๐ฒ๐ฟ๐ฎ๐ฐ๐๐ถ๐ผ๐ป: ๐๐ฟ๐ผ๐๐-๐ฒ๐ป๐ฐ๐ผ๐ฑ๐ฒ๐ฟ๐ ๐๏ธ
Basically you run the concatenated query + document in a model to get a final score.
This is the most accurate, but also the slowest since it gets really long when you have many docs to rerank! And you cannot pre-store embeddings.
โก๏ธ Notable examples: MixedBread or Jina AI rerankers!
๐ So what you choose is a trade-off between speed and accuracy: I think ColBERT is often a really good choice!
Based on this great post by Jina AI ๐ https://jina.ai/news/what-is-colbert-and-late-interaction-and-why-they-matter
Letโs say youโre doing RAG, and in an effort to improve performance, you try to rerank a few possible source snippets by their relevancy to a query.
How can you score similarity between your query and any source document? ๐ค ๐ โ๏ธ ๐
๐ญ. ๐๐๐๐ ๐๐๐ฒ ๐ฒ๐บ๐ฏ๐ฒ๐ฑ๐ฑ๐ถ๐ป๐ด๐ : ๐ก๐ผ-๐ถ๐ป๐๐ฒ๐ฟ๐ฎ๐ฐ๐๐ถ๐ผ๐ป ๐๏ธ
This means that you encode each token from both the query and the doc as separate vectors, then average the tokens of each separately to get in total 2 vectors, then you compute similarity via cosine or something.
โก๏ธ Notable examples: Check the top of the MTEB leaderboard!
๐ฎ. ๐๐ฎ๐๐ฒ-๐ถ๐ป๐๐ฒ๐ฟ๐ฎ๐ฐ๐๐ถ๐ผ๐ป: ๐๐ต๐ถ๐ ๐ถ๐ ๐๐ผ๐น๐๐๐ฅ๐ง ๐
These encode each token from both query and doc as separate vectors as before, but compare all together without previously averaging them and losing information.
This is more accurate than no-interaction but also slower because you have to compare n*m vectors instead of 2. At least you can store documents in memory. And ColBERT has some optimisations like pooling to be faster.
โก๏ธ Notable examples: ColBERTv2, mxbai-colbert-large-v1, jina-colbert-v2
๐ฏ. ๐๐ฎ๐ฟ๐น๐ ๐ถ๐ป๐๐ฒ๐ฟ๐ฎ๐ฐ๐๐ถ๐ผ๐ป: ๐๐ฟ๐ผ๐๐-๐ฒ๐ป๐ฐ๐ผ๐ฑ๐ฒ๐ฟ๐ ๐๏ธ
Basically you run the concatenated query + document in a model to get a final score.
This is the most accurate, but also the slowest since it gets really long when you have many docs to rerank! And you cannot pre-store embeddings.
โก๏ธ Notable examples: MixedBread or Jina AI rerankers!
๐ So what you choose is a trade-off between speed and accuracy: I think ColBERT is often a really good choice!
Based on this great post by Jina AI ๐ https://jina.ai/news/what-is-colbert-and-late-interaction-and-why-they-matter
posted
an
update
about 1 month ago
Post
1692
By far the coolest release of the day!
> The Open LLM Leaderboard, most comprehensive suite for comparing Open LLMs on many benchmarks, just released a comparator tool that lets you dig into the detail of differences between any models.
Here's me checking how the new Llama-3.1-Nemotron-70B that we've heard so much compares to the original Llama-3.1-70B. ๐ค๐
Try it out here ๐ open-llm-leaderboard/comparator
> The Open LLM Leaderboard, most comprehensive suite for comparing Open LLMs on many benchmarks, just released a comparator tool that lets you dig into the detail of differences between any models.
Here's me checking how the new Llama-3.1-Nemotron-70B that we've heard so much compares to the original Llama-3.1-70B. ๐ค๐
Try it out here ๐ open-llm-leaderboard/comparator
- 2 replies