Shaping capabilities with token-level data filtering
Paper
โข
2601.21571
โข
Published
โข
11
AI & ML interests
None defined yet.
Recent Activity
View all activity
julien-cย
submitted
a
paper
to Daily Papers
about 17 hours ago
sergiopaniegoย
posted
an
update
2 days ago
Post
171
Meet the Post-Training Toolkit (PTT), which easily integrates with TRL via a single callback, by Aditya Challapally (
@microsoft
):
๐ Detects training issues early
๐ Lets you intervene safely
๐ Keeps long training runs stable, auditable & efficient
Microsoft blog: https://devblogs.microsoft.com/engineering-at-microsoft/diagnosing-instability-in-production-scale-agent-rl/
Integration guide: https://huggingface.co/docs/trl/main/en/ptt_integration
Code: https://github.com/microsoft/post-training-toolkit
๐ Detects training issues early
๐ Lets you intervene safely
๐ Keeps long training runs stable, auditable & efficient
Microsoft blog: https://devblogs.microsoft.com/engineering-at-microsoft/diagnosing-instability-in-production-scale-agent-rl/
Integration guide: https://huggingface.co/docs/trl/main/en/ptt_integration
Code: https://github.com/microsoft/post-training-toolkit
sergiopaniegoย
posted
an
update
3 days ago
Post
2402
New TRL + OpenEnv example! ๐ฅ
Fine tune an LLM for playing Sudoku using an RL env via OpenEnv
Includes a script that runs on 1 or multiple GPUs with vLLM, plus a Colab-ready notebook.
Enjoy!
Notebook: https://colab.research.google.com/github/huggingface/trl/blob/main/examples/notebooks/openenv_sudoku_grpo.ipynb
Script: https://github.com/huggingface/trl/blob/main/examples/scripts/openenv/sudoku.py
Fine tune an LLM for playing Sudoku using an RL env via OpenEnv
Includes a script that runs on 1 or multiple GPUs with vLLM, plus a Colab-ready notebook.
Enjoy!
Notebook: https://colab.research.google.com/github/huggingface/trl/blob/main/examples/notebooks/openenv_sudoku_grpo.ipynb
Script: https://github.com/huggingface/trl/blob/main/examples/scripts/openenv/sudoku.py
- 1 reply
qgallouedecย
updated
a
dataset
3 days ago
qgallouedecย
updated
a
dataset
4 days ago
qgallouedecย
updated
a
Space
4 days ago
i'm really liking the GPU usage and perf tracking here
1
#1 opened 5 months ago
by
Tonic
sergiopaniegoย
posted
an
update
5 days ago
Post
2045
Date idea: read the entire Transformers v5.0.0 release notes
Officially stable now: https://github.com/huggingface/transformers/releases/tag/v5.0.0
Officially stable now: https://github.com/huggingface/transformers/releases/tag/v5.0.0
- 1 reply
sergiopaniegoย
posted
an
update
12 days ago
Post
1554
FunctionGemma Tuning Lab is a new no-code tool by
@google
that lets you fine-tune a model directly from the browser, with no coding knowledge required, using TRL behind the scenes.
blog: https://developers.googleblog.com/a-guide-to-fine-tuning-functiongemma/
try it out: google/functiongemma-tuning-lab
This example builds on a more advanced one for learning fine-tuning with SFT using TRL: https://ai.google.dev/gemma/docs/functiongemma/finetuning-with-functiongemma
blog: https://developers.googleblog.com/a-guide-to-fine-tuning-functiongemma/
try it out: google/functiongemma-tuning-lab
This example builds on a more advanced one for learning fine-tuning with SFT using TRL: https://ai.google.dev/gemma/docs/functiongemma/finetuning-with-functiongemma
- 1 reply
sergiopaniegoย
posted
an
update
15 days ago
Post
760
TRL v0.27.0 is out!! ๐ฅณ
It includes GDPO, the latest variant of GRPO for multi-reward RL โจ
GDPO decouples reward normalization to avoid reward collapse and improve per-reward convergence โ developed by
@sliuau @SimonX et al.
Explore the paper: GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL Optimization (2601.05242)
Explore the full set of changes here:
https://github.com/huggingface/trl/releases/tag/v0.27.0
It includes GDPO, the latest variant of GRPO for multi-reward RL โจ
GDPO decouples reward normalization to avoid reward collapse and improve per-reward convergence โ developed by
@sliuau @SimonX et al.
Explore the paper: GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL Optimization (2601.05242)
Explore the full set of changes here:
https://github.com/huggingface/trl/releases/tag/v0.27.0
sergiopaniegoย
posted
an
update
18 days ago
Post
2970
New REPL environment in OpenEnv available! โจ
Used in the Recursive Language Models (RLM) paper by Alex Zhang.
Ready for inference & post-training using trajectories. Handles long contexts:
> Run Python code in a sandbox
> Make recursive calls to LMs
> Explore data programmatically
> Return final result
Docs: https://meta-pytorch.org/OpenEnv/environments/repl/
Inference script: https://github.com/meta-pytorch/OpenEnv/blob/main/examples/repl_oolong_simple.py
Used in the Recursive Language Models (RLM) paper by Alex Zhang.
Ready for inference & post-training using trajectories. Handles long contexts:
> Run Python code in a sandbox
> Make recursive calls to LMs
> Explore data programmatically
> Return final result
Docs: https://meta-pytorch.org/OpenEnv/environments/repl/
Inference script: https://github.com/meta-pytorch/OpenEnv/blob/main/examples/repl_oolong_simple.py
sergiopaniegoย
posted
an
update
19 days ago
Post
438
Recursive Language Models (RLM) is a new interface for LLMs with cool ideas by Alex Zhang!
โ ๏ธ LLMs struggle with long prompts โ attention overload & lost info
๐ RLMs inspect, split & call themselves on chunks, then aggregate results
โ Handles millions of tokens, reduces noise, improves reasoning
๐ก System prompt guides recursion
๐ฏ RLM trajectories can be used for RL training or distillation (OpenEnv+TRL!!)
We're adding it to OpenEnv (with Kashif Rasul): https://github.com/meta-pytorch/OpenEnv/pull/282
More resources:
> Paper: Recursive Language Models (2512.24601)
> Paper blog: https://alexzhang13.github.io/blog/2025/rlm/
> RLM repo: https://github.com/alexzhang13/rlm
โ ๏ธ LLMs struggle with long prompts โ attention overload & lost info
๐ RLMs inspect, split & call themselves on chunks, then aggregate results
โ Handles millions of tokens, reduces noise, improves reasoning
๐ก System prompt guides recursion
๐ฏ RLM trajectories can be used for RL training or distillation (OpenEnv+TRL!!)
We're adding it to OpenEnv (with Kashif Rasul): https://github.com/meta-pytorch/OpenEnv/pull/282
More resources:
> Paper: Recursive Language Models (2512.24601)
> Paper blog: https://alexzhang13.github.io/blog/2025/rlm/
> RLM repo: https://github.com/alexzhang13/rlm
- 2 replies
sergiopaniegoย
posted
an
update
23 days ago
Post
2247
New GRPO + TRL free Colab notebook out! ๐ฅ
Fine-tune 7B+ models on T4 GPUs thanks to a ton of memory optimizations for GRPO
7B model uses only 9.2 GB VRAM (~7ร reduction) ๐คฏ
Try the notebook here ๐ https://colab.research.google.com/github/huggingface/trl/blob/main/examples/notebooks/grpo_trl_lora_qlora.ipynb
Fine-tune 7B+ models on T4 GPUs thanks to a ton of memory optimizations for GRPO
7B model uses only 9.2 GB VRAM (~7ร reduction) ๐คฏ
Try the notebook here ๐ https://colab.research.google.com/github/huggingface/trl/blob/main/examples/notebooks/grpo_trl_lora_qlora.ipynb
sergiopaniegoย
posted
an
update
29 days ago
Post
2590
The list of hands-on notebooks (some beginner-friendly!) to get started with fine-tuning using TRL keeps growing!!
โข SFT
โข GRPO
โข Tool calling & agents
โข RL environments with OpenEnv
โข LLMs and VLMs
โจ Many run on FREE Colab, making it super easy to get started fast!
https://github.com/huggingface/trl/tree/main/examples/notebooks
โข SFT
โข GRPO
โข Tool calling & agents
โข RL environments with OpenEnv
โข LLMs and VLMs
โจ Many run on FREE Colab, making it super easy to get started fast!
https://github.com/huggingface/trl/tree/main/examples/notebooks
sergiopaniegoย
posted
an
update
about 1 month ago
Post
486
As the year comes to an end, itโs a good moment to catch up on some of the best long-form pieces published by the
@huggingface
team.
Iโve gathered them all here if you want to read or save them for later:
https://huggingface.co/collections/sergiopaniego/research-and-long-form-blog-posts
Iโve gathered them all here if you want to read or save them for later:
https://huggingface.co/collections/sergiopaniego/research-and-long-form-blog-posts
sergiopaniegoย
posted
an
update
about 1 month ago
Post
2309
This super detailed tutorial by
@Paulescu
is pure gold ๐ช "Fine-tuning a Small Language Model for browser control with GRPO and OpenEnv"
LFM2-350M ( @LiquidAI ) + BrowserGym (OpenEnv) + GRPO (TRL) for learning browser control ๐ค
https://paulabartabajo.substack.com/p/fine-tuning-lfm2-350m-for-browser
LFM2-350M ( @LiquidAI ) + BrowserGym (OpenEnv) + GRPO (TRL) for learning browser control ๐ค
https://paulabartabajo.substack.com/p/fine-tuning-lfm2-350m-for-browser
qgallouedecย
submitted
a
paper
to Daily Papers
about 1 month ago
sergiopaniegoย
posted
an
update
about 1 month ago
Post
494
if youโre on holidays ๐ and want some reading, here are blogs I contributed to this year:
๐ฆ VLMs in TRL: https://huggingface.co/blog/trl-vlm-alignment
๐ฆ VLMs in 2025: https://huggingface.co/blog/vlms-2025
๐พ tokenization in transformers v5: https://huggingface.co/blog/tokenizers
๐ธ faster transformers: https://huggingface.co/blog/faster-transformers
๐ฆ VLMs in TRL: https://huggingface.co/blog/trl-vlm-alignment
๐ฆ VLMs in 2025: https://huggingface.co/blog/vlms-2025
๐พ tokenization in transformers v5: https://huggingface.co/blog/tokenizers
๐ธ faster transformers: https://huggingface.co/blog/faster-transformers
sergiopaniegoย
posted
an
update
about 1 month ago
Post
2009
The Christmas holidays are here! ๐
Thinking about learning something new in AI?
@huggingface offers 12 FREE courses covering all the relevant topics, for every level of experience. A great challenge for the holidays (and worth saving for later ๐)
Letโs explore them!
๐ง ๐๐๐ ๐๐ผ๐๐ฟ๐๐ฒ: large language models with HF tools
https://huggingface.co/learn/llm-course
๐ค ๐๐ด๐ฒ๐ป๐๐ ๐๐ผ๐๐ฟ๐๐ฒ: build and deploy AI agents
https://huggingface.co/learn/agents-course
๐จ ๐๐ถ๐ณ๐ณ๐๐๐ถ๐ผ๐ป ๐๐ผ๐๐ฟ๐๐ฒ: diffusion models with ๐ค Diffusers
https://huggingface.co/learn/diffusion-course
๐ ๐๐๐ฑ๐ถ๐ผ ๐๐ผ๐๐ฟ๐๐ฒ: transformers for audio tasks
https://huggingface.co/learn/audio-course
๐ฎ ๐๐ฒ๐ฒ๐ฝ ๐ฅ๐ ๐๐ผ๐๐ฟ๐๐ฒ: deep reinforcement learning
https://huggingface.co/learn/deep-rl-course
๐๏ธ ๐๐ผ๐บ๐บ๐๐ป๐ถ๐๐ ๐๐ผ๐บ๐ฝ๐๐๐ฒ๐ฟ ๐ฉ๐ถ๐๐ถ๐ผ๐ป ๐๐ผ๐๐ฟ๐๐ฒ: modern computer vision with HF
https://huggingface.co/learn/computer-vision-course
๐ฆพ ๐ฅ๐ผ๐ฏ๐ผ๐๐ถ๐ฐ๐ ๐๐ผ๐๐ฟ๐๐ฒ (๐๐ฒ๐ฅ๐ผ๐ฏ๐ผ๐): learning-based robotics
https://huggingface.co/learn/robotics-course
๐งฉ ๐ ๐๐ฃ ๐๐ผ๐๐ฟ๐๐ฒ: Model Context Protocol explained
https://huggingface.co/learn/mcp-course
๐งช ๐ ๐ฆ๐บ๐ผ๐น ๐๐ผ๐๐ฟ๐๐ฒ: post-training AI models
https://huggingface.co/learn/a-smol-course
๐น๏ธ ๐ ๐ ๐ณ๐ผ๐ฟ ๐๐ฎ๐บ๐ฒ๐: AI in game development
https://huggingface.co/learn/ml-for-games-course
๐ง ๐ ๐ ๐ณ๐ผ๐ฟ ๐ฏ๐: machine learning for 3D data
https://huggingface.co/learn/ml-for-3d-course
๐ ๐ข๐ฝ๐ฒ๐ป-๐ฆ๐ผ๐๐ฟ๐ฐ๐ฒ ๐๐ ๐๐ผ๐ผ๐ธ๐ฏ๐ผ๐ผ๐ธ: practical AI notebooks
https://huggingface.co/learn/cookbook
All of them can be found here: https://huggingface.co/learn
Thinking about learning something new in AI?
@huggingface offers 12 FREE courses covering all the relevant topics, for every level of experience. A great challenge for the holidays (and worth saving for later ๐)
Letโs explore them!
๐ง ๐๐๐ ๐๐ผ๐๐ฟ๐๐ฒ: large language models with HF tools
https://huggingface.co/learn/llm-course
๐ค ๐๐ด๐ฒ๐ป๐๐ ๐๐ผ๐๐ฟ๐๐ฒ: build and deploy AI agents
https://huggingface.co/learn/agents-course
๐จ ๐๐ถ๐ณ๐ณ๐๐๐ถ๐ผ๐ป ๐๐ผ๐๐ฟ๐๐ฒ: diffusion models with ๐ค Diffusers
https://huggingface.co/learn/diffusion-course
๐ ๐๐๐ฑ๐ถ๐ผ ๐๐ผ๐๐ฟ๐๐ฒ: transformers for audio tasks
https://huggingface.co/learn/audio-course
๐ฎ ๐๐ฒ๐ฒ๐ฝ ๐ฅ๐ ๐๐ผ๐๐ฟ๐๐ฒ: deep reinforcement learning
https://huggingface.co/learn/deep-rl-course
๐๏ธ ๐๐ผ๐บ๐บ๐๐ป๐ถ๐๐ ๐๐ผ๐บ๐ฝ๐๐๐ฒ๐ฟ ๐ฉ๐ถ๐๐ถ๐ผ๐ป ๐๐ผ๐๐ฟ๐๐ฒ: modern computer vision with HF
https://huggingface.co/learn/computer-vision-course
๐ฆพ ๐ฅ๐ผ๐ฏ๐ผ๐๐ถ๐ฐ๐ ๐๐ผ๐๐ฟ๐๐ฒ (๐๐ฒ๐ฅ๐ผ๐ฏ๐ผ๐): learning-based robotics
https://huggingface.co/learn/robotics-course
๐งฉ ๐ ๐๐ฃ ๐๐ผ๐๐ฟ๐๐ฒ: Model Context Protocol explained
https://huggingface.co/learn/mcp-course
๐งช ๐ ๐ฆ๐บ๐ผ๐น ๐๐ผ๐๐ฟ๐๐ฒ: post-training AI models
https://huggingface.co/learn/a-smol-course
๐น๏ธ ๐ ๐ ๐ณ๐ผ๐ฟ ๐๐ฎ๐บ๐ฒ๐: AI in game development
https://huggingface.co/learn/ml-for-games-course
๐ง ๐ ๐ ๐ณ๐ผ๐ฟ ๐ฏ๐: machine learning for 3D data
https://huggingface.co/learn/ml-for-3d-course
๐ ๐ข๐ฝ๐ฒ๐ป-๐ฆ๐ผ๐๐ฟ๐ฐ๐ฒ ๐๐ ๐๐ผ๐ผ๐ธ๐ฏ๐ผ๐ผ๐ธ: practical AI notebooks
https://huggingface.co/learn/cookbook
All of them can be found here: https://huggingface.co/learn
sergiopaniegoย
updated
a
dataset
about 1 month ago